Welcome to whisper_live’s documentation!¶
+-
+
- +class whisper_live.server.ServeClient(websocket, task='transcribe', device=None, multilingual=False, language=None, client_uid=None)¶ +
-
+
- Attributes:
RATE (int): The audio sampling rate (constant) set to 16000. +SERVER_READY (str): A constant message indicating that the server is ready. +DISCONNECT (str): A constant message indicating that the client should disconnect. +client_uid (str): A unique identifier for the client. +data (bytes): Accumulated audio data. +frames (bytes): Accumulated audio frames. +language (str): The language for transcription. +task (str): The task type, e.g., “transcribe.” +transcriber (WhisperModel): The Whisper model for speech-to-text. +timestamp_offset (float): The offset in audio timestamps. +frames_np (numpy.ndarray): NumPy array to store audio frames. +frames_offset (float): The offset in audio frames. +text (list): List of transcribed text segments. +current_out (str): The current incomplete transcription. +prev_out (str): The previous incomplete transcription. +t_start (float): Timestamp for the start of transcription. +exit (bool): A flag to exit the transcription thread. +same_output_threshold (int): Threshold for consecutive same output segments. +show_prev_out_thresh (int): Threshold for showing previous output segments. +add_pause_thresh (int): Threshold for adding a pause (blank) segment. +transcript (list): List of transcribed segments. +send_last_n_segments (int): Number of last segments to send to the client. +wrapper (textwrap.TextWrapper): Text wrapper for formatting text. +pick_previous_segments (int): Number of previous segments to include in the output. +websocket: The WebSocket connection for the client.
+
+
-
+
- +add_frames(frame_np)¶ +
Add audio frames to the ongoing audio stream buffer.
+This method is responsible for maintaining the audio stream buffer, allowing the continuous addition +of audio frames as they are received. It also ensures that the buffer does not exceed a specified size +to prevent excessive memory usage.
+If the buffer size exceeds a threshold (45 seconds of audio data), it discards the oldest 30 seconds +of audio data to maintain a reasonable buffer size. If the buffer is empty, it initializes it with the provided +audio frame. The audio stream buffer is used for real-time processing of audio data for transcription.
+-
+
- Args:
frame_np (numpy.ndarray): The audio frame data as a NumPy array.
+
+
-
+
- +cleanup()¶ +
Perform cleanup tasks before exiting the transcription service.
+This method performs necessary cleanup tasks, including stopping the transcription thread, marking +the exit flag to indicate the transcription thread should exit gracefully, and destroying resources +associated with the transcription process.
+
-
+
- +disconnect()¶ +
Notify the client of disconnection and send a disconnect message.
+This method sends a disconnect message to the client via the WebSocket connection to notify them +that the transcription service is disconnecting gracefully.
+
-
+
- +fill_output(output)¶ +
Format the current incomplete transcription output by combining it with previous complete segments. +The resulting transcription is wrapped into two lines, each containing a maximum of 50 characters.
+It ensures that the combined transcription fits within two lines, with a maximum of 50 characters per line. +Segments are concatenated in the order they exist in the list of previous segments, with the most +recent complete segment first and older segments prepended as needed to maintain the character limit. +If a 3-second pause is detected in the previous segments, any text preceding it is discarded to ensure +the transcription starts with the most recent complete content. The resulting transcription is returned +as a single string.
+-
+
- Args:
output(str): The current incomplete transcription segment.
+
+- Returns:
str: A formatted transcription wrapped in two lines.
+
+
-
+
- +speech_to_text()¶ +
Process an audio stream in an infinite loop, continuously transcribing the speech.
+This method continuously receives audio frames, performs real-time transcription, and sends +transcribed segments to the client via a WebSocket connection.
+If the client’s language is not detected, it waits for 30 seconds of audio input to make a language prediction. +It utilizes the Whisper ASR model to transcribe the audio, continuously processing and streaming results. Segments +are sent to the client in real-time, and a history of segments is maintained to provide context.Pauses in speech +(no output from Whisper) are handled by showing the previous output for a set duration. A blank segment is added if +there is no speech for a specified duration to indicate a pause.
+-
+
- Raises:
Exception: If there is an issue with audio processing or WebSocket communication.
+
+
-
+
- +update_segments(segments, duration)¶ +
Processes the segments from whisper. Appends all the segments to the list +except for the last segment assuming that it is incomplete.
+Updates the ongoing transcript with transcribed segments, including their start and end times. +Complete segments are appended to the transcript in chronological order. Incomplete segments +(assumed to be the last one) are processed to identify repeated content. If the same incomplete +segment is seen multiple times, it updates the offset and appends the segment to the transcript. +A threshold is used to detect repeated content and ensure it is only included once in the transcript. +The timestamp offset is updated based on the duration of processed segments. The method returns the +last processed segment, allowing it to be sent to the client for real-time updates.
+-
+
- Args:
segments(dict) : dictionary of segments as returned by whisper +duration(float): duration of the current chunk
+
+- Returns:
-
+
- dict or None: The last processed segment with its start time, end time, and transcribed text.
Returns None if there are no valid segments to process.
+
+
+
-
+
- +class whisper_live.server.TranscriptionServer¶ +
Represents a transcription server that handles incoming audio from clients.
+-
+
- Attributes:
RATE (int): The audio sampling rate (constant) set to 16000. +vad_model (torch.Module): The voice activity detection model. +vad_threshold (float): The voice activity detection threshold. +clients (dict): A dictionary to store connected clients. +websockets (dict): A dictionary to store WebSocket connections. +clients_start_time (dict): A dictionary to track client start times. +max_clients (int): Maximum allowed connected clients. +max_connection_time (int): Maximum allowed connection time in seconds.
+
+
-
+
- +get_wait_time()¶ +
Calculate and return the estimated wait time for clients.
+-
+
- Returns:
float: The estimated wait time in minutes.
+
+
-
+
- +recv_audio(websocket)¶ +
Receive audio chunks from a client in an infinite loop.
+Continuously receives audio frames from a connected client +over a WebSocket connection. It processes the audio frames using a +voice activity detection (VAD) model to determine if they contain speech +or not. If the audio frame contains speech, it is added to the client’s +audio data for ASR. +If the maximum number of clients is reached, the method sends a +“WAIT” status to the client, indicating that they should wait +until a slot is available. +If a client’s connection exceeds the maximum allowed time, it will +be disconnected, and the client’s resources will be cleaned up.
+-
+
- Args:
websocket (WebSocket): The WebSocket connection for the client.
+
+- Raises:
Exception: If there is an error during the audio frame processing.
+
+
-
+
- +run(host, port=9090)¶ +
Run the transcription server.
+-
+
- Args:
host (str): The host address to bind the server. +port (int): The port number to bind the server.
+
+
Indices and tables¶
+-
+
- +
- +
- +