diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fd5097a..3457eea 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-22.04 strategy: matrix: - python-version: [3.8, 3.9, '3.10', 3.11, 3.12] + python-version: [3.9, '3.10', 3.11, 3.12] steps: - uses: actions/checkout@v2 @@ -25,7 +25,7 @@ jobs: python-version: ${{ matrix.python-version }} - name: Cache Python dependencies - uses: actions/cache@v2 + uses: actions/cache@v4 with: path: | ~/.cache/pip @@ -52,7 +52,7 @@ jobs: runs-on: ubuntu-22.04 strategy: matrix: - python-version: [3.8, 3.9, '3.10', 3.11, 3.12] + python-version: [3.9, '3.10', 3.11, 3.12] steps: - uses: actions/checkout@v2 @@ -157,6 +157,35 @@ jobs: push: true tags: ghcr.io/collabora/whisperlive-gpu:latest + build-and-push-docker-openvino: + needs: [run-tests, check-code-format, build-and-push-docker-cpu] + timeout-minutes: 20 + runs-on: ubuntu-22.04 + if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/')) + steps: + - uses: actions/checkout@v2 + + - name: Log in to GitHub Container Registry + uses: docker/login-action@v1 + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GHCR_TOKEN }} + + - name: Docker Prune + run: docker system prune -af + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v1 + + - name: Build and push Docker GPU image + uses: docker/build-push-action@v2 + with: + context: . + file: docker/Dockerfile.openvino + push: true + tags: ghcr.io/collabora/whisperlive-openvino:latest + publish-to-pypi: needs: [run-tests, check-code-format] runs-on: ubuntu-22.04 @@ -164,20 +193,20 @@ jobs: steps: - uses: actions/checkout@v2 - - name: Set up Python 3.8 + - name: Set up Python 3.9 uses: actions/setup-python@v2 with: - python-version: 3.8 + python-version: 3.9 - name: Cache Python dependencies - uses: actions/cache@v2 + uses: actions/cache@v4 with: path: | ~/.cache/pip !~/.cache/pip/log - key: ubuntu-latest-pip-3.8-${{ hashFiles('requirements/server.txt', 'requirements/client.txt') }} + key: ubuntu-latest-pip-3.9-${{ hashFiles('requirements/server.txt', 'requirements/client.txt') }} restore-keys: | - ubuntu-latest-pip-3.8- + ubuntu-latest-pip-3.9- - name: Install system dependencies run: sudo apt-get update && sudo apt-get install -y portaudio19-dev diff --git a/README.md b/README.md index 88d966a..2119540 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,16 @@ This project is a real-time transcription application that uses the OpenAI Whisp to convert speech input into text output. It can be used to transcribe both live audio input from microphone and pre-recorded audio files. +- [Installation](#installation) +- [Getting Started](#getting-started) +- [Running the Server](#running-the-server) +- [Running the Client](#running-the-client) +- [Browser Extensions](#browser-extensions) +- [Whisper Live Server in Docker](#whisper-live-server-in-docker) +- [Future Work](#future-work) +- [Contact](#contact) +- [Citations](#citations) + ## Installation - Install PyAudio ```bash @@ -26,7 +36,7 @@ input from microphone and pre-recorded audio files. - Please follow [TensorRT_whisper readme](https://github.com/collabora/WhisperLive/blob/main/TensorRT_whisper.md) for setup of [NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) and for building Whisper-TensorRT engine. ## Getting Started -The server supports two backends `faster_whisper` and `tensorrt`. If running `tensorrt` backend follow [TensorRT_whisper readme](https://github.com/collabora/WhisperLive/blob/main/TensorRT_whisper.md) +The server supports 3 backends `faster_whisper`, `tensorrt` and `openvino`. If running `tensorrt` backend follow [TensorRT_whisper readme](https://github.com/collabora/WhisperLive/blob/main/TensorRT_whisper.md) ### Running the Server - [Faster Whisper](https://github.com/SYSTRAN/faster-whisper) backend @@ -53,6 +63,16 @@ python3 run_server.py -p 9090 \ -trt /home/TensorRT-LLM/examples/whisper/whisper_small \ -m ``` + +- WhisperLive now supports the [OpenVINO](https://github.com/openvinotoolkit/openvino) backend for efficient inference on Intel CPUs, iGPU and dGPUs. Currently, we tested the models uploaded to [huggingface by OpenVINO](https://huggingface.co/OpenVINO?search_models=whisper). + - > **Docker Recommended:** Running WhisperLive with OpenVINO inside Docker automatically enables GPU support (iGPU/dGPU) without requiring additional host setup. + - > **Native (non-Docker) Use:** If you prefer running outside Docker, ensure the Intel drivers and OpenVINO runtime are installed and properly configured on your system. Refer to the documentation for [installing OpenVINO](https://docs.openvino.ai/2025/get-started/install-openvino.html?PACKAGE=OPENVINO_BASE&VERSION=v_2025_0_0&OP_SYSTEM=LINUX&DISTRIBUTION=PIP#). + +``` +python3 run_server.py -p 9090 -b openvino +``` + + #### Controlling OpenMP Threads To control the number of threads used by OpenMP, you can set the `OMP_NUM_THREADS` environment variable. This is useful for managing CPU resources and ensuring consistent performance. If not specified, `OMP_NUM_THREADS` is set to `1` by default. You can change this by using the `--omp_num_threads` argument: ```bash @@ -147,15 +167,21 @@ client(hls_url="http://as-hls-ww-live.akamaized.net/pool_904/live/ww/bbc_1xtra/b --trt_model_path "/app/TensorRT-LLM-examples/whisper/whisper_small_en_int4" ``` + - OpenVINO + ``` + docker run -it --device=/dev/dri -p 9090:9090 ghcr.io/collabora/whisperlive-openvino + ``` + - CPU -```bash -docker run -it -p 9090:9090 ghcr.io/collabora/whisperlive-cpu:latest -``` + - Faster-whisper + ```bash + docker run -it -p 9090:9090 ghcr.io/collabora/whisperlive-cpu:latest + ``` + **Note**: By default we use "small" model size. To build docker image for a different model size, change the size in server.py and then build the docker image. ## Future Work - [ ] Add translation to other languages on top of transcription. -- [x] TensorRT backend for Whisper. ## Contact diff --git a/docker/Dockerfile.openvino b/docker/Dockerfile.openvino new file mode 100644 index 0000000..12f20eb --- /dev/null +++ b/docker/Dockerfile.openvino @@ -0,0 +1,19 @@ +FROM openvino/ubuntu22_runtime:latest + +ARG DEBIAN_FRONTEND=noninteractive + +USER root + +RUN apt update && apt install -y portaudio19-dev python-is-python3 && apt-get clean && rm -rf /var/lib/apt/lists/* + +RUN pip install --no-cache-dir -U "pip>=24" + +RUN mkdir /app +WORKDIR /app + +COPY requirements/server.txt /app/ +RUN pip install --no-cache-dir -r server.txt && rm server.txt + +COPY whisper_live /app/whisper_live +COPY run_server.py /app +CMD ["python", "run_server.py", "--backend", "openvino"] diff --git a/requirements/server.txt b/requirements/server.txt index 37df171..d85ab59 100644 --- a/requirements/server.txt +++ b/requirements/server.txt @@ -10,4 +10,12 @@ jiwer evaluate numpy<2 openai-whisper==20240930 -tokenizers==0.20.3 \ No newline at end of file +tokenizers==0.20.3 + +# openvino +librosa +openvino +openvino-genai +openvino-tokenizers +optimum +optimum-intel \ No newline at end of file diff --git a/run_server.py b/run_server.py index db66da4..2dd5581 100644 --- a/run_server.py +++ b/run_server.py @@ -10,7 +10,7 @@ if __name__ == "__main__": parser.add_argument('--backend', '-b', type=str, default='faster_whisper', - help='Backends from ["tensorrt", "faster_whisper"]') + help='Backends from ["tensorrt", "faster_whisper", "openvino"]') parser.add_argument('--faster_whisper_custom_model_path', '-fw', type=str, default=None, help="Custom Faster Whisper Model") diff --git a/setup.py b/setup.py index 5382982..95512a8 100644 --- a/setup.py +++ b/setup.py @@ -54,7 +54,14 @@ setup( "openai-whisper==20240930", "kaldialign", "soundfile", - "tokenizers==0.20.3" + "tokenizers==0.20.3", + "librosa", + "numpy==1.26.4" + "openvino", + "openvino-genai", + "openvino-tokenizers", + "optimum", + "optimum-intel", ], - python_requires=">=3.8" + python_requires=">=3.9" ) diff --git a/whisper_live/backend/base.py b/whisper_live/backend/base.py index bb09fe5..2e83bf9 100644 --- a/whisper_live/backend/base.py +++ b/whisper_live/backend/base.py @@ -1,6 +1,7 @@ import json import logging import threading +import time import numpy as np @@ -26,6 +27,7 @@ class ServeClientBase(object): self.add_pause_thresh = 3 # add a blank to segment list as a pause(no speech) for 3 seconds self.transcript = [] self.send_last_n_segments = 10 + self.no_speech_thresh = 0.45 # text formatting self.pick_previous_segments = 2 @@ -34,13 +36,76 @@ class ServeClientBase(object): self.lock = threading.Lock() def speech_to_text(self): - raise NotImplementedError + """ + Process an audio stream in an infinite loop, continuously transcribing the speech. + + This method continuously receives audio frames, performs real-time transcription, and sends + transcribed segments to the client via a WebSocket connection. + + If the client's language is not detected, it waits for 30 seconds of audio input to make a language prediction. + It utilizes the Whisper ASR model to transcribe the audio, continuously processing and streaming results. Segments + are sent to the client in real-time, and a history of segments is maintained to provide context.Pauses in speech + (no output from Whisper) are handled by showing the previous output for a set duration. A blank segment is added if + there is no speech for a specified duration to indicate a pause. + + Raises: + Exception: If there is an issue with audio processing or WebSocket communication. + + """ + while True: + if self.exit: + logging.info("Exiting speech to text thread") + break + + if self.frames_np is None: + continue + + self.clip_audio_if_no_valid_segment() + + input_bytes, duration = self.get_audio_chunk_for_processing() + if duration < 1.0: + time.sleep(0.1) # wait for audio chunks to arrive + continue + try: + input_sample = input_bytes.copy() + result = self.transcribe_audio(input_sample) + + if result is None or self.language is None: + self.timestamp_offset += duration + time.sleep(0.25) # wait for voice activity, result is None when no voice activity + continue + self.handle_transcription_output(result, duration) + + except Exception as e: + logging.error(f"[ERROR]: Failed to transcribe audio chunk: {e}") + time.sleep(0.01) def transcribe_audio(self): raise NotImplementedError def handle_transcription_output(self): raise NotImplementedError + + def format_segment(self, start, end, text, completed=False): + """ + Formats a transcription segment with precise start and end times alongside the transcribed text. + + Args: + start (float): The start time of the transcription segment in seconds. + end (float): The end time of the transcription segment in seconds. + text (str): The transcribed text corresponding to the segment. + + Returns: + dict: A dictionary representing the formatted transcription segment, including + 'start' and 'end' times as strings with three decimal places and the 'text' + of the transcription. + """ + return { + 'start': "{:.3f}".format(start), + 'end': "{:.3f}".format(end), + 'text': text, + 'completed': completed + } def add_frames(self, frame_np): """ @@ -161,6 +226,33 @@ class ServeClientBase(object): except Exception as e: logging.error(f"[ERROR]: Sending data to client: {e}") + def get_previous_output(self): + """ + Retrieves previously generated transcription outputs if no new transcription is available + from the current audio chunks. + + Checks the time since the last transcription output and, if it is within a specified + threshold, returns the most recent segments of transcribed text. It also manages + adding a pause (blank segment) to indicate a significant gap in speech based on a defined + threshold. + + Returns: + segments (list): A list of transcription segments. This may include the most recent + transcribed text segments or a blank segment to indicate a pause + in speech. + """ + segments = [] + if self.t_start is None: + self.t_start = time.time() + if time.time() - self.t_start < self.show_prev_out_thresh: + segments = self.prepare_segments() + + # add a blank if there is no speech for 3 seconds + if len(self.text) and self.text[-1] != '': + if time.time() - self.t_start > self.add_pause_thresh: + self.text.append('') + return segments + def disconnect(self): """ Notify the client of disconnection and send a disconnect message. @@ -185,4 +277,94 @@ class ServeClientBase(object): """ logging.info("Cleaning up.") self.exit = True + + def get_segment_no_speech_prob(self, segment): + return getattr(segment, "no_speech_prob", 0) + def get_segment_start(self, segment): + return getattr(segment, "start", getattr(segment, "start_ts", 0)) + + def get_segment_end(self, segment): + return getattr(segment, "end", getattr(segment, "end_ts", 0)) + + def update_segments(self, segments, duration): + """ + Processes the segments from Whisper and updates the transcript. + Uses helper methods to account for differences between backends. + + Args: + segments (list): List of segments returned by the transcriber. + duration (float): Duration of the current audio chunk. + + Returns: + dict or None: The last processed segment (if any). + """ + offset = None + self.current_out = '' + last_segment = None + + # Process complete segments only if there are more than one + # and if the last segment's no_speech_prob is below the threshold. + if len(segments) > 1 and self.get_segment_no_speech_prob(segments[-1]) <= self.no_speech_thresh: + for s in segments[:-1]: + text_ = s.text + self.text.append(text_) + with self.lock: + start = self.timestamp_offset + self.get_segment_start(s) + end = self.timestamp_offset + min(duration, self.get_segment_end(s)) + if start >= end: + continue + if self.get_segment_no_speech_prob(s) > self.no_speech_thresh: + continue + self.transcript.append(self.format_segment(start, end, text_, completed=True)) + offset = min(duration, self.get_segment_end(s)) + + # Process the last segment if its no_speech_prob is acceptable. + if self.get_segment_no_speech_prob(segments[-1]) <= self.no_speech_thresh: + self.current_out += segments[-1].text + with self.lock: + last_segment = self.format_segment( + self.timestamp_offset + self.get_segment_start(segments[-1]), + self.timestamp_offset + min(duration, self.get_segment_end(segments[-1])), + self.current_out, + completed=False + ) + + # Handle repeated output logic. + if self.current_out.strip() == self.prev_out.strip() and self.current_out != '': + self.same_output_count += 1 + + # if we remove the audio because of same output on the nth reptition we might remove the + # audio thats not yet transcribed so, capturing the time when it was repeated for the first time + if self.end_time_for_same_output is None: + self.end_time_for_same_output = self.get_segment_end(segments[-1]) + time.sleep(0.1) # wait briefly for any new voice activity + else: + self.same_output_count = 0 + self.end_time_for_same_output = None + + # If the same incomplete segment is repeated too many times, + # append it to the transcript and update the offset. + if self.same_output_count > self.same_output_threshold: + if not self.text or self.text[-1].strip().lower() != self.current_out.strip().lower(): + self.text.append(self.current_out) + with self.lock: + self.transcript.append(self.format_segment( + self.timestamp_offset, + self.timestamp_offset + min(duration, self.end_time_for_same_output), + self.current_out, + completed=True + )) + self.current_out = '' + offset = min(duration, self.end_time_for_same_output) + self.same_output_count = 0 + last_segment = None + self.end_time_for_same_output = None + else: + self.prev_out = self.current_out + + if offset is not None: + with self.lock: + self.timestamp_offset += offset + + return last_segment diff --git a/whisper_live/backend/faster_whisper_backend.py b/whisper_live/backend/faster_whisper_backend.py index a00ea84..eb19022 100644 --- a/whisper_live/backend/faster_whisper_backend.py +++ b/whisper_live/backend/faster_whisper_backend.py @@ -44,7 +44,7 @@ class ServeClientFasterWhisper(ServeClientBase): self.task = task self.initial_prompt = initial_prompt self.vad_parameters = vad_parameters or {"onset": 0.5} - self.no_speech_thresh = 0.45 + self.same_output_threshold = 10 self.end_time_for_same_output = None @@ -221,161 +221,3 @@ class ServeClientFasterWhisper(ServeClientBase): if len(segments): self.send_transcription_to_client(segments) - - def speech_to_text(self): - """ - Process an audio stream in an infinite loop, continuously transcribing the speech. - - This method continuously receives audio frames, performs real-time transcription, and sends - transcribed segments to the client via a WebSocket connection. - - If the client's language is not detected, it waits for 30 seconds of audio input to make a language prediction. - It utilizes the Whisper ASR model to transcribe the audio, continuously processing and streaming results. Segments - are sent to the client in real-time, and a history of segments is maintained to provide context.Pauses in speech - (no output from Whisper) are handled by showing the previous output for a set duration. A blank segment is added if - there is no speech for a specified duration to indicate a pause. - - Raises: - Exception: If there is an issue with audio processing or WebSocket communication. - - """ - while True: - if self.exit: - logging.info("Exiting speech to text thread") - break - - if self.frames_np is None: - continue - - self.clip_audio_if_no_valid_segment() - - input_bytes, duration = self.get_audio_chunk_for_processing() - if duration < 1.0: - time.sleep(0.1) # wait for audio chunks to arrive - continue - try: - input_sample = input_bytes.copy() - result = self.transcribe_audio(input_sample) - - if result is None or self.language is None: - self.timestamp_offset += duration - time.sleep(0.25) # wait for voice activity, result is None when no voice activity - continue - self.handle_transcription_output(result, duration) - - except Exception as e: - logging.error(f"[ERROR]: Failed to transcribe audio chunk: {e}") - time.sleep(0.01) - - def format_segment(self, start, end, text, completed=False): - """ - Formats a transcription segment with precise start and end times alongside the transcribed text. - - Args: - start (float): The start time of the transcription segment in seconds. - end (float): The end time of the transcription segment in seconds. - text (str): The transcribed text corresponding to the segment. - - Returns: - dict: A dictionary representing the formatted transcription segment, including - 'start' and 'end' times as strings with three decimal places and the 'text' - of the transcription. - """ - return { - 'start': "{:.3f}".format(start), - 'end': "{:.3f}".format(end), - 'text': text, - 'completed': completed - } - - def update_segments(self, segments, duration): - """ - Processes the segments from whisper. Appends all the segments to the list - except for the last segment assuming that it is incomplete. - - Updates the ongoing transcript with transcribed segments, including their start and end times. - Complete segments are appended to the transcript in chronological order. Incomplete segments - (assumed to be the last one) are processed to identify repeated content. If the same incomplete - segment is seen multiple times, it updates the offset and appends the segment to the transcript. - A threshold is used to detect repeated content and ensure it is only included once in the transcript. - The timestamp offset is updated based on the duration of processed segments. The method returns the - last processed segment, allowing it to be sent to the client for real-time updates. - - Args: - segments(dict) : dictionary of segments as returned by whisper - duration(float): duration of the current chunk - - Returns: - dict or None: The last processed segment with its start time, end time, and transcribed text. - Returns None if there are no valid segments to process. - """ - offset = None - self.current_out = '' - last_segment = None - - # process complete segments - if len(segments) > 1 and segments[-1].no_speech_prob <= self.no_speech_thresh: - for i, s in enumerate(segments[:-1]): - text_ = s.text - self.text.append(text_) - with self.lock: - start, end = self.timestamp_offset + s.start, self.timestamp_offset + min(duration, s.end) - - if start >= end: - continue - if s.no_speech_prob > self.no_speech_thresh: - continue - - self.transcript.append(self.format_segment(start, end, text_, completed=True)) - offset = min(duration, s.end) - - # only process the last segment if it satisfies the no_speech_thresh - if segments[-1].no_speech_prob <= self.no_speech_thresh: - self.current_out += segments[-1].text - with self.lock: - last_segment = self.format_segment( - self.timestamp_offset + segments[-1].start, - self.timestamp_offset + min(duration, segments[-1].end), - self.current_out, - completed=False - ) - - if self.current_out.strip() == self.prev_out.strip() and self.current_out != '': - self.same_output_count += 1 - - # if we remove the audio because of same output on the nth reptition we might remove the - # audio thats not yet transcribed so, capturing the time when it was repeated for the first time - if self.end_time_for_same_output is None: - self.end_time_for_same_output = segments[-1].end - time.sleep(0.1) # wait for some voice activity just in case there is an unitended pause from the speaker for better punctuations. - else: - self.same_output_count = 0 - self.end_time_for_same_output = None - - # if same incomplete segment is seen multiple times then update the offset - # and append the segment to the list - if self.same_output_count > self.same_output_threshold: - if not len(self.text) or self.text[-1].strip().lower() != self.current_out.strip().lower(): - self.text.append(self.current_out) - with self.lock: - self.transcript.append(self.format_segment( - self.timestamp_offset, - self.timestamp_offset + min(duration, self.end_time_for_same_output), - self.current_out, - completed=True - )) - self.current_out = '' - offset = min(duration, self.end_time_for_same_output) - self.same_output_count = 0 - last_segment = None - self.end_time_for_same_output = None - else: - self.prev_out = self.current_out - - # update offset - if offset is not None: - with self.lock: - self.timestamp_offset += offset - - return last_segment - diff --git a/whisper_live/backend/openvino_backend.py b/whisper_live/backend/openvino_backend.py new file mode 100644 index 0000000..dae1cfc --- /dev/null +++ b/whisper_live/backend/openvino_backend.py @@ -0,0 +1,125 @@ +import json +import logging +import threading +import time + +from openvino import Core +from whisper_live.backend.base import ServeClientBase +from whisper_live.transcriber.transcriber_openvino import WhisperOpenVINO + + +class ServeClientOpenVINO(ServeClientBase): + SINGLE_MODEL = None + SINGLE_MODEL_LOCK = threading.Lock() + + def __init__(self, websocket, task="transcribe", device=None, language=None, client_uid=None, model="small.en", + initial_prompt=None, vad_parameters=None, use_vad=True, single_model=False): + """ + Initialize a ServeClient instance. + The Whisper model is initialized based on the client's language and device availability. + The transcription thread is started upon initialization. A "SERVER_READY" message is sent + to the client to indicate that the server is ready. + + Args: + websocket (WebSocket): The WebSocket connection for the client. + task (str, optional): The task type, e.g., "transcribe." Defaults to "transcribe". + device (str, optional): The device type for Whisper, "cuda" or "cpu". Defaults to None. + language (str, optional): The language for transcription. Defaults to None. + client_uid (str, optional): A unique identifier for the client. Defaults to None. + model (str, optional): Huggingface model_id for a valid OpenVINO model. + initial_prompt (str, optional): Prompt for whisper inference. Defaults to None. + single_model (bool, optional): Whether to instantiate a new model for each client connection. Defaults to False. + """ + super().__init__(client_uid, websocket) + self.language = "en" if language is None else language + if not self.language.startswith("<|"): + self.language = f"<|{self.language}|>" + + self.task = "transcribe" if task is None else task + self.same_output_threshold = 10 + self.end_time_for_same_output = None + + core = Core() + available_devices = core.available_devices + if 'GPU' in available_devices: + selected_device = 'GPU' + else: + gpu_devices = [d for d in available_devices if d.startswith('GPU')] + selected_device = gpu_devices[0] if gpu_devices else 'CPU' + self.device = selected_device + + + if single_model: + if ServeClientOpenVINO.SINGLE_MODEL is None: + self.create_model(model) + ServeClientOpenVINO.SINGLE_MODEL = self.transcriber + else: + self.transcriber = ServeClientOpenVINO.SINGLE_MODEL + else: + self.create_model(model) + + # threading + self.trans_thread = threading.Thread(target=self.speech_to_text) + self.trans_thread.start() + + self.websocket.send(json.dumps({ + "uid": self.client_uid, + "message": self.SERVER_READY, + "backend": "openvino" + })) + logging.info(f"Using OpenVINO device: {self.device}") + logging.info(f"Running OpenVINO backend with language: {self.language} and task: {self.task}") + + def create_model(self, model_id): + """ + Instantiates a new model, sets it as the transcriber. + """ + self.transcriber = WhisperOpenVINO( + model_id, + device=self.device, + language=self.language, + task=self.task + ) + + def transcribe_audio(self, input_sample): + """ + Transcribes the provided audio sample using the configured transcriber instance. + + If the language has not been set, it updates the session's language based on the transcription + information. + + Args: + input_sample (np.array): The audio chunk to be transcribed. This should be a NumPy + array representing the audio data. + + Returns: + The transcription result from the transcriber. The exact format of this result + depends on the implementation of the `transcriber.transcribe` method but typically + includes the transcribed text. + """ + if ServeClientOpenVINO.SINGLE_MODEL: + ServeClientOpenVINO.SINGLE_MODEL_LOCK.acquire() + result = self.transcriber.transcribe(input_sample) + if ServeClientOpenVINO.SINGLE_MODEL: + ServeClientOpenVINO.SINGLE_MODEL_LOCK.release() + return result + + def handle_transcription_output(self, result, duration): + """ + Handle the transcription output, updating the transcript and sending data to the client. + + Args: + result (str): The result from whisper inference i.e. the list of segments. + duration (float): Duration of the transcribed audio chunk. + """ + segments = [] + if len(result): + self.t_start = None + last_segment = self.update_segments(result, duration) + segments = self.prepare_segments(last_segment) + else: + # show previous output if there is pause i.e. no output from whisper + segments = self.get_previous_output() + + if len(segments): + self.send_transcription_to_client(segments) diff --git a/whisper_live/server.py b/whisper_live/server.py index eceb353..5a59ff7 100644 --- a/whisper_live/server.py +++ b/whisper_live/server.py @@ -11,7 +11,7 @@ import numpy as np from websockets.sync.server import serve from websockets.exceptions import ConnectionClosed from whisper_live.vad import VoiceActivityDetector - +from whisper_live.backend.base import ServeClientBase logging.basicConfig(level=logging.INFO) diff --git a/whisper_live/transcriber/transcriber_openvino.py b/whisper_live/transcriber/transcriber_openvino.py new file mode 100644 index 0000000..68a71a8 --- /dev/null +++ b/whisper_live/transcriber/transcriber_openvino.py @@ -0,0 +1,23 @@ +import librosa +import os + +import openvino_genai as ov_genai +import huggingface_hub as hf_hub + + +class WhisperOpenVINO(object): + def __init__(self, model_id="OpenVINO/whisper-tiny-fp16-ov", device="CPU", language="en", task="transcribe"): + model_path = model_id.split('/')[-1] + cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "openvino_whisper_models") + os.makedirs(cache_dir, exist_ok=True) + model_path = os.path.join(cache_dir, model_path) + if not os.path.exists(model_path): + hf_hub.snapshot_download(model_id, local_dir=model_path) + self.model = ov_genai.WhisperPipeline(str(model_path), device=device) + self.language = language + self.task = task + + def transcribe(self, input_audio): + outputs = self.model.generate(input_audio, return_timestamps=True, language=self.language, task=self.task) + outputs = [seg for seg in outputs.chunks] + return outputs