WhisperLive/TensorRT_whisper.md

Whisper-TensorRT

We have only tested the TensorRT backend in docker so, we recommend docker for a smooth TensorRT backend setup. Note: We use our fork to setup TensorRT

Installation

• Install docker

• Install nvidia-container-toolkit

• Clone this repo.

git clone https://github.com/collabora/WhisperLive.git
cd WhisperLive

• Build the TensorRT-LLM docker image

docker build --file docker/Dockerfile.tensorrt --tag tensorrt_llm:latest .

NOTE: This could take some time.

• Next, we run the docker image and mount WhisperLive repo to the containers /home directory.

docker run -it --gpus all --shm-size=8g \
       --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
       -v /path/to/WhisperLive:/home/WhisperLive \
       tensorrt_llm:latest

cd /home/TensorRT-LLM
python3 scripts/build_wheel.py --cuda_architectures "89-real" --clean --trt_root /usr/local/tensorrt
pip install build/tensorrt_llm*.whl

NOTE: --cuda_architectures "89-real" builds for 4090, change according to your device.

• Make sure to test the installation.

# export ENV=${ENV:-/etc/shinit_v2} 
# source $ENV
python -c "import torch; import tensorrt; import tensorrt_llm"

NOTE: Uncomment and update library paths if imports fail.

Whisper TensorRT Engine

• We build small.en and small multilingual TensorRT engine. The script logs the path of the directory with Whisper TensorRT engine. We need the model_path to run the server.

# convert small.en
bash build_whisper_tensorrt /path/to/TensorRT-LLM/examples small.en

# convert small multilingual model
bash build_whisper_tensorrt /path/to/TensorRT-LLM/examples small

Run WhisperLive Server with TensorRT Backend

cd /home/WhisperLive

# Required to create mel spectogram
wget --directory-prefix=assets assets/mel_filters.npz https://raw.githubusercontent.com/openai/whisper/main/whisper/assets/mel_filters.npz

# Run English only model
python3 run_server.py --port 9090 \
                      --backend tensorrt \
                      --trt_model_path "path/from/build/step"

# Run Multilingual model
python3 run_server.py --port 9090 \
                      --backend tensorrt \
                      --trt_model_path "path/from/build/step" \
                      --trt_multilingual