Upgrade tensorrt_llm to v0.18.2

Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
This commit is contained in:
makaveli10
2025-04-22 12:33:03 +00:00
parent d9cb4ffdd0
commit 47ee035f65
8 changed files with 134 additions and 57 deletions
+9 -1
View File
@@ -1,6 +1,6 @@
# WhisperLive-TensorRT
We have only tested the TensorRT backend in docker so, we recommend docker for a smooth TensorRT backend setup.
**Note**: We use `tensorrt_llm==0.15.0.dev2024111200`
**Note**: We use `tensorrt_llm==0.18.2`
## Installation
- Install [docker](https://docs.docker.com/engine/install/)
@@ -36,3 +36,11 @@ python3 run_server.py --port 9090 \
--trt_model_path "/app/TensorRT-LLM-examples/whisper/whisper_small_float16" \
--trt_multilingual
```
By default trt_backend uses cpp_session, to use python session pass `--trt_py_session` to run_server.py
```bash
python3 run_server.py --port 9090 \
--backend tensorrt \
--trt_model_path "/app/TensorRT-LLM-examples/whisper/whisper_small_float16" \
--trt_py_session
```