Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (and the Triton backend) this February 27, 2024.

This update includes:

• Model Support
  • Adding distil-whisper model support, thanks to the contribution from @Bhuvanesh09 in PR Adding distil-whisper model support to TensorRT-LLM #1061
  • Migrate BLIP-2 examples to examples/multimodal
• Features
  • BART LoRA support (limited to the Python runtime)
  • Support early_stopping=False in beam search for C++ Runtime
  • Add logits post processor to the batch manager (see docs here)
• Bug fixes
  • Multi-GPU fixes for multimodal examples How to use multi-gpu in running llava？ #1003
  • Fix wrong end_id issue for Qwen qwen end_id setting is wrong so cannot stop at right postition! #987
  • Fix a non-stopping generation issue LLAVA is slow due to unnecessary output tokens #1118 Why is llava trt-llm not much faster than transformers? #1123
• Documentation
  • Update precision support matrix for Whisper

Thanks,
The TensorRT-LLM Engineering Team

[akhoroshev]

Add logits post processor to the batch manager (see docs here)

It's very good and helps to customize for product tasks, thanks!

Please add the ability to configure the sampling pipeline (adding custom BaseSamplingLayer)