How to run Qwen2.5-Omni GGUF with llama-mtmd-cli locally?

[jordanqi]

I saw that Qwen2.5-Omni now supports both image and audio inputs, which is great! I downloaded the following GGUF files:

Qwen2.5-Omni-3B-Q8_0.gguf (language model)

mmproj-Qwen2.5-Omni-3B-Q8_0.gguf (multimodal projector)

I'm trying to run local inference using llama-mtmd-cli from llama.cpp.

What is the correct command to run inference locally with an image or an audio input? Or is there a related readme that introduces the usage of the audio model?

[ali0une]

Hi there.

Both llama-mtmd-cli and llama-server support audio for some models, models can be found at https://huggingface.co/collections/ggml-org/multimodal-ggufs

Doc is here https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md

See also #13759 #13760 and #13784 for some more details.

[audio] OK: llama-mtmd-cli ggml-org/Qwen2.5-Omni-3B-GGUF:Q4_K_M
[audio] OK: llama-mtmd-cli ggml-org/Qwen2.5-Omni-7B-GGUF:Q4_K_M
[audio] OK: llama-mtmd-cli ggml-org/ultravox-v0_5-llama-3_1-8b-GGUF:Q4_K_M
[audio] OK: llama-mtmd-cli ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF:Q8_0
[vision] OK: llama-mtmd-cli bartowski/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli cjpais/llava-1.6-mistral-7b-gguf:Q3_K_M
[vision] OK: llama-mtmd-cli ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/InternVL2_5-1B-GGUF:Q8_0
[vision] OK: llama-mtmd-cli ggml-org/InternVL3-14B-Instruct-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/InternVL3-1B-Instruct-GGUF:Q8_0
[vision] OK: llama-mtmd-cli ggml-org/InternVL3-8B-Instruct-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/Mistral-Small-3.1-24B-Instruct-2503-GGUF
[vision] OK: llama-mtmd-cli ggml-org/pixtral-12b-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/Qwen2.5-Omni-3B-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/Qwen2.5-Omni-7B-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/Qwen2.5-VL-7B-Instruct-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/Qwen2-VL-7B-Instruct-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/SmolVLM2-2.2B-Instruct-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli ggml-org/SmolVLM2-500M-Video-Instruct-GGUF:Q8_0
[vision] OK: llama-mtmd-cli ggml-org/SmolVLM-500M-Instruct-GGUF:Q8_0
[vision] OK: llama-mtmd-cli ibm-research/granite-vision-3.2-2b-GGUF:Q4_K_M
[vision] OK: llama-mtmd-cli openbmb/MiniCPM-o-2_6-gguf:Q4_0
[vision] OK: llama-mtmd-cli openbmb/MiniCPM-V-2_6-gguf:Q2_K
[vision] OK: llama-mtmd-cli second-state/Llava-v1.5-7B-GGUF:Q2_K
[vision] OK: llama-mtmd-cli second-state/MiniCPM-Llama3-V-2_5-GGUF:Q2_K
[vision] OK: llama-mtmd-cli THUDM/glm-edge-v-5b-gguf:Q4_K_M

As you can see both Qwen2.5-Omni-3B and Qwen2.5-Omni-7B are supported for audio+vision
Try with the 3B but it does not perform very well on audio, and 7B is not any better at this iMHO at the time i'm speaking.

Audio can be wav or mp3.

For llama-mtmd-cli you can do :

./llama.cpp/bin/llama-mtmd-cli \
-m ./whatever/Qwen2.5-Omni-7B/Qwen2.5-Omni-7B-Q4_K_M.gguf \
--mmproj ./whatever/Qwen2.5-Omni-7B/mmproj-Qwen2.5-Omni-7B-Q8_0 \
--ctx-size 8192 \
--threads 8 \
-ngl 999 \
--temp 0.1 \
--top-p 0.8 \
--top-k 100 \
--repeat-penalty 1.05 \
--audio "/path/to/audio.wav" \
--prompt "transcribe audio"

For llama-server you can do :

./llama.cpp/bin/llama-server \
-m ./whatever/Qwen2.5-Omni-7B/Qwen2.5-Omni-7B-Q4_K_M.gguf \
--mmproj ./whatever/Qwen2.5-Omni-7B/mmproj-Qwen2.5-Omni-7B-Q8_0.gguf \
--threads 8 \
-ngl 999 \
--host 0.0.0.0 \
--port 5000 \
--temp 0.1 \
--top-p 0.8 \
--top-k 100 \
--repeat-penalty 1.05

But you'd better try with Meta-Llama-3.1-8B-Instruct as results are much better :

llama-mtmd-cli :

./llama.cpp/bin/llama-mtmd-cli \
-m ./whatever/Meta-Llama-3.1-8B-Instruct/Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf \
--mmproj ./whatever/Meta-Llama-3.1-8B-Instruct/mmproj-ultravox-v0_5-llama-3_1-8b-f16.gguf \
--ctx-size 8192 \
--threads 8 \
-ngl 999 \
--temp 0.1 \
--top-p 0.8 \
--top-k 100 \
--repeat-penalty 1.05 \
--audio "/path/to/audio.wav" \
--prompt "transcribe audio"

llama-server :

./llama.cpp/bin/llama-server \
-m ./whatever/Meta-Llama-3.1-8B-Instruct/Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf \
--mmproj ./whatever/Meta-Llama-3.1-8B-Instruct/mmproj-ultravox-v0_5-llama-3_1-8b-f16.gguf \
--ctx-size 8192 \
--threads 8 \
-ngl 999 \
--host 0.0.0.0 \
--port 5000 \
--temp 0.1 \
--top-p 0.8 \
--top-k 100 \
--repeat-penalty 1.05