Support for large-v3

[noe]

OpenAI released their large-v3 whisper model: openai/whisper#1762

It would be great for whisper.cpp to support it

[dataf3l]

I came here to post this exact same issue, thank you @noe, from the looks of the commit, maybe you can try swap the model and see if that kinda works I guess.

[cspenn]

If the model does get revved, will it be in the new GGUF format?

[jordibruin]

I tried to convert the pt to ggml but ran into some issues. If anyone wants to try:

https://openaipublic.azureedge.net/main/whisper/models/e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/large-v3.pt

[lanma]

large-v3.pt download code:

import whisper
model = whisper.load_model("large-v3")

I tried using this script to convert the large-v3.pt to the ggml file, but it seems the output ggml model file is not correct.
python models/convert-pt-to-ggml.py ~/.cache/whisper/large-v3.pt ../whisper ./models/
(Output file: ./models/ggml-model.bin)
Any suggestions?

[jordibruin]

@lanma what is wrong with the model file for you? Do you get errors?

[lanma]

test.wav same file results

Large-v2 ggml transcribe with zh language setting
main: processing './test.wav' (36096 samples, 2.3 sec), 4 threads, 1 processors, lang = zh, task = transcribe, timestamps = 1 ...
ggml_metal_add_buffer: allocated 'kv_self_1 ' buffer, size = 70.02 MB, ( 3629.22 / 21845.34)
[00:00:00.000 --> 00:00:02.000] 這是一個中文測試

Large-v3 ggml after convert, transcribe with zh language setting
system_info: n_threads = 4 / 10 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | METAL = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | COREML = 0 | OPENVINO = 0 |
main: WARNING: model is not multilingual, ignoring language and translation options
main: processing './test.wav' (36096 samples, 2.3 sec), 4 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...
ggml_metal_add_buffer: allocated 'kv_self_1 ' buffer, size = 70.02 MB, ( 3629.77 / 21845.34)

Large-v3 ggml after convert, transcribe with en language setting
system_info: n_threads = 4 / 10 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | METAL = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | COREML = 0 | OPENVINO = 0 |
main: processing './test.wav' (36096 samples, 2.3 sec), 4 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...
ggml_metal_add_buffer: allocated 'kv_self_1 ' buffer, size = 70.02 MB, ( 3629.77 / 21845.34)
[00:00:00.040 --> 00:00:30.000] sifications 20 It

[bobqianic]

Hey, can you give this PR a spin and check if it's working alright? #1444

[piotr-sikora-v]

Hi, I have problem with Polish language.
After change to new large model nothing is generated.
I run download and after that ./models/generate-coreml-model.sh

Also I see some strange warnign:

main: WARNING: model is not multilingual, ignoring language and translation options

Of corse standard openai-whisper works without problem on large-v3 with same file and Polish language.

[bobqianic]

Hi, I have problem with Polish language. After change to new large model nothing is generated. I run download and after that ./models/generate-coreml-model.sh

Also I see some strange warnign:

main: WARNING: model is not multilingual, ignoring language and translation options

Of corse standard openai-whisper works without problem on large-v3 with same file and Polish language.

Please make sure to download the latest version from the master branch and compile it yourself, as the version of whisper.cpp you're currently using is outdated.

I can provide a Windows binary for testing purposes right here:
whisper.cpp-39a240b-win64-openblas.zip

[lanma]

After updating my whisper.cpp to the latest version, everything is working well! Thank you, everyone.

[piotr-sikora-v]

Hi, I have problem with Polish language. After change to new large model nothing is generated. I run download and after that ./models/generate-coreml-model.sh
Also I see some strange warnign:

main: WARNING: model is not multilingual, ignoring language and translation options

Of corse standard openai-whisper works without problem on large-v3 with same file and Polish language.

Please make sure to download the latest version from the master branch and compile it yourself, as the version of whisper.cpp you're currently using is outdated.

I can provide a Windows binary for testing purposes right here: whisper.cpp-39a240b-win64-openblas.zip

ok, after upgrade now I have error with build CoreML model:

# ./models/generate-coreml-model.sh large
ModelDimensions(n_mels=128, n_audio_ctx=1500, n_audio_state=1280, n_audio_head=20, n_audio_layer=32, n_vocab=51866, n_text_ctx=448, n_text_state=1280, n_text_head=20, n_text_layer=32)
Traceback (most recent call last):
  File "/xxxxx/whisper.cpp/models/convert-whisper-to-coreml.py", line 323, in <module>
    encoder = convert_encoder(hparams, encoder, quantize=args.quantize)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxxxx/whisper.cpp/models/convert-whisper-to-coreml.py", line 257, in convert_encoder
    traced_model = torch.jit.trace(model, input_data)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/jit/_trace.py", line 798, in trace
    return trace_module(
           ^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/jit/_trace.py", line 1065, in trace_module
    module._c._create_method_from_trace(
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/whisper/model.py", line 162, in forward
    x = F.gelu(self.conv1(x))
               ^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 310, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/whisper/model.py", line 48, in _conv_forward
    return super()._conv_forward(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 306, in _conv_forward
    return F.conv1d(input, weight, bias, self.stride,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Given groups=1, weight of size [1280, 128, 3], expected input[1, 80, 3000] to have 128 channels, but got 80 channels instead
coremlc: error: Model does not exist at models/coreml-encoder-large.mlpackage -- file:///xxxxx/whisper.cpp/
mv: rename models/coreml-encoder-large.mlmodelc to models/ggml-large-encoder.mlmodelc: No such file or directory

[bobqianic]

Hi, I have problem with Polish language. After change to new large model nothing is generated. I run download and after that ./models/generate-coreml-model.sh
Also I see some strange warnign:

main: WARNING: model is not multilingual, ignoring language and translation options

Of corse standard openai-whisper works without problem on large-v3 with same file and Polish language.

Please make sure to download the latest version from the master branch and compile it yourself, as the version of whisper.cpp you're currently using is outdated.
I can provide a Windows binary for testing purposes right here: whisper.cpp-39a240b-win64-openblas.zip

ok, after upgrade now I have error with build CoreML model:

# ./models/generate-coreml-model.sh large
ModelDimensions(n_mels=128, n_audio_ctx=1500, n_audio_state=1280, n_audio_head=20, n_audio_layer=32, n_vocab=51866, n_text_ctx=448, n_text_state=1280, n_text_head=20, n_text_layer=32)
Traceback (most recent call last):
  File "/xxxxx/whisper.cpp/models/convert-whisper-to-coreml.py", line 323, in <module>
    encoder = convert_encoder(hparams, encoder, quantize=args.quantize)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxxxx/whisper.cpp/models/convert-whisper-to-coreml.py", line 257, in convert_encoder
    traced_model = torch.jit.trace(model, input_data)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/jit/_trace.py", line 798, in trace
    return trace_module(
           ^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/jit/_trace.py", line 1065, in trace_module
    module._c._create_method_from_trace(
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/whisper/model.py", line 162, in forward
    x = F.gelu(self.conv1(x))
               ^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1508, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 310, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/whisper/model.py", line 48, in _conv_forward
    return super()._conv_forward(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 306, in _conv_forward
    return F.conv1d(input, weight, bias, self.stride,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Given groups=1, weight of size [1280, 128, 3], expected input[1, 80, 3000] to have 128 channels, but got 80 channels instead
coremlc: error: Model does not exist at models/coreml-encoder-large.mlpackage -- file:///xxxxx/whisper.cpp/
mv: rename models/coreml-encoder-large.mlmodelc to models/ggml-large-encoder.mlmodelc: No such file or directory

Yes, there is a bug in the script. The input dimensions should not be hardcoded.

[piotr-sikora-v]

Hi, I have problem with Polish language. After change to new large model nothing is generated. I run download and after that `./models/generate-coreml-model.sh
ok, after upgrade now I have error with build CoreML model:

# ./models/generate-coreml-model.sh large
ModelDimensions(n_mels=128, n_audio_ctx=1500, n_audio_state=1280, n_audio_head=20, n_audio_layer=32, n_vocab=51866, n_text_ctx=448, n_text_state=1280, n_text_head=20, n_text_layer=32)
Traceback (most recent call last):
.....
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Given groups=1, weight of size [1280, 128, 3], expected input[1, 80, 3000] to have 128 channels, but got 80 channels instead
coremlc: error: Model does not exist at models/coreml-encoder-large.mlpackage -- file:///xxxxx/whisper.cpp/
mv: rename models/coreml-encoder-large.mlmodelc to models/ggml-large-encoder.mlmodelc: No such file or directory

Yes, there is a bug in the script. The input dimensions should not be hardcoded.

I tried to change it from 80 to 128 in this file, and after that now I have error here:

/opt/homebrew/lib/python3.11/site-packages/whisper/model.py:166: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert x.shape[1:] == self.positional_embedding.shape, "incorrect audio shape"

file is generated, but transcription not working :/

any sugestion?