What exactly are the differences between the Hugging Face models, besides their size?

[RndyP]

What are the differences between tiny, small, base, large. etc? Are they different datasets with more or less input data for their training? Are they different "resolutions" with the same training data? Do they have different vocabularies?

[KitaitiMakoto]

Models for Whisper.cpp is converted from ones for OpenAI's Whisper. Differences between model sizes is described in its README: https://github.com/openai/whisper?tab=readme-ov-file#available-models-and-languages

[RndyP]

So is the vocabulary the same, but with different accuracy? Is the training set the same?

The reason I am asking is because I am going to a lot of work to create an error dictionary (in code) for the base model, and I wonder if I have to redo all this work if I change models.

[KitaitiMakoto]

So is the vocabulary the same, but with different accuracy? Is the training set the same?

As reading OpenAI's model card, that seems true, though it is not explicitly said.

I hope someone who is more familiar with Whisper than me answers. Or, you should ask OpenAI's developers if you need to make sure to know.

[misutoneko]

I know there's some difference between the .en models and multilingual ones, but yes it would be nice to know to what extent exactly.
But I haven't taken a deep dive into actual token usage, I only know this because with the original whisper I usually need to suppress token 50363 or 50364 to mitigate some hallucinations. The token number to suppress depends on which type of model I'm using.