Whisper with Urro: Native speech recognition, speaker diarization, and word-level timestamps, all in one model

[urroxyz]

Whisper with Urro WHISPER + URRO

Multilingual automatic speech recognition (ASR) with speaker segmentation (SS) / speaker diarization (SD) and word-level timestamps (WLT)

Installation

Latest

pip install git+https://github.com/urroxyz/[email protected]

Development

pip install git+https://github.com/urroxyz/whisper

Introduction

After immense experimentation, I have discovered that, yes, Whisper can segment speakers and timestamp words! And I have created WHISPER + URRO to offer an easy solution therefor.

By modifying the thinking process of the OpenAI model, we can force it to delimit new speakers with symbols like hyphens (-) or greater-thans (>), or even with complete labels such as [SPEAKER 1] and [SPEAKER 2] to keep track of who is speaking and when.¹ By extracting cross-attentions and processesing them with dynamic-time warping, we can reconstruct timestamps on the word level rather than relying on occasional generated time tokens.²

Supported models

Official

Size	Parameters	New-speaker segmentation	Speaker diarization	Word-level timestamps
tiny3 tiny.en4	39 M	✓	x	✓
base5 base.en6	74 M	✓	x	✓
small7 small.en8	244 M	✓	✓	✓
medium9 medium.en10	769 M	✓	✓	x
large-v311	1550 M	✓	✓	x
large-v3-turbo12	809 M	✓	x	✓

Third-party

Model	Parameters	New-speaker segmentation	Speaker diarization	Word-level timestamps
whisper-d-v1a13	1550 M	✓	✓	x

Comparison

Source	Transcript
Ground truth	[SPEAKER 1] Down in front. [SPEAKER 2] Hey, sit down, that’s wrong of you. [SPEAKER 1] The little lady who is to become Mrs. Harvey Yates over my dead body. [SPEAKER 3] I know I have the sincere wishes of all my friends… and can only tell you how much I appreciate it. I think I can honestly say this is the happiest moment of my life. Look what I have here. It’s a little engagement present just given me by Mr. Yates.
Pretrained model (medium) No speaker labels	Down in front. Hey, sit down, that’s fine. The little lady who is to become Mrs. Harvey Yates over my dead body. [APPLAUSE] I know I have the sincere wishes of all my friends, and can only tell you how much I appreciate it. I think I can honestly say this is the happiest moment of my life. Look what I have here… It’s a little engagement present just given me by Mr. Yates.
Pretrained model (medium) with WHISPER + URRO delimiter=SPEAKER() prompt=SPEAKERS(3, "en") Correct speaker labels	[SPEAKER 1] Down in front. [SPEAKER 2] Hey, sit down, that’s fine. [SPEAKER 1] The little lady who is to become Mrs. Harvey Yates over my dead body. [APPLAUSE] [SPEAKER 3] I know I have the sincere wishes of all my friends, and can only tell you how much I appreciate it. I think I can honestly say this is the happiest moment of my life. Look what I have here… It’s a little engagement present just given me by Mr. Yates.
Finetuned model (d-v1a) Incorrect speaker labels	[S1] Down in front. [S2] Hey, sit down, it’s warm. [S1] The little lady who is to become Mrs. Harvey Yates, over my dead body. [S2] I know I have the sincere wishes of all my friends, and can only tell you how much I appreciate it. I think I can honestly say this is the happiest moment of my life. Look what I have here.
Finetuned model (d-v1a) with WHISPER + URRO delimiter=SPEAKER(short=True) prompt=SPEAKERS(3, "en", short=True) Correct speaker labels	[S1] Down in front. [S2] Hey, sit down, it’s warm. [S1] The little lady who is to become Mrs. Harvey Yates, over my dead body. [S3] I know I have the sincere wishes of all my friends, and can only tell you how much I appreciate it. I think I can honestly say this is the happiest moment of my life. Look what I have here.

Footnotes

1. Unique to WHISPER + URRO. ↩

2. As explicitly implemented in whisper-timestamped, alongside other libraries, such as openai-whisper. ↩

3. https://huggingface.co/onnx-community/whisper-tiny_timestamped ↩

4. https://huggingface.co/onnx-community/whisper-tiny.en_timestamped ↩

5. https://huggingface.co/onnx-community/whisper-base_timestamped ↩

6. https://huggingface.co/onnx-community/whisper-base.en_timestamped ↩

7. https://huggingface.co/onnx-community/whisper-small_timestamped ↩

8. https://huggingface.co/onnx-community/whisper-small.en_timestamped ↩

9. https://huggingface.co/onnx-community/whisper-medium-ONNX ↩

10. https://huggingface.co/onnx-community/whisper-medium.en-ONNX ↩

11. https://huggingface.co/onnx-community/whisper-large-v3-ONNX ↩

12. https://huggingface.co/onnx-community/whisper-large-v3-turbo_timestamped ↩

13. onnx-community/whisper-d-v1a-ONNX ↩

[kinchitagg]

Hey this looks interesting to me. The speaker segregation is an important aspect of generating subtitle as it can give clues for video editing via automation. However, is your project support api if I ran it locally ? Also to test your project should I just install your package ? can you help ?

[urroxyz]

Thank you so much for your interest.
Please visit the repo page and review its README.md for more information.