Request for Guidance on Running whisper.cpp with cuBLAS on Windows

[bbscout]

I am new to both Whisper.cpp and C++, and I would appreciate some guidance on how to run whisper.cpp with cuBLAS. I have successfully downloaded the Windows binaries (whisper-blas-bin-x64.zip) and executed main.exe using the following command:

$ ./main.exe -m ./model/ggml-large.bin -l cs -osrt audio_16khz.wav

My system has a working CUDAToolkit installed, as I can use it with PyTorch (torch.cuda.is_available() >>> true). The CUDA Toolkit is installed in the following directory: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1.

I expect the find_package(CUDAToolkit) function to locate the Toolkit and set BLAS = 0. However, I am unsure if this is happening correctly. Please forgive me if I am overlooking something obvious. Here is the console output I get when running the command:

whisper_init_from_file_no_state: loading model from './model/ggml-large.bin'
whisper_model_load: loading model
...
system_info: n_threads = 4 / 20 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | COREML = 0 | 

main: processing 'audio_16khz.wav' (5341101 samples, 333.8 sec), 4 threads, 1 processors, lang = cs, task = transcribe, timestamps = 1 ...

[matrix07012]

I believe you either need to compile it yourself or they've just added GitHub action for a cublas build so you can get the latest build here: https://github.com/ggerganov/whisper.cpp/actions/runs/4885055280

[bbscout]

I already downloaded generated binaries with cuBLAS support. But it is not using BLAS (BLAS = 0) while running.

[savchenko]

I already downloaded generated binaries with cuBLAS support. But it is not using BLAS (BLAS = 0 while running.

I have noticed that Windows blas-enabled build of v1.4.0 reports BLAS = 0 as well.

@ggerganov , is it a regression?

[ggerganov]

@ggerganov , is it a regression?

Looks like it - I can't test on Windows. Hopefully someone will investigate and help resolve

[nalbion]

I've built it locally with WHISPER_CUBLAS set and get

System info: AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | COREML = 0 | 

[cddqssc]

system_info: n_threads = 4 / 20 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | COREML = 0 |

Although BLAS = 1， I see in task manager that GPU(RTX 3070TI) is not working, CPU is working.

[savchenko]

Compiled locally, works fine: #89 (comment)

[bobqianic]

@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:

Preparation

• Install Visual Studio 2022
  
• Install latest version CUDA Toolkit
• Download CMake (Portable Windows x64 ZIP)
• Download whisper.cpp source code from GitHub

Build

• Open CMD console
• Input cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_CUBLAS=1
• Input cmake.exe --build whisper_build_location --config release

Well done!!!

• All binaries can be found in whisper_build_location\bin\Release

[ShinichiYao]

@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:

Preparation

• Install Visual Studio 2022
  
• Install latest version CUDA Toolkit
• Download CMake (Portable Windows x64 ZIP)
• Download whisper.cpp source code from GitHub

Build

• Open CMD console
• Input cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_OPENBLAS=1
• Input cmake.exe --build whisper_build_location --config release

Well done!!!

• All binaries can be found in whisper_build_location\bin\Release

I get this error
Could NOT find BLAS (missing: BLAS_LIBRARIES)

[bobqianic]

@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:

Preparation

• Install Visual Studio 2022
  
• Install latest version CUDA Toolkit
• Download CMake (Portable Windows x64 ZIP)
• Download whisper.cpp source code from GitHub

Build

• Open CMD console
• Input cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_OPENBLAS=1
• Input cmake.exe --build whisper_build_location --config release

Well done!!!

• All binaries can be found in whisper_build_location\bin\Release

I get this error Could NOT find BLAS (missing: BLAS_LIBRARIES)

Oops, seems like I goofed up there. The command should be -D WHISPER_CUBLAS=1, not -D WHISPER_OPENBLAS=1. Thanks for bearing with me. Oh, and by the way, if you're looking to compile the OpenBLAS version, you'll need to download and install OpenBLAS first.

[LLLYF]

@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:

Preparation

• Install Visual Studio 2022
  
• Install latest version CUDA Toolkit
• Download CMake (Portable Windows x64 ZIP)
• Download whisper.cpp source code from GitHub

Build

• Open CMD console
• Input cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_CUBLAS=1
• Input cmake.exe --build whisper_build_location --config release

Well done!!!

• All binaries can be found in whisper_build_location\bin\Release

Sorry to bother you, sir, I meet some problems when I follow your command

[Connum]

For anyone coming across this in 2024: WHISPER_CUBLAS is deprecated, you have to use GGML_CUDA instead. If you keep getting the deprecation message, delete your build location first. So in order to build directly from the checked-out repository into a /dist folder, you'd use the following:

• Input cmake.exe -S ./ -B ./dist -D GGML_CUDA=1
• Input cmake.exe --build ./dist --config release

It might have to do with me using only the Build Tools instead of the full VS installation, but I also had to copy the files from
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\extras\visual_studio_integration\MSBuildExtensions
to
C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\BuildCustomizations
for the build configuration to run through.

Also make sure that your CUDA_PATH environment variable is set correctly.