CLBlast support for Mixtral

[paralin]

The Mixtral PR has been merged: #4428

Testing latest commit 948ff13 with CLBlast and Mixtral I see some assertion errors:

GGML_ASSERT: ggml.c:7765: ggml_are_same_shape(src0, src1)
GGML_ASSERT: ggml.c:7765: ggml_are_same_shape(src0, src1)
GGML_ASSERT: ggml.c:7765: ggml_are_same_shape(src0, src1)
GGML_ASSERT: ggml.c:7765: ggml_are_same_shape(src0, src1)
GGML_ASSERT: ggml.c:7765: ggml_are_same_shape(src0, src1)
GGML_ASSERT: ggml.c:7765: ggml_are_same_shape(src0, src1)
GGML_ASSERT: ggml.c:7765: ggml_are_same_shape(src0, src1)
GGML_ASSERT: ggml.c:7765: ggml_are_same_shape(src0, src1)

I assume that the CLBlast implementation has not yet been updated to support Mixtral, so filing this issue to request this improvement. Thanks!

[clemens98]

i manged to launch it alright but could not add any input

[paryska99]

Yeah, the Blas prompt processing seems to not work like it should, hopefuly we'll get a fix for that pretty soon.

[paralin]

I discovered that all models are broken with CLBlast and git bisected the bad commit, see #4453

Fix: git revert 4d98d9a

[shibe2]

Is this still a problem after 55e87c3?

[kurnevsky]

Somehow it works, but performance is not that great. It makes inference just a bit faster:

CPU

llama_print_timings:        load time =    2068.49 ms
llama_print_timings:      sample time =       2.45 ms /    98 runs   (    0.03 ms per token, 39983.68 tokens per second)
llama_print_timings: prompt eval time =  215288.33 ms /    74 tokens ( 2909.30 ms per token,     0.34 tokens per second)
llama_print_timings:        eval time =   21885.91 ms /    97 runs   (  225.63 ms per token,     4.43 tokens per second)
llama_print_timings:       total time =  240440.55 ms

OpenCL

llama_print_timings:        load time =    6595.12 ms
llama_print_timings:      sample time =       8.04 ms /   342 runs   (    0.02 ms per token, 42547.90 tokens per second)
llama_print_timings: prompt eval time =  195997.10 ms /    74 tokens ( 2648.61 ms per token,     0.38 tokens per second)
llama_print_timings:        eval time =   63946.82 ms /   342 runs   (  186.98 ms per token,     5.35 tokens per second)
llama_print_timings:       total time =  356240.05 ms

(these numbers are for offloading only 14 layers)

[ggerganov]

Mixtral support with OpenCL requires some extra work

[shibe2]

It would be nice to have full to-do list specifically for this issue. I see OpenCL-related remarks in ggml.c in ggml_compute_forward_mul_f32, ggml_compute_forward_out_prod_f32, ggml_compute_forward_out_prod_q_f32. Which ones are needed for Mixtral?

[clemens98]

input still does not work . the input bar is blinking and stops blinking when i input something but no text does actually show up pressing enter also does not do anything

thanks to all the people working with this never ending project feels like there are several models released each day that break things or need support

weird was about to shutdown my pc and i realized i didn't shutdown the program and it actually worked but was extremely context focused until it started to ramble about pubs with itself guess it did get thirsty
it was basically answering most questions very well but couldn't handle context changes at all
biggest city it gave a list of biggest city's
biggest mountains gave a list of the biggest mountains
but changing the subject like "who are you" it just continued rambling about city's (the old mistral 7b did handle that question better than expected even finding the context of its own name )

it seams to completely fail to read and understand the bob transcript (which worked for mistral 7b) .
Looks like getting the mountain right was just pure luck

[paryska99]

input still does not work . the input bar is blinking and stops blinking when i input something but no text does actually show up pressing enter also does not do anything

thanks to all the people working with this never ending project feels like there are several models released each day that break things or need support

weird was about to shutdown my pc and i realized i didn't shutdown the program and it actually worked but was extremely context focused until it started to ramble about pubs with itself guess it did get thirsty it was basically answering most questions very well but couldn't handle context changes at all biggest city it gave a list of biggest city's biggest mountains gave a list of the biggest mountains but changing the subject like "who are you" it just continued rambling about city's (the old mistral 7b did handle that question better than expected even finding the context of its own name )

it seams to completely fail to read and understand the bob transcript (which worked for mistral 7b) . Looks the the getting the mountain right was just pure luck

Yeah for me the CuBlas or any other Blas basically does the same evaluation speed as the cpu (maybe 1 tk/s faster) making mixtral completely unusable

[clemens98]

shouldn't this be marked has a bug not "enhancement"

[clemens98]

sadly no news anywhere regarding mixtral and clblast and it isn't mentioned on the roadmap either

[github-actions]

This issue is stale because it has been open for 30 days with no activity.