Segmentation fault

[TruongGiangBT]

I apologize for the inconvenience. I am deploying a server with the following parameters: -cb -v --embedding -np 3 -c 8192 --host "0.0.0.0" -ngl 64. When I perform multiple embedding requests, a segmentation fault occurs. I noticed that if there are 2 slots performing the embedding task simultaneously, it causes an error. I hope to receive a solution soon. Thank you very much.

Originally posted by @TruongGiangBT in #3876 (comment)

[phymbert]

Added in: issues.feature

@ggerganov @ngxson On it 👍

[phymbert]

@TruongGiangBT Please confirm you do not face any error anymore. Feel free to reopen if any.

[TruongGiangBT]

@phymbert The segmentation fault error has been fixed. However, when I simultaneously execute multiple requests with the same input value, the embedding results are different.
My run command:
docker run -dp 6900:8080
--gpus '"device=1"'
--name vistral_llm
--restart=always
-v /home/gianght/llm_models/:/models mistral-wrapped-llama.cpp:full-cuda
--server -m /models/ggml-vistral-7B-chat-q4_1.gguf
-cb -v --embedding
-np 8
-c 32768
--host "0.0.0.0"
-ngl 64
-b 4096
Test code:

[phymbert]

Could you please open a dedicated discussion on embedding feature ?

[TruongGiangBT]

llama.cpp:

server.cpp:

What do you think if i fix it like this?
I have tried. With the same input, if the tokens of the sequence are entirely within one batch when decoding, then the embedding result will be the same. If the tokens of that slot are decoded multiple times, the results will be different, but the cosine similarity with the correct embedding result is > 0.999.