Can llamacpp server serve inferences for 100 pararel users in good conditions ?

[lbarasc]

Hi,
I need to build a custom computer to serve llama 3.2 model for 100 parallel users in good conditions with llamacpp under windows or linux.
What GPU do you use for that ? 5090 with 24gb ? what CPU (intel core 9 or Amd Epyc ?) how much RAM (128 gb, 256 gb ?)
Do you think with all these, speed of token generation will be enough for all my 100 max users ?

[ExtReMLapin]

No, to some degree batched inference increases total t/s your GPU can produce but individual prompt speeds shuts down because of this bug : #10860

For efficient production serving 100 users, VLLM is closer to the 'production readiness' you're looking for