NVIDIA GeForce RTX 5090 Support

[aliasaria]

NVIDIA 5090's require an updated PyTorch to get CUDA 12.8. This issue is to track the work required to make sure we work on 5090's.

Transformer Lab currently installs CUDA 12.1 using conda in install.sh https://github.com/transformerlab/transformerlab-api/blob/main/install.sh

More documentation:

https://docs.salad.com/tutorials/pytorch-rtx5090

pytorch/pytorch#145949

https://forums.developer.nvidia.com/t/software-migration-guide-for-nvidia-blackwell-rtx-gpus-a-guide-to-cuda-12-8-pytorch-tensorrt-and-llama-cpp/321330

We need to investigate:

• Upgrading CUDA for everyone -- will any existing plugins break? Certain tools like vLLM, FlashAttention2 etc are known to be weird with specific CUDA versions
• Upgrading CUDA to 12.1 just for 5090 users -- how would we do this? What will work and what wont?

[aliasaria]

From vLLM docs:

https://docs.vllm.ai/en/stable/getting_started/installation/gpu.html

As of now, vLLM’s binaries are compiled with CUDA 12.4 and public PyTorch release versions by default. We also provide vLLM binaries compiled with CUDA 12.1, 11.8, and public PyTorch release versions

vLLM is not a required part of our platform (it is only supported in a specific plugin) but it would be nice if we could still support it.

[aliasaria]

From FlashAttention2
https://github.com/Dao-AILab/flash-attention

We highly recommend CUDA 12.8 for best performance. (when discussing H100 / H800 GPU but probably applies to all GPUs)