multi GPU setting

[chesterout]

@Qubitium
I am trying to quantize deepseek bf 16 from modelcloud.
How do we use 2 gpus. I tried three ways,

1. in the code: os.environ["CUDA_VISIBLE_DEVICES"] = "0,1".
2. in the linux command, export CUDA_VISIBLE_DEVICES=0,1.
3. CUDA_VISIBLE_DEVICES=0,1 python code.py

However, none of these works. Can you clarify?

[Qubitium]

Please clarify none of these works? Show me the code and errors.

1. Are you using main branch?
2. You need to install Python 3.13t (free-threading) branch. not the regular Python 3.13 for true multi-gpu support.
3. Your gpus must be on PCIE 4.0 or SXM4+ (best).

[chesterout]

@Qubitium
By none of these works, it means that only one GPU is used and the second one does not load any weight.
In the nvidia-smi, only the first one is working. And finally the command line shows not enough memory.

I am now researching for the python 3.13t.

[Qubitium]

@chesterout All the weights are loaded to GPU0. During quantization, some (not all) weights will be dynamically moved to gpu 1...N and only if you use python 3.13t.