In order to see available GPU
Process related to GPU¶
Sometimes nvidia-smi show that a GPU is available, while it is inaccessible. It is most of the time due to GPU process not killed properly. You can list the processes related to the GPU's by typing
Then you can kill the process bonded to the GPU.
PS: Sometimes you have to repeat this command to ensure that the process has been completely terminated.
Limit the number of GPU used¶
If the server have 4 GPUs [0,1,2,3]
CUDA_VISIBLE_DEVICES=GPU_number python myprogram.py
CUDA_VISIBLE_DEVICES=1 python myprogram.py
Depending on the program, if you don't specify it, it could use all the GPU which is usually not what we want. The other option is that if
CUDA_VISIBLE_DEVICES isn't specified GPU 0 will be used by default and that GPU might be busy while others are free.