In order to see what are the GPU's available
Sometimes nvidia-smi show that a GPU is available, while it is inaccessible. It is most of the time due to GPU process not killed properly. You can list the processes related to the GPU's by typing
Then you can kill the process bonded to the GPU.
PS: Sometimes you have to repeat this command to ensure that the process has been completely terminated.
If the server have 4 GPUs [0,1,2,3]
CUDA_VISIBLE_DEVICES=GPU_number python myprogram.py
CUDA_VISIBLE_DEVICES=1 python myprogram.py
If no CUDA_VISIBLE_DEVICES is specified GPU 0 will be used by default.