In order to see what are the GPU's available

nvidia-smi

Sometimes nvidia-smi show that a GPU is available, while it is inaccessible. It is most of the time due to GPU process not killed properly. You can list the processes related to the GPU's by typing

lsof /dev/nvidia*

Then you can kill the process bonded to the GPU.

kill ProcessPIDNumber

PS: Sometimes you have to repeat this command to ensure that the process has been completely terminated.

If the server have 4 GPUs [0,1,2,3]

CUDA_VISIBLE_DEVICES=GPU_number python myprogram.py

ex

CUDA_VISIBLE_DEVICES=1 python myprogram.py 

If no CUDA_VISIBLE_DEVICES is specified GPU 0 will be used by default.