pytorch在同一臺服務器上同時訓練多個任務

使用pytoch在同一臺服務器上同時訓練多個任務時會出現以下問題：

File "./torch-dist/mnist-dist.py", line 201, in <module>
    init_processes(args.rank, args.world_size, run, args.batch_size, backend=args.backend)
  File "./torch-dist/mnist-dist.py", line 196, in init_processes
    dist.init_process_group(backend=backend, world_size=world_size, rank=rank, init_method="env://")
  File "/home/dl/anaconda2/envs/torch-dist-py3.6/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 354, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/home/dl/anaconda2/envs/torch-dist-py3.6/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, start_daemon)
RuntimeError: Address already in use

解決辦法：
try specifying a different master_addr and master_port in torch.distributed.launch

CUDA_VISIBLE_DEVICES=${GPU_ID} python -m torch.distributed.launch --nproc_per_node=$NGPUS --master_addr 127.0.0.2 --master_port 29501 tools/train_net.py 

ref: https://github.com/facebookresearch/maskrcnn-benchmark/issues/241