ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)
問題
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)
出現這個錯誤的情況是,在服務器上的docker中運行訓練代碼時,batch size設置得過大,shared memory不夠(因爲docker限制了shm).
根據PyTorch README：
Please note that PyTorch uses shared memory to share data between processes, so if torch multiprocessing is used (e.g. for multithreaded data loaders) the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host or --shm-size command line options to nvidia-docker run.
解決方案
- 這裏說明了,PyTorch的IPC會利用共享內存,所以共享內存必須足夠大,可以通過
docker run --shm-size進行修改 - 通過設置
--ipc=host - 將Dataloader的num_workers設置爲0.但訓練會變慢