pytorch index out of bounds

 

IndexKernel.cu:60: block: [0,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"

 

C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [37,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [3,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [8,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [12,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [15,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [17,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [18,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
THCudaCheck FAIL file=C:/w/b/windows/pytorch/aten/src\THC/THCReduceAll.cuh line=327 error=710 : device-side assert triggered
   0/29999     3.05G    0.1034   0.05052    0.3308    0.4847        10       640:   0%|          | 9/2491 [00:06<31:41,  1.31it/s]
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2019.2.3\helpers\pydev\pydevd.py", line 2073, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm 2019.2.3\helpers\pydev\pydevd.py", line 2067, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm 2019.2.3\helpers\pydev\pydevd.py", line 1418, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "C:\Program Files\JetBrains\PyCharm 2019.2.3\helpers\pydev\pydevd.py", line 1425, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2019.2.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "D:/train.py", line 383, in <module>
    train(hyp)
  File "D:/train.py", line 207, in train
    loss, loss_items = compute_loss(pred, targets.to(device), model)
  File "D:\utils\utils.py", line 492, in compute_loss
    lobj += BCEobj(pi[..., 4], tobj)  # obj loss
  File "D:\Users\Administrator\miniconda3\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\Users\Administrator\miniconda3\lib\site-packages\torch\nn\modules\loss.py", line 617, in forward
    reduction=self.reduction)
  File "D:\Users\Administrator\miniconda3\lib\site-packages\torch\nn\functional.py", line 2435, in binary_cross_entropy_with_logits
    return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
RuntimeError: CUDA error: device-side assert triggered

gpu不能正確定位異常位置，用cpu能定位到異常，索引錯了。

並不是loss那裏報的錯，而是別的地方報的錯,，找到報錯就解決了，超出索引了。