[debug] RuntimeError: CUDA error: no kernel image is available for execution on the device

野性酷女 2022-10-18 15:54 261阅读 0赞

## 问题描述 ##

运行程序时出现报错：

RuntimeError: CUDA error: no kernel image is available for execution on the device (launch_kernel at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/ATen/native/cuda/Loops.cuh:72)

## 问题原因 ##

当前使用的硬件算力不支持相应的pytorch函数，需要更换pytorch到一个合适的版本。

## 解决方法 ##

1.  查看当前设备的算力  
    在[官网][Link 1]查看即可
2.  查看当前pytorch版本支持的算力
    
        torch.cuda.get_arch_list()
    
        Python 3.7.10 (default, Jun  4 2021, 14:48:32) 
        [GCC 7.5.0] :: Anaconda, Inc. on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> import torch
        >>> torch.cuda.get_arch_list()
        ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75']
3.  选择合适的pytorch版本，支持当前的GPU算力

## 附录：完整报错信息 ##

Traceback (most recent call last):
      File "work/main.py", line 171, in <module>
        alg(args.input_dir, args.output_dir,args)
      File "work/main.py", line 127, in alg
        outputs = single_gpu_test(model, data_loader,save_path =split_result )
      File "/workspace/work/tools/TianZhi_test.py", line 29, in single_gpu_test
        result = model(return_loss=False, rescale=not show, **data)
      File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
        return self.module(*inputs[0], **kwargs[0])
      File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "/workspace/work/mmdet/core/fp16/decorators.py", line 49, in new_func
        return old_func(*args, **kwargs)
      File "/workspace/work/mmdet/models/detectors/base.py", line 88, in forward
        return self.forward_test(img, img_meta, **kwargs)
      File "/workspace/work/mmdet/models/detectors/base.py", line 79, in forward_test
        return self.simple_test(imgs[0], img_metas[0], **kwargs)
      File "/workspace/work/mmdet/models/detectors/polarmask.py", line 79, in simple_test
        bbox_list = self.bbox_head.get_bboxes(*bbox_inputs)
      File "/workspace/work/mmdet/core/fp16/decorators.py", line 127, in new_func
        return old_func(*args, **kwargs)
      File "/workspace/work/mmdet/models/anchor_heads/polarmask_last_head.py", line 603, in get_bboxes
        scale_factor, cfg, rescale)
      File "/workspace/work/mmdet/models/anchor_heads/polarmask_last_head.py", line 691, in get_bboxes_single
        cfg.max_per_img,)
      File "/workspace/work/mmdet/core/post_processing/bbox_nms.py", line 111, in multiclass_nms_with_mask
        cls_dets, index = nms_op(cls_dets, **nms_cfg_)
      File "/workspace/work/mmdet/ops/nms/nms_wrapper.py", line 43, in nms
        inds = nms_cuda.nms(dets_th, iou_thr)
    RuntimeError: CUDA error: no kernel image is available for execution on the device (launch_kernel at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/ATen/native/cuda/Loops.cuh:72)
    frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7f38c8b12dc5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
    frame #1: void at::native::gpu_index_kernel<__nv_dl_wrapper_t<__nv_dl_tag<void (*)(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>), &(void at::native::index_kernel_impl<at::native::OpaqueType<8> >(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>)), 1u>> >(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>, __nv_dl_wrapper_t<__nv_dl_tag<void (*)(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>), &(void at::native::index_kernel_impl<at::native::OpaqueType<8> >(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>)), 1u>> const&) + 0x33e (0x7f38ce75013e in /opt/conda/lib/python3.7/site-packages/torch/lib/libcaffe2_gpu.so)
    frame #2: <unknown function> + 0x27e9cca (0x7f38ce74bcca in /opt/conda/lib/python3.7/site-packages/torch/lib/libcaffe2_gpu.so)
    frame #3: <unknown function> + 0x27ea555 (0x7f38ce74c555 in /opt/conda/lib/python3.7/site-packages/torch/lib/libcaffe2_gpu.so)
    frame #4: <unknown function> + 0x6cb2aa (0x7f38c93f72aa in /opt/conda/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
    frame #5: at::native::index(at::Tensor const&, c10::ArrayRef<at::Tensor>) + 0x3e8 (0x7f38c93f4ed8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
    frame #6: at::TypeDefault::index(at::Tensor const&, c10::ArrayRef<at::Tensor>) const + 0x6c (0x7f38c97beb0c in /opt/conda/lib/python3.7/site-packages/torch/lib/libcaffe2.so)
    frame #7: torch::autograd::VariableType::index(at::Tensor const&, c10::ArrayRef<at::Tensor>) const + 0x6f9 (0x7f38c5ec5289 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so.1)
    frame #8: at::Tensor::index(c10::ArrayRef<at::Tensor>) const + 0x59 (0x7f3849a5599d in /workspace/work/mmdet/ops/nms/nms_cuda.cpython-37m-x86_64-linux-gnu.so)
    frame #9: nms_cuda(at::Tensor, float) + 0x779 (0x7f3849a5414d in /workspace/work/mmdet/ops/nms/nms_cuda.cpython-37m-x86_64-linux-gnu.so)
    frame #10: nms(at::Tensor const&, float) + 0x130 (0x7f3849a46e10 in /workspace/work/mmdet/ops/nms/nms_cuda.cpython-37m-x86_64-linux-gnu.so)
    frame #11: <unknown function> + 0x2084d (0x7f3849a5284d in /workspace/work/mmdet/ops/nms/nms_cuda.cpython-37m-x86_64-linux-gnu.so)
    frame #12: <unknown function> + 0x1df0a (0x7f3849a4ff0a in /workspace/work/mmdet/ops/nms/nms_cuda.cpython-37m-x86_64-linux-gnu.so)
    <omitting python frames>

[Link 1]: https://developer.nvidia.com/zh-cn/cuda-gpus