CUDA Error: out of memory darknet: ./src/cuda.c:36: check_error: Assertion `0' failed. Aborted

Love The Way You Lie 2021-11-19 13:38 422阅读 0赞

CUDA Error: out of memory  
darknet: ./src/cuda.c:36: check\_error: Assertion \`0' failed.  
Aborted (core dumped)

原因1：你的GPU被占用了，电脑在跑其他的程序

解决方法：等现在的程序跑完再开始训练

原因2：需要修改所使用的模型cfg文件中的subdivision的参数。

解决方法：由subdivisions=8改成subdivisions=16，64等。

cfg 文件夹下的 yolov3-voc.cfg文件部分内容，把subdivisions=8改成subdivisions==64，程序即可正常：

[net]
    # Testing
    # batch=1
    # subdivisions=1
    # Training
    batch=64
    subdivisions=8
    width=416
    height=416
    channels=3
    momentum=0.9
    decay=0.0005
    angle=0
    saturation = 1.5
    exposure = 1.5
    hue=.1
    
    learning_rate=0.001
    burn_in=1000
    max_batches = 50200
    policy=steps
    steps=40000,45000
    scales=.1,.1

xiugaihou

wu@wu-X555LF:~/darknet$ ./darknet detector train cfg/voc.data cfg/yolov3-voc.cfg darknet53.conv.74
    yolov3-voc
    layer     filters    size              input                output
        0 conv     32  3 x 3 / 1   416 x 416 x   3   ->   416 x 416 x  32  0.299 BFLOPs
        1 conv     64  3 x 3 / 2   416 x 416 x  32   ->   208 x 208 x  64  1.595 BFLOPs
        2 conv     32  1 x 1 / 1   208 x 208 x  64   ->   208 x 208 x  32  0.177 BFLOPs
        3 conv     64  3 x 3 / 1   208 x 208 x  32   ->   208 x 208 x  64  1.595 BFLOPs
        4 res    1                 208 x 208 x  64   ->   208 x 208 x  64
        5 conv    128  3 x 3 / 2   208 x 208 x  64   ->   104 x 104 x 128  1.595 BFLOPs
        6 conv     64  1 x 1 / 1   104 x 104 x 128   ->   104 x 104 x  64  0.177 BFLOPs
        7 conv    128  3 x 3 / 1   104 x 104 x  64   ->   104 x 104 x 128  1.595 BFLOPs
        8 res    5                 104 x 104 x 128   ->   104 x 104 x 128
        9 conv     64  1 x 1 / 1   104 x 104 x 128   ->   104 x 104 x  64  0.177 BFLOPs
       10 conv    128  3 x 3 / 1   104 x 104 x  64   ->   104 x 104 x 128  1.595 BFLOPs
       11 res    8                 104 x 104 x 128   ->   104 x 104 x 128
       12 conv    256  3 x 3 / 2   104 x 104 x 128   ->    52 x  52 x 256  1.595 BFLOPs
       13 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
       14 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
       15 res   12                  52 x  52 x 256   ->    52 x  52 x 256
       16 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
       17 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
       18 res   15                  52 x  52 x 256   ->    52 x  52 x 256
       19 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
       20 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
       21 res   18                  52 x  52 x 256   ->    52 x  52 x 256
       22 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
       23 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
       24 res   21                  52 x  52 x 256   ->    52 x  52 x 256
       25 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
       26 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
       27 res   24                  52 x  52 x 256   ->    52 x  52 x 256
       28 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
       29 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
       30 res   27                  52 x  52 x 256   ->    52 x  52 x 256
       31 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
       32 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
       33 res   30                  52 x  52 x 256   ->    52 x  52 x 256
       34 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
       35 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
       36 res   33                  52 x  52 x 256   ->    52 x  52 x 256
       37 conv    512  3 x 3 / 2    52 x  52 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       38 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       39 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       40 res   37                  26 x  26 x 512   ->    26 x  26 x 512
       41 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       42 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       43 res   40                  26 x  26 x 512   ->    26 x  26 x 512
       44 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       45 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       46 res   43                  26 x  26 x 512   ->    26 x  26 x 512
       47 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       48 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       49 res   46                  26 x  26 x 512   ->    26 x  26 x 512
       50 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       51 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       52 res   49                  26 x  26 x 512   ->    26 x  26 x 512
       53 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       54 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       55 res   52                  26 x  26 x 512   ->    26 x  26 x 512
       56 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       57 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       58 res   55                  26 x  26 x 512   ->    26 x  26 x 512
       59 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       60 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       61 res   58                  26 x  26 x 512   ->    26 x  26 x 512
       62 conv   1024  3 x 3 / 2    26 x  26 x 512   ->    13 x  13 x1024  1.595 BFLOPs
       63 conv    512  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 512  0.177 BFLOPs
       64 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024  1.595 BFLOPs
       65 res   62                  13 x  13 x1024   ->    13 x  13 x1024
       66 conv    512  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 512  0.177 BFLOPs
       67 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024  1.595 BFLOPs
       68 res   65                  13 x  13 x1024   ->    13 x  13 x1024
       69 conv    512  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 512  0.177 BFLOPs
       70 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024  1.595 BFLOPs
       71 res   68                  13 x  13 x1024   ->    13 x  13 x1024
       72 conv    512  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 512  0.177 BFLOPs
       73 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024  1.595 BFLOPs
       74 res   71                  13 x  13 x1024   ->    13 x  13 x1024
       75 conv    512  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 512  0.177 BFLOPs
       76 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024  1.595 BFLOPs
       77 conv    512  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 512  0.177 BFLOPs
       78 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024  1.595 BFLOPs
       79 conv    512  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 512  0.177 BFLOPs
       80 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024  1.595 BFLOPs
       81 conv     75  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x  75  0.026 BFLOPs
       82 yolo
       83 route  79
       84 conv    256  1 x 1 / 1    13 x  13 x 512   ->    13 x  13 x 256  0.044 BFLOPs
       85 upsample            2x    13 x  13 x 256   ->    26 x  26 x 256
       86 route  85 61
       87 conv    256  1 x 1 / 1    26 x  26 x 768   ->    26 x  26 x 256  0.266 BFLOPs
       88 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       89 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       90 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       91 conv    256  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x 256  0.177 BFLOPs
       92 conv    512  3 x 3 / 1    26 x  26 x 256   ->    26 x  26 x 512  1.595 BFLOPs
       93 conv     75  1 x 1 / 1    26 x  26 x 512   ->    26 x  26 x  75  0.052 BFLOPs
       94 yolo
       95 route  91
       96 conv    128  1 x 1 / 1    26 x  26 x 256   ->    26 x  26 x 128  0.044 BFLOPs
       97 upsample            2x    26 x  26 x 128   ->    52 x  52 x 128
       98 route  97 36
       99 conv    128  1 x 1 / 1    52 x  52 x 384   ->    52 x  52 x 128  0.266 BFLOPs
      100 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
      101 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
      102 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
      103 conv    128  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x 128  0.177 BFLOPs
      104 conv    256  3 x 3 / 1    52 x  52 x 128   ->    52 x  52 x 256  1.595 BFLOPs
      105 conv     75  1 x 1 / 1    52 x  52 x 256   ->    52 x  52 x  75  0.104 BFLOPs
      106 yolo
    Loading weights from darknet53.conv.74...Done!
    Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005
    Loaded: 0.405206 seconds
    Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.614804, .5R: -nan, .75R: -nan,  count: 0
    Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.512939, .5R: -nan, .75R: -nan,  count: 0
    Region 106 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.467721, .5R: -nan, .75R: -nan,  count: 0
    Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.615213, .5R: -nan, .75R: -nan,  count: 0
    Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.513037, .5R: -nan, .75R: -nan,  count: 0