linux 进程 ksoftirqd/n 占用cpu 100%

Love The Way You Lie 2021-09-07 15:40 1324阅读 0赞

1.背景

当来自设备的中断时，操作系统会暂停它正在执行的操作并开始寻址该中断。

在某些情况下，IRQ一个接一个地非常快，操作系统无法在另一个到达之前完成一个服务。当高速网卡在短时间内收到大量数据包时，就会发生这种情况。

因为操作系统在到达时无法处理IRQ（因为它们一个接一个地到达得太快），

操作系统会将它们排队等待稍后由名为ksoftirqd的/n(n为cpu的逻辑号)的内核线程处理。

每个ksoftirqd/n内核线程都运行ksoftirqd()函数，实际上该函数执行下面的循环：

for (;;)
    {
        set_current_state(TASK_INTERRUPTIBLE);
        schedule();
        while (local_softirq_pending())
        {
            preempt_disable();
            do_softirq(); // 处理软中断
            preempt_enable();
            cond_resched();
        }
    }

如果`ksoftirqd`占用的CPU时间超过一小部分，则表示机器处于严重的中断负载下。

2.解决方法

可以用 cat /proc/interrupts 来查看设备造成的中断情况

例如某天我的是这样

CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
      0:         44          0          0          0          0          0          0          0   IO-APIC   2-edge      timer
      1:          0          1          1          0          0          0          0          1   IO-APIC   1-edge      i8042
      8:          1          0          0          0          0          0          0          0   IO-APIC   8-edge      rtc0
      9:          0          0          0          0          0          0          0          0   IO-APIC   9-fasteoi   acpi
     12:          0          0          0          0          2          1          1          0   IO-APIC  12-edge      i8042
     14:         16      35619    2171878      33951    2172032      35710    2172366      35859   IO-APIC  14-edge      ata_piix
     15:          0          0          0          0          0          0          0          0   IO-APIC  15-edge      ata_piix
     16:       1492       1492       1492   38096398       1489       1492       1496       1491   IO-APIC  16-fasteoi   ioc0
     19:         38         38         37         38         37         39         38         38   IO-APIC  19-fasteoi   radeon
     20:          3          3          4          4          5          3          4          4   IO-APIC  20-fasteoi   uhci_hcd:usb3, uhci_hcd:usb5
     21:          4          3          4          3          5          4          4          4   IO-APIC  21-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb4
     24:          0          0          0          0          0          0          0          0   PCI-MSI 32768-edge      PCIe PME
     25:          0          0          0          0          0          0          0          0   PCI-MSI 49152-edge      PCIe PME
     26:          0          0          0          0          0          0          0          0   PCI-MSI 65536-edge      PCIe PME
     27:          0          0          0          0          0          0          0          0   PCI-MSI 81920-edge      PCIe PME
     28:          0          0          0          0          0          0          0          0   PCI-MSI 98304-edge      PCIe PME
     29:          0          0          0          0          0          0          0          0   PCI-MSI 114688-edge      PCIe PME
     30:          0          0          0          0          0          0          0          0   PCI-MSI 458752-edge      PCIe PME
     31:       1380       1346       1353       1339       1339       1357       1329       1360   PCI-MSI 3670016-edge      eno2
     32:   85511967   85513711   85083641   83994544   85084225   85501922   85093691   85515746   PCI-MSI 1572864-edge      eno1
    NMI:     234694     241328     233682     235747     227371     232679     239850     230941   Non-maskable interrupts
    LOC:  809615451  844980307  827224481  872760389  798336595  856256586  824716208  858800687   Local timer interrupts
    SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
    PMI:     234694     241328     233682     235747     227371     232679     239850     230941   Performance monitoring interrupts
    IWI:     234691     241323     233680     235741     227368     232677     239846     230941   IRQ work interrupts
    RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
    RES:   50636771   36959166   32328971   32915351   29242823   29889193   29928230   31502838   Rescheduling interrupts
    CAL:    4443659    7479033    5592654    1507128    4465404    8053047    5214542    1501877   Function call interrupts
    TLB:    6476601    6460251    6517362    6225272    6466684    6345076    6491077    6193616   TLB shootdowns
    TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
    THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
    DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
    MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
    MCP:      22724      22724      22724      22724      22724      22724      22724      22724   Machine check polls
    ERR:          0
    MIS:          0
    PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
    PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event

注意到序号32那爆炸的中断数量，可以断定是网卡瘫痪了。只需要ifdown eno1 && ifup eno1重启网卡即可。