使用 VSCode + qemu 搭建 Linux 内核调试环境

电玩女神 2022-11-01 10:52 663阅读 0赞

Posted on 2019-10-24 by [howardlau][]

目录 \[[隐藏][Link 1]\]

*  [1 编译调试版 Linux 内核][1_ Linux]
    
     *  [1.1 Ubuntu][1.1_Ubuntu]
     *  [1.2 CentOS][1.2_CentOS]
     *  [1.3 Fedora][1.3_Fedora]
 *  [2 制作 initramfs][2_ initramfs]
 *  [3 启动内核并调试][3]
 *  [4 配置 VSCode][4_ VSCode]
    
     *  [4.1 .vscode/c\_cpp\_properties.json][4.1_.vscode_c_cpp_properties.json]
     *  [4.2 .vscode/tasks.json][4.2_.vscode_tasks.json]
     *  [4.3 .vscode/launch.json][4.3_.vscode_launch.json]
     *  [4.4 .vscode/settings.json][4.4_.vscode_settings.json]

## 编译调试版 Linux 内核 ##

首先下载源代码：

wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.3.7.tar.gzCOPY

具体下载哪个版本可以自己选择。

然后解压缩源代码，这里假设解压到了 `~/linux-5.3.7`。

然后安装编译依赖：

### Ubuntu ###

sudo apt-get install build-essential libncurses-dev bison flex libssl-dev libelf-devCOPY

### CentOS ###

sudo yum group install "Development Tools"
    sudo yum install ncurses-devel bison flex elfutils-libelf-devel openssl-develCOPY

### Fedora ###

sudo dnf group install "Development Tools"
    sudo dnf install ncurses-devel bison flex elfutils-libelf-devel openssl-develCOPY

进入源码目录，配置编译选项：

cd ~/linux-5.3.7
    make menuconfigCOPY

比较重要的配置项有：

Kernel hacking  --->
        [*] Kernel debugging
        Compile-time checks and compiler options  --->
            [*] Compile the kernel with debug info
            [*]   Provide GDB scripts for kernel debuggingCOPY

一定要打开。  
还有下面的选项会导致打断点失败，一定要关闭：

Processor type and features ---->
        [] Randomize the address of the kernel image (KASLR)COPY

保存并退出 menuconfig，开始编译之旅：

make -j$(nproc)COPY

编译内核会花掉大概 20GB 的硬盘空间，而且一般需要几十分钟来编译，所以请保证有充足的硬盘空间还有时间。

编译的时候你可以抽空安装一下 qemu：

sudo apt install qemu qemu-system qemu-kvmCOPY

## 制作 initramfs ##

内核启动的时候非常矛盾， boot loader 加载完内核文件 vmlinuz 后，内核紧接着需要挂载磁盘根文件系统，但如果此时内核没有相应驱动，无法识别磁盘，就需要先加载驱动，而驱动又位于 `/lib/modules`，得挂载根文件系统才能读取，这就陷入了一个两难境地，系统无法顺利启动。于是有了 `initramfs` 根文件系统，其中包含必要的设备驱动和工具，boot loader加载 `initramfs` 到内存中，内核会将其挂载到根目录 `/`，然后运行 `/init` 脚本，挂载真正的磁盘根文件系统。

这里借助 BusyBox 构建极简 initramfs，提供基本的用户态可执行程序。

下载 busybox 源代码，解压，假设解压到 `~/busybox-1.31.0`，解压完成后，跟内核一样先配置编译选项：

cd ~/busybox-1.31.0
    make menuconfigCOPY

记得要编译成静态链接的：

Settings  --->
        [*] Build static binary (no shared libs)COPY

然后编译，并且安装到 `_install` 目录：

make -j$(nproc) && make installCOPY

之后制作 initramfs 镜像，需要注意的是里面不包含内核模块，如果需要启用内核模块，则需要手动将 `.ko` 文件拷贝到 `lib/modules` 目录里，然后在 `init` 脚本中 `insmod`：

mkdir initramfs
    cd initramfs
    cp ../_install/* -rf ./
    mkdir dev proc sys
    mkdir -p lib/modules
    mkdir -p etc/init.d
    touch etc/init.d/rcS
    chmod a+x etc/init.d/rcS
    sudo cp -a /dev/{null,console,tty,tty1,tty2,tty3,tty4} dev/
    cp ~/linux-5.3.7/drivers/net/ethernet/intel/e1000/e1000.ko lib/modules
    rm linuxrc
    vim init
    chmod a+x initCOPY

`init` 文件内容如下：

#!/bin/sh
    mount -t proc none /proc
    mount -t sysfs none /sys
    insmod /lib/modules/e1000.ko
    /sbin/mdev -s
    setsid cttyhack /bin/sh
    exec /sbin/initCOPY

最后打包initramfs:

find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../initramfs.cpio.gzCOPY

## 启动内核并调试 ##

执行：

qemu-system-x86_64 -s -S -kernel ~/linux-5.3.7/arch/x86/boot/bzImage -initrd ~/busybox-1.31.0/initramfs.cpio.gz -nographic -append "console=ttyS0" -serial mon:stdio -device e1000,netdev=net0 -netdev user,id=net0,hostfwd=tcp::5555-:22COPY

启动内核以及 GDB Server，其中 -s 选项是 -gdb 的简写，会在本地的 :1234 启动一个 GDB 服务，然后 -S 代表暂停虚拟机，等待 GDB 执行 continue 指令。append 后面是内核的启动选项。

然后切换到内核源码目录，启动 `gdb`，不过在启动之前，请向 `~/linux-5.3.7/.gdbinit` 添加如下内容：

add-auto-load-safe-path ./scripts/gdb/vmlinux-gdb.pyCOPY

来加载内核调试工具，然后执行：

cd ~/linux-5.3.7
    gdb vmlinux
    (gdb) target remote :1234COPY

来连接到虚拟机上的 gdb 服务。

到这里，你就可以像调试普通程序一样调试 Linux 内核了。Linux 的内核入口函数是位于 `init/main.c` 中的 `start_kernel` ，在这里完成各种内核数据结构的初始化。但是这已经是 bootstrap 很后面的过程了。要想看真正的第一行代码，请到 `arch/x86/boot/header.S`

可以通过下面的命令来看看内核调试工具：

(gdb) apropos lx
    function lx_clk_core_lookup -- Find struct clk_core by name
    function lx_current -- Return current task
    function lx_device_find_by_bus_name -- Find struct device by bus and name (both strings)
    function lx_device_find_by_class_name -- Find struct device by class and name (both strings)
    function lx_module -- Find module by name and return the module variable
    function lx_per_cpu -- Return per-cpu variable
    function lx_rb_first -- Lookup and return a node from an RBTree
    function lx_rb_last -- Lookup and return a node from an RBTree
    function lx_rb_next -- Lookup and return a node from an RBTree
    function lx_rb_prev -- Lookup and return a node from an RBTree
    function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
    function lx_thread_info -- Calculate Linux thread_info from task variable
    function lx_thread_info_by_pid -- Calculate Linux thread_info from task variable found by pid
    lx-clk-summary -- Print clk tree summary
    lx-cmdline --  Report the Linux Commandline used in the current kernel
    lx-configdump -- Output kernel config to the filename specified as the command
    lx-cpus -- List CPU status arrays
    lx-device-list-bus -- Print devices on a bus (or all buses if not specified)
    lx-device-list-class -- Print devices in a class (or all classes if not specified)
    lx-device-list-tree -- Print a device and its children recursively
    lx-dmesg -- Print Linux kernel log buffer
    lx-fdtdump -- Output Flattened Device Tree header and dump FDT blob to the filename
    lx-genpd-summary -- Print genpd summary
    lx-iomem -- Identify the IO memory resource locations defined by the kernel
    lx-ioports -- Identify the IO port resource locations defined by the kernel
    lx-list-check -- Verify a list consistency
    lx-lsmod -- List currently loaded modules
    lx-mounts -- Report the VFS mounts of the current process namespace
    lx-ps -- Dump Linux tasks
    lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
    lx-timerlist -- Print /proc/timer_list
    lx-version --  Report the Linux Version of the current kernel
    (gdb) lx-cmdline
    console=ttyS0COPY

我们试下打下断点：

(gdb) break cmdline_proc_show
    Breakpoint 1 at 0xffffffff81353110: file fs/proc/cmdline.c, line 8.
    (gdb) continue
    Continuing.
    
    Breakpoint 1, cmdline_proc_show (m=0xffff8880076ebb80, v=0x1 <fixed_percpu_data+1>) at fs/proc/cmdline.c:8
    8       {
    (gdb) n
    9               seq_puts(m, saved_command_line);
    (gdb) bt
    #0  cmdline_proc_show (m=0xffff8880076ebb80, v=0x1 <fixed_percpu_data+1>) at fs/proc/cmdline.c:9
    #1  0xffffffff812e7d77 in seq_read (file=<optimized out>, buf=<optimized out>, size=<optimized out>, ppos=<optimized out>)
        at fs/seq_file.c:229
    #2  0xffffffff8134a01e in proc_reg_read (file=<optimized out>, buf=<optimized out>, count=<optimized out>,
        ppos=<optimized out>) at fs/proc/inode.c:223
    #3  0xffffffff812bae45 in do_loop_readv_writev (flags=<optimized out>, type=<optimized out>, ppos=<optimized out>,
        iter=<optimized out>, filp=<optimized out>) at fs/read_write.c:714
    #4  do_loop_readv_writev (flags=<optimized out>, type=<optimized out>, ppos=<optimized out>, iter=<optimized out>,
        filp=<optimized out>) at fs/read_write.c:701
    #5  do_iter_read (file=0xffff8880076a5d00, iter=0xffffc900001f3ac8, pos=0xffffc900001f3bf0, flags=<optimized out>)
        at fs/read_write.c:935
    #6  0xffffffff812bd8dd in vfs_readv (file=0xffff8880076a5d00, vec=<optimized out>, vlen=<optimized out>,
        pos=0xffffc900001f3bf0, flags=0) at fs/read_write.c:997
    #7  0xffffffff812f65b1 in kernel_readv (offset=<optimized out>, vlen=<optimized out>, vec=<optimized out>,
        file=<optimized out>) at fs/splice.c:359
    #8  default_file_splice_read (in=<optimized out>, ppos=0xffffc900001f3dd0, pipe=<optimized out>, len=<optimized out>,
        flags=<optimized out>) at fs/splice.c:414
    #9  0xffffffff812f43a9 in do_splice_to (in=0xffff8880076a5d00, ppos=0xffffc900001f3dd0, pipe=0xffff88800611c180,
        len=<optimized out>, flags=0) at fs/splice.c:877
    #10 0xffffffff812f4491 in splice_direct_to_actor (in=<optimized out>, sd=0x1 <fixed_percpu_data+1>, actor=<optimized out>)
        at fs/splice.c:954
    #11 0xffffffff812f4688 in do_splice_direct (in=0xffff8880076a5d00, ppos=0xffffc900001f3ea8, out=<optimized out>,
        opos=<optimized out>, len=<optimized out>, flags=<optimized out>) at fs/splice.c:1063
    #12 0xffffffff812bbf12 in do_sendfile (out_fd=<optimized out>, in_fd=<optimized out>, ppos=0x0 <fixed_percpu_data>,
        count=<optimized out>, max=<optimized out>) at fs/read_write.c:1464
    #13 0xffffffff812bc5d6 in __do_sys_sendfile64 (count=<optimized out>, offset=<optimized out>, in_fd=<optimized out>,
        out_fd=<optimized out>) at fs/read_write.c:1525
    #14 __se_sys_sendfile64 (count=<optimized out>, offset=<optimized out>, in_fd=<optimized out>, out_fd=<optimized out>)
        at fs/read_write.c:1511
    #15 __x64_sys_sendfile64 (regs=<optimized out>) at fs/read_write.c:1511
    #16 0xffffffff810043aa in do_syscall_64 (nr=<optimized out>, regs=0x1 <fixed_percpu_data+1>) at arch/x86/entry/common.c:296
    #17 0xffffffff81c0008c in entry_SYSCALL_64 () at arch/x86/entry/entry_64.S:175
    #18 0x0000000000000000 in ?? ()
    (gdb)COPY

在执行 continue 之后，到 qemu 中执行 `cat /proc/cmdline` ，就可以看到断点被触发了。这时候可以进行堆栈打印，或者是其他操作。不过，神奇的是，内核强制打开了 -O2 选项编译，而且不能关闭，因为有很多代码依赖 -O2 优化特性！所以，你会看到很多变量的值被优化掉了。。

gdb 中调用函数的方法是 `$func_name(params)`

设置变量的方法是 `set $var = xxx：`

(gdb) p $lx_current().pid
    $1 = 176
    (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
    (gdb) p *$container_of($next, "struct hrtimer", "node")
    $3 = {node = {node = {__rb_parent_color = 18446612682195984864, rb_right = 0x0 <fixed_percpu_data>,
          rb_left = 0x0 <fixed_percpu_data>}, expires = 9316000000}, _softexpires = 9316000000,
      function = 0xffffffff8112f4f0 <tick_sched_timer>, base = 0xffff88800781da80, state = 1 '01', is_rel = 0 '00',
      is_soft = 0 '00'}COPY

## 配置 VSCode ##

由于 Linux 内核高度定制化，所以没有办法直接通过配置 `includePath` 等让 Intellisense 正常提示，这里借助一个 Python 脚本来生成 `compile_commands.json` 文件帮助 Intellisense 正常提示（包括头文件和宏定义等）

from __future__ import print_function, division
    
    import fnmatch
    import json
    import math
    import multiprocessing
    import os
    import re
    import sys
    
    CMD_VAR_RE = re.compile(r'^s*cmd_(S+)s*:=s*(.+)s*$', re.MULTILINE)
    SOURCE_VAR_RE = re.compile(r'^s*source_(S+)s*:=s*(.+)s*$', re.MULTILINE)
    
    directory = os.path.abspath(os.getcwd())
    
    def print_progress_bar(progress):
        progress_bar = '[' + '|' * int(50 * progress) + '-' * int(50 * (1.0 - progress)) + ']'
        print('r', progress_bar, "{0:.1%}".format(progress), end='r', file=sys.stderr)
    
    def parse_cmd_file(cmdfile_path):
        with open(cmdfile_path, 'r') as cmdfile:
            cmdfile_content = cmdfile.read()
    
        commands = { match.group(1): match.group(2) for match in CMD_VAR_RE.finditer(cmdfile_content) }
        sources = { match.group(1): match.group(2) for match in SOURCE_VAR_RE.finditer(cmdfile_content) }
    
        return [{
                'directory': directory,
                'command': commands[o_file_name],
                'file': source,
                'output': o_file_name
            } for o_file_name, source in sources.items()]
    
    def main():
        print("Building *.o.cmd file list...", file=sys.stderr)
    
        cmd_files = []
        for cur_dir, subdir, files in os.walk(directory):
            cmd_files.extend(os.path.join(cur_dir, cmdfile_name) for cmdfile_name in fnmatch.filter(files, '*.o.cmd'))
    
        print("Parsing *.o.cmd files...", file=sys.stderr)
    
        n_processed = 0
        print_progress_bar(0)
    
        compdb = []
        pool = multiprocessing.Pool()
        try:
            for compdb_chunk in pool.imap_unordered(parse_cmd_file, cmd_files, chunksize=int(math.sqrt(len(cmd_files)))):
                compdb.extend(compdb_chunk)
                n_processed += 1
                print_progress_bar(n_processed / len(cmd_files))
    
        finally:
            pool.terminate()
            pool.join()
    
        print(file=sys.stderr)
        print("Writing compile_commands.json...", file=sys.stderr)
        with open('compile_commands.json', 'w') as compdb_file:
            json.dump(compdb, compdb_file, indent=1)
    
    if __name__ == '__main__':
        main()COPY

将这个 Python 脚本保存成 `gen_commands.py` 之后在 Linux 源代码目录下直接运行就可以生成 `compile_commands.json` 了。

最后分别配置 VSCode 配置文件：

### .vscode/c\_cpp\_properties.json ###

{
        "configurations": [
            {
                "name": "Linux",
                "cStandard": "c11",
                "intelliSenseMode": "gcc-x64",
                "compileCommands": "${workspaceFolder}/compile_commands.json"
            }
        ],
        "version": 4
    }COPY

### .vscode/tasks.json ###

{
        // See https://go.microsoft.com/fwlink/?LinkId=733558
        // for the documentation about the tasks.json format
        "version": "2.0.0",
        "tasks": [
          {
            "label": "vm",
            "type": "shell",
            "command": "qemu-system-x86_64 -s -S -kernel ~/linux-5.3.7/arch/x86/boot/bzImage -initrd ~/busybox-1.31.0/initramfs.cpio.gz -nographic -append "console=ttyS0" -serial mon:stdio -device e1000,netdev=net0 -netdev user,id=net0,hostfwd=tcp::5555-:22",
            "presentation": {
              "echo": true,
              "clear": true,
              "group": "vm"
            },
            "isBackground": true,
            "problemMatcher": [
              {
                "pattern": [
                  {
                    "regexp": ".",
                    "file": 1,
                    "location": 2,
                    "message": 3
                  }
                ],
                "background": {
                  "activeOnStart": true,
                  "beginsPattern": ".",
                  "endsPattern": ".",
                }
              }
            ]
          },
          {
            "label": "build",
            "type": "shell",
            "command": "make",
            "group": {
              "kind": "build",
              "isDefault": true
            },
            "presentation": {
              "echo": false,
              "group": "build"
            }
          }
        ]
      }COPY

### .vscode/launch.json ###

{
        // Use IntelliSense to learn about possible attributes.
        // Hover to view descriptions of existing attributes.
        // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
        "version": "0.2.0",
        "configurations": [
          {
            "name": "(gdb) linux",
            "type": "cppdbg",
            "request": "launch",
            "preLaunchTask": "vm",
            "program": "${workspaceRoot}/vmlinux",
            "miDebuggerServerAddress": "localhost:1234",
            "args": [],
            "stopAtEntry": true,
            "cwd": "${workspaceFolder}",
            "environment": [],
            "externalConsole": false,
            "MIMode": "gdb",
            "miDebuggerArgs": "-n",
            "targetArchitecture": "x64",
            "setupCommands": [
              {
                "text": "set arch i386:x86-64:intel",
                "ignoreFailures": false
              },
              {
                "text": "dir .",
                "ignoreFailures": false
              },
              {
                "text": "add-auto-load-safe-path ./",
                "ignoreFailures": false
              },
              {
                "text": "-enable-pretty-printing",
                "ignoreFailures": true
              }
            ]
          }
        ]
      }COPY

### .vscode/settings.json ###

{
        "files.exclude": {
            "**/.*.*.cmd": true,
            "**/.*.d": true,
            "**/.*.S": true
        },
        "[c]": {
            "editor.detectIndentation": false,
            "editor.tabSize": 8,
            "editor.insertSpaces": false
        }
    }COPY

然后就可以到处打断点，按 F5 开始欢快地调试了~

[howardlau]: https://howardlau.me/author/howardlau
[Link 1]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#
[1_ Linux]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#_Linux
[1.1_Ubuntu]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#Ubuntu
[1.2_CentOS]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#CentOS
[1.3_Fedora]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#Fedora
[2_ initramfs]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#_initramfs
[3]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#i
[4_ VSCode]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#_VSCode
[4.1_.vscode_c_cpp_properties.json]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#vscodec_cpp_propertiesjson
[4.2_.vscode_tasks.json]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#vscodetasksjson
[4.3_.vscode_launch.json]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#vscodelaunchjson
[4.4_.vscode_settings.json]: https://howardlau.me/programming/debugging-linux-kernel-with-vscode-qemu.html#vscodesettingsjson