docker的联合文件系统 UnionFS《深入docker底层原理》

小鱼儿 2024-03-17 14:48 105阅读 0赞

## 介绍 ##

docker最大的贡献就是定义了容器镜像的分层的存储格式，docker镜像技术的基础是联合文件系统(UnionFS)，其文件系统是分层的。这样既可以充分利用共享层，又可以减少存储空间占用。

联合挂载系统的工作原理：读：如果文件在upperdir(容器)层，直接读取文件；如果文件不在upperdir(容器)层，则从镜像层(lowerdir)读取。

目前docker支持的联合文件系统有很多种，包括：AUFS、overlay、overlay2、DeviceMapper、VSF等。

当选择Docker的存储驱动时，有几个常用的选项可供选择，包括AUFS, Overlay, Overlay2, DeviceMapper和VFS。下面是它们的对比：

1.  AUFS (Advanced Multi-Layered Unification Filesystem)：AUFS是一种基于union mount技术的存储驱动，它允许将多个文件系统合并为一个只读的虚拟文件系统。AUFS在多个容器之间共享基础镜像的文件，并在每个容器中提供一个可写的层。然而，AUFS对于大型镜像和多层容器的性能可能有一些负面影响。此外，AUFS在最新的Linux内核版本中已不再被广泛支持。
2.  Overlay：Overlay是一种较新的存储驱动，它通过在底层文件系统上创建一个只读的基础镜像并叠加一个可写的层来实现容器的文件系统。Overlay相对于AUFS有更好的性能，而且它在最新的Linux内核中得到了更广泛的支持。然而，Overlay的缺点是不支持在同一个文件中存储大型镜像。
3.  Overlay2：Overlay2是Overlay的改进版，它在Overlay的基础上解决了一些性能和功能的问题。Overlay2使用一个单独的只读层来存储基础镜像，并且可以支持更大的镜像。它是Docker默认的存储驱动，因为它在大多数情况下都提供了更好的性能和可靠性。
4.  DeviceMapper：DeviceMapper是一种成熟的Linux存储驱动，它利用LVM（逻辑卷管理器）提供了高级功能，如快照和克隆。DeviceMapper是Docker支持的可插拔的存储后端之一，并被广泛用于可靠和复杂的存储需求。
5.  VFS (Virtual File System)：VFS是Docker的默认存储驱动，它直接在宿主机上操作文件。VFS基于操作系统的本地文件系统，但在性能和可扩展性方面不如其他驱动。它适用于开发和测试环境，但不适合生产环境。

综上所述，当选择Docker存储驱动时，Overlay2是最常见和推荐的选择，因为它提供了较好的性能和可靠性。另外，如果需要高级功能和复杂的存储需求，可以考虑使用DeviceMapper。

查看docker容器使用的文件系统使用的命令如下，其中Storage Driver: overlay2代表使用的是overlay2联合文件系统。

[root@home]# docker info
    Client:
     Context:    default
     Debug Mode: false
     Plugins:
      app: Docker App (Docker Inc., v0.9.1-beta3)
      buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
      scan: Docker Scan (Docker Inc., v0.21.0)
    
    Server:
     Containers: 2
      Running: 1
      Paused: 0
      Stopped: 1
     Images: 17
     Server Version: 20.10.21
     Storage Driver: overlay2
      Backing Filesystem: xfs
      Supports d_type: true
      Native Overlay Diff: true
      userxattr: false
     Logging Driver: json-file
     Cgroup Driver: cgroupfs
     Cgroup Version: 1
     Plugins:
      Volume: local
      Network: bridge host ipvlan macvlan null overlay
      Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
     Swarm: inactive
     Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
     Default Runtime: runc
     Init Binary: docker-init
     containerd version: 1c90a442489720eec95342e1789ee8a5e1b9536f
     runc version: v1.1.4-0-g5fd4c4d
     init version: de40ad0
     Security Options:
      seccomp
       Profile: default
     Kernel Version: 3.10.0-957.el7.x86_64
     Operating System: CentOS Linux 7 (Core)
     OSType: linux
     Architecture: x86_64
     CPUs: 2
     Total Memory: 3.701GiB
     Name: 10-6-9-59
     ID: XISV:HHG5:YV5H:AQIM:WE4G:6IXJ:2N2Q:SZOM:XBCP:BTGB:HI7P:4W7O
     Docker Root Dir: /var/lib/docker
     Debug Mode: false
     Registry: https://index.docker.io/v1/
     Labels:
     Experimental: false
     Insecure Registries:
      harbor.lys.site
     Registry Mirrors:
      https://harbor.lys.site/
      https://vre6wzor.mirror.aliyuncs.com/
     Live Restore Enabled: false

## 修改 ##

修改对应文件系统，可以通过/etc/docker/daemon.json文件的 “storage-driver”:参数

[root@home]# cat /etc/docker/daemon.json 
    {
      "registry-mirrors": ["https://vre6wzor.mirror.aliyuncs.com"],
      "exec-opts": ["native.cgroupdriver=systemd"],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "100m"
      },
      "storage-driver": "overlay2",
      "storage-opts": [
        "overlay2.override_kernel_check=true"
      ]
    }

## 原理实践 ##

### overlay2 ###

![在这里插入图片描述][375cb8353e204480bf17b16fb82fdc1e.png]  
如上图所示，OverlayFS将单个Linux主机上的两个目录合并成一个目录，这些目录被称为层，统一过程被称为联合挂载。OverlayFS关联的底层目录称为lowerdir，lowerdir是只读的镜像层(image layer)，其中就包含bootfs/rootfs层，bootfs(boot file system)主要包含bootloader和kernel，bootloader主要是引导加载kernel，当boot成功 kernel 被加载到内存中，bootfs就被umount了，rootfs(root file system)包含的就是典型Linux系统中的/dev、/proc、/bin、/etc等标准目录。lowerdir是可以分很多层的，除了bootfs/rootfs层以外，还可以通过Dockerfile建立很多image层。

对应的高层目录upperdir层是lowerdir的上一层，只有这一层可读可写的，其实就是Container层，在启动一个容器的时候会在最后的image层的上一层自动创建，所有对容器数据的更改都会发生在这一层。

联合挂载后merged层就是联合挂载层，也就是给用户暴露的统一视觉，将image层和container层结合，就如最上边的图中描述一致，同一文件，在此层会展示离它最近的层级里的文件内容，或者可以理解为，只要container层中有此文件，便展示container层中的文件内容，若container层中没有，则展示image层中的。

如下容器的整体结构

![在这里插入图片描述][7f915894954d43ad821389b67308817b.png]  
在centos操作系统下，对应联合文件系统overlay2目录是:/var/lib/docker/overlay2,使用docker inspect \[容器ID\]就可以看到这几个层所在的位置。示例

docker inspect 03cf6fafe267
    [
        {
            "Id": "03cf6fafe26723252ba50f559354575c9ee8518af9c94380f2286a7ded7fe5d2",
            "Created": "2023-06-30T10:07:07.228528513Z",
            "Path": "/portainer",
            "Args": [],
            "State": {
                "Status": "running",
                "Running": true,
                "Paused": false,
                "Restarting": false,
                "OOMKilled": false,
                "Dead": false,
                "Pid": 3670,
                "ExitCode": 0,
                "Error": "",
                "StartedAt": "2023-06-30T10:07:07.822465986Z",
                "FinishedAt": "0001-01-01T00:00:00Z"
            },
            "Image": "sha256:580c0e4e98b06d258754cf28c55f21a6fa0dc386e6fe0bf67e453c3642de9b8b",
            "ResolvConfPath": "/var/lib/docker/containers/03cf6fafe26723252ba50f559354575c9ee8518af9c94380f2286a7ded7fe5d2/resolv.conf",
            "HostnamePath": "/var/lib/docker/containers/03cf6fafe26723252ba50f559354575c9ee8518af9c94380f2286a7ded7fe5d2/hostname",
            "HostsPath": "/var/lib/docker/containers/03cf6fafe26723252ba50f559354575c9ee8518af9c94380f2286a7ded7fe5d2/hosts",
            "LogPath": "/var/lib/docker/containers/03cf6fafe26723252ba50f559354575c9ee8518af9c94380f2286a7ded7fe5d2/03cf6fafe26723252ba50f559354575c9ee8518af9c94380f2286a7ded7fe5d2-json.log",
            "Name": "/prtainer-test",
            "RestartCount": 0,
            "Driver": "overlay2",
            "Platform": "linux",
            "MountLabel": "",
            "ProcessLabel": "",
            "AppArmorProfile": "",
            "ExecIDs": null,
            "HostConfig": {
                "Binds": [
                    "/var/run/docker.sock:/var/run/docker.sock"
                ],
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "json-file",
                    "Config": {}
                },
                "NetworkMode": "default",
                "PortBindings": {
                    "9000/tcp": [
                        {
                            "HostIp": "",
                            "HostPort": "9000"
                        }
                    ]
                },
                "RestartPolicy": {
                    "Name": "always",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "CapAdd": null,
                "CapDrop": null,
                "CgroupnsMode": "host",
                "Dns": [],
                "DnsOptions": [],
                "DnsSearch": [],
                "ExtraHosts": null,
                "GroupAdd": null,
                "IpcMode": "private",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 67108864,
                "Runtime": "runc",
                "ConsoleSize": [
                    0,
                    0
                ],
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": [],
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": [],
                "DeviceCgroupRules": null,
                "DeviceRequests": null,
                "KernelMemory": 0,
                "KernelMemoryTCP": 0,
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": null,
                "OomKillDisable": false,
                "PidsLimit": null,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": [
                    "/proc/asound",
                    "/proc/acpi",
                    "/proc/kcore",
                    "/proc/keys",
                    "/proc/latency_stats",
                    "/proc/timer_list",
                    "/proc/timer_stats",
                    "/proc/sched_debug",
                    "/proc/scsi",
                    "/sys/firmware"
                ],
                "ReadonlyPaths": [
                    "/proc/bus",
                    "/proc/fs",
                    "/proc/irq",
                    "/proc/sys",
                    "/proc/sysrq-trigger"
                ]
            },
            "GraphDriver": {
                "Data": {
                    "LowerDir": "/var/lib/docker/overlay2/840faba1dd5726eec55029b0da956195e2234853313e3c78e8f3f96e51244401-init/diff:/var/lib/docker/overlay2/0b1f62ad330df70a06a47e85689eedbb1c59f48df4b775c166468286f6aa3198/diff:/var/lib/docker/overlay2/5d83a80e080ae21964392ebfb9868e217bc223f9b5b70018042765b7a9ea3995/diff:/var/lib/docker/overlay2/fce6d44bff996fcd898fb8cc182389be24c9798d2cef20ecf0caa7eac428a316/diff",
                    "MergedDir": "/var/lib/docker/overlay2/840faba1dd5726eec55029b0da956195e2234853313e3c78e8f3f96e51244401/merged",
                    "UpperDir": "/var/lib/docker/overlay2/840faba1dd5726eec55029b0da956195e2234853313e3c78e8f3f96e51244401/diff",
                    "WorkDir": "/var/lib/docker/overlay2/840faba1dd5726eec55029b0da956195e2234853313e3c78e8f3f96e51244401/work"
                },
                "Name": "overlay2"
            },
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/var/run/docker.sock",
                    "Destination": "/var/run/docker.sock",
                    "Mode": "",
                    "RW": true,
                    "Propagation": "rprivate"
                },
                {
                    "Type": "volume",
                    "Name": "e77fbc451ca1b8366c10a52dc30d87aa6b482f78e48c52f8082e607e316128f0",
                    "Source": "/var/lib/docker/volumes/e77fbc451ca1b8366c10a52dc30d87aa6b482f78e48c52f8082e607e316128f0/_data",
                    "Destination": "/data",
                    "Driver": "local",
                    "Mode": "",
                    "RW": true,
                    "Propagation": ""
                }
            ],
            "Config": {
                "Hostname": "03cf6fafe267",
                "Domainname": "",
                "User": "",
                "AttachStdin": false,
                "AttachStdout": false,
                "AttachStderr": false,
                "ExposedPorts": {
                    "9000/tcp": {}
                },
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                ],
                "Cmd": null,
                "Image": "portainer/portainer",
                "Volumes": {
                    "/data": {}
                },
                "WorkingDir": "/",
                "Entrypoint": [
                    "/portainer"
                ],
                "OnBuild": null,
                "Labels": {}
            },
            "NetworkSettings": {
                "Bridge": "",
                "SandboxID": "3a7037ed8c5eee246cc4cee7a525bca0fad7dfc3f3aa455d34e2333c58e460b8",
                "HairpinMode": false,
                "LinkLocalIPv6Address": "",
                "LinkLocalIPv6PrefixLen": 0,
                "Ports": {
                    "9000/tcp": [
                        {
                            "HostIp": "0.0.0.0",
                            "HostPort": "9000"
                        },
                        {
                            "HostIp": "::",
                            "HostPort": "9000"
                        }
                    ]
                },
                "SandboxKey": "/var/run/docker/netns/3a7037ed8c5e",
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "EndpointID": "7eee2f172e17fe50813a1ba82b76a0b32cdd9b8011eb3337ac530d73dcac9b5f",
                "Gateway": "172.17.0.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "172.17.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "MacAddress": "02:42:ac:11:00:02",
                "Networks": {
                    "bridge": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": null,
                        "NetworkID": "5c0d2dd61ef719f9e167485e3f80151c47c60e973d16b8ec40f41367adb4cf91",
                        "EndpointID": "7eee2f172e17fe50813a1ba82b76a0b32cdd9b8011eb3337ac530d73dcac9b5f",
                        "Gateway": "172.17.0.1",
                        "IPAddress": "172.17.0.2",
                        "IPPrefixLen": 16,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "MacAddress": "02:42:ac:11:00:02",
                        "DriverOpts": null
                    }
                }
            }
        }
    ]

### overlay的模拟实践 ###

#### 1.显示已挂载的Overlay文件系统 ####

[root@newhostname to]# mount | grep overlay
    overlay on /var/lib/docker/overlay2/083352f2addd2a15848f6f2742595c7706d74721b688f2665766c2397690febb/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/RFCU37H7W5QLVKZQYWK4WKS2ZR:/var/lib/docker/overlay2/l/A2GRPWBIX7ETOJISQK6HBFAXDR:/var/lib/docker/overlay2/l/722WJHOAIK2L42LBVVRQAKDSA4:/var/lib/docker/overlay2/l/M6XHLB5ZALNKEPMJBDGUAZ5LTG,upperdir=/var/lib/docker/overlay2/083352f2addd2a15848f6f2742595c7706d74721b688f2665766c2397690febb/diff,workdir=/var/lib/docker/overlay2/083352f2addd2a15848f6f2742595c7706d74721b688f2665766c2397690febb/work)

请按照以下步骤进行操作：

#### 2. 创建一个存在的目录作为挂载点： ####

sudo mkdir /path/to/mount/point

#### 3. 使用overlay文件系统挂载： ####

sudo mount -t overlay overlay -o lowerdir=/path/to/lower,upperdir=/path/to/upper,workdir=/path/to/work /path/to/mount/point

请将`/path/to/lower`替换为底层目录的实际路径，将`/path/to/upper`替换为上层目录的实际路径，将`/path/to/work`替换为工作目录的实际路径。

#### 4. 卸载Overlay文件系统时，请使用以下命令： ####

sudo umount /path/to/mount/point

请注意修改命令中的路径以正确适配您的系统配置，确保目录的存在和权限设置正确。

[375cb8353e204480bf17b16fb82fdc1e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/15/fa369b4e5fe543b09f12ea4b9a013025.png
[7f915894954d43ad821389b67308817b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/15/17d6a6b4fbd14b87a508bdc5e0afcd24.png