calico网络学习一(容器和calico的veth pair通讯过程)

傷城~ 2022-05-24 06:05 446阅读 0赞

## 容器和calico的veth pair通讯过程 ##

本文章分析的前提是在calico部署在k8s上，并开启了ipip的功能

root@test-master-113 ~]# calicoctl get ippool -oyaml
    - apiVersion: v1
      kind: ipPool
      metadata:
        cidr: 192.168.0.0/16
      spec:
        ipip:
          enabled: true
        nat-outgoing: true

每一个容器通过calico cni-plugin 插件设置默认的路由，都会在指定的network-ns创建veth-pair,位于容器内的veth将会被设置成`169.254.1.1`

# ip route
    default via 169.254.1.1 dev eth0
    169.254.1.1 dev eth0  scope link

容器内默认路由的mac地址被设置成主机节点上的某个一`cali`开头的网卡

# arp
    Address                  HWtype  HWaddress           Flags Mask            Iface
    169.254.1.1              ether   4e:e1:53:f4:e0:06   C                     eth0
    
    # ip neigh
    169.254.1.1 dev eth0 lladdr 4e:e1:53:f4:e0:06 REACHABLE

mac地址为`4e:e1:53:f4:e0:06`

cni-plugin创建了endpoint之后，会将其保存到etcd中，felix从而感知到endpoint的变化。felix会在host端设置一条静态arp:

root@test-slave-114 ~]# ip link show cali34d45bc515f
    18704: cali34d45bc515f@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
        link/ether 4e:e1:53:f4:e0:06 brd ff:ff:ff:ff:ff:ff link-netnsid 14

查看容器的网卡信息

[root@test-master-113 ~]# calicoctl get workloadendpoint --workload=tanyanliao.beegotest-3324938723-l2tkm -oyaml
    - apiVersion: v1
      kind: workloadEndpoint
      metadata:
        labels:
          ClusterID: CID-ca4135da3326
          UserID: "102"
          calico/k8s_ns: tanyanliao
          diskType: ""
          name: beegotest
          pod-template-hash: "3324938723"
          tenxcloud.com/appName: beegotest
          tenxcloud.com/svcName: beegotest
        name: eth0
        node: test-slave-114
        orchestrator: k8s
        workload: tanyanliao.beegotest-3324938723-l2tkm
      spec:
        interfaceName: cali34d45bc515f
        ipNetworks:
        - 192.168.134.64/32
        mac: d6:e5:13:88:83:8c
        profiles:
        - k8s_ns.tanyanliao

可以看到这个容器`tanyanliao.beegotest-3324938723-l2tkm`的网卡名称为`cali34d45bc515f`,在主机上查看对应的网卡`cali34d45bc515f`的信息

[root@test-slave-114 ~]# ip link show cali34d45bc515f
    18704: cali34d45bc515f@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
        link/ether 4e:e1:53:f4:e0:06 brd ff:ff:ff:ff:ff:ff link-netnsid 14

可以看到网卡的mac地址是`4e:e1:53:f4:e0:06`跟在容器内执行`arp`命令查询出来的mac地址信息是一致的，也就是说容器内的流量不管是流进还是流出都是通过默认的路由流进流出的

容器内的网卡mac地址

# ip addr
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1
        link/ipip 0.0.0.0 brd 0.0.0.0
    4: eth0@if18704: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
        link/ether d6:e5:13:88:83:8c brd ff:ff:ff:ff:ff:ff link-netnsid 0
        inet 192.168.134.64/32 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::d4e5:13ff:fe88:838c/64 scope link
           valid_lft forever preferred_lft forever

以上说明：  
cni-plugin插件在容器生成了默认的路由，由于默认的IP`169.254.1.1`是无效的路由，因此，该mac地址设置成了该容器在node节点上的mac地址4e:e1:53:f4:e0:06(这一步是通过calico felix完成的)，也就是说,容器内的veth mac地址为`d6:e5:13:88:83:8c`的流量转发是通过默认路由转发到主机生对应的网卡，然后通过ipip隧道走出去的(前提是开了ipip功能的calico)，IPIP隧道是通过tunl0，通讯的

[root@test-slave-114 ~]# ip addr |grep tun
    4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN qlen 1
        inet 192.168.134.113/32 brd 192.168.134.113 scope global tunl0

参考:  
[calico的架构设计与组件交互过程][calico]

[calico]: http://www.lijiaocn.com/%E9%A1%B9%E7%9B%AE/2017/08/04/calico-arch.html