k8s（二）、对外服务(Traefik部署/https/session sticky使用)

Myth丶恋晨 2022-05-27 03:24 300阅读 0赞

## 前言 ##

在上文完成k8s集群搭建的基础上[k8s（一）、1.9.0高可用集群本地离线部署记录][k8s_1.9.0]，承接上文，本文简单介绍一下k8s对外暴露服务

拓扑图：  
![这里写图片描述][70]

## 一、k8s对外暴露服务方式介绍 ##

### 1.Load Blance ###

目前已经有很多主流的公有云平台已经都提供基于k8s服务的组件，工作的原理本质上是k8s的服务向云平台申请一个负载均衡器来对外服务。因此这种方式只适合在云平台内部使用，这里略过.

### 2.Nodeport ###

针对某一个k8s service，k8s集群内的每一个node都将暴露node的一个指定接口，来为此service提供服务。此方式要求每一个node都提供一个端口，即使此node上没有承载有该service 的pod服务载体，因此此方式会带来一定资源的浪费和管理不便。

### 3.Ingress ###

Ingress注入方式，有三个组件，来协同完成对外服务的过程：

*  reverse proxy LB
 *  ingress controller
 *  k8s Ingress resource

#### reverse proxy LB ####

将服务请求反向代理至后端服务对应的node上，node收到后再由kube-proxy将请求转交给endpoint pod.

#### ingress controller ####

监控apiserver上的svc关联关系的变化，若svc关联发生变化（例如svc后端对应的pods发生变化），则动态地获取变化，更新前面反向代理的配置文件和热重载。

#### k8s ingress ####

k8s的一种资源类型，可以基于访问的虚拟主机、字段等进行区别路由映射到后端不同的k8s service上，ingress controller要实时监控每个ingress对象上指定的service来保证LB配置文件的热更新
    
    
    在本文，采用traefik作为ingress工具演示。

### traefik是什么 ###

由于微服务架构以及 Docker 、 kubernetes 编排工具最近才开始流行，因此常用的反向代理服务器 nginx并未提供对k8s ingress的支持，所以Ingress Controller 这个中间件的存在就是用来做 kubernetes 和前端LB来做衔接的；Ingress Controller工作的方式是读取k8s ingress，通过与k8s api交互实时监控service，当serivce映射关系变化时，能重写 nginx 配置和热更新。traefik的出现简化了这个流程，traefik本身是一个轻量级的reverse proxy LB，并且它原生支持跟 kubernetes API 交互，感知后端变化，因此traefik可以取代上面的1、2组件，简化结构。

**traefik官方介绍图：**  
![这里写图片描述][70 1]

## 二、traefik服务部署 ##

首先来部署一组简单的nginx应用容器。  
准备好yaml文件：

[root@171 nginx]# ls
    nginx-deploy.yaml  nginx_ingress.yaml  nginx_service.yaml

[root@171 nginx]# cat nginx-deploy.yaml 
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "2"
      creationTimestamp: 2018-04-09T04:02:02Z
      generation: 4
      labels:
        app: nginx
      name: nginx-deploy
      namespace: default
      resourceVersion: "111504"
      selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx-deploy
      uid: c28090c0-3baa-11e8-b75a-000c29858eab
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: nginx
      strategy:
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 1
        type: RollingUpdate
      template:
        metadata:
          creationTimestamp: null
          labels:
            app: nginx
        spec:
          containers:
          - image: nginx:1.9.1
            imagePullPolicy: IfNotPresent
            name: nginx
            ports:
            - containerPort: 80
              protocol: TCP
            resources: { }
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext: { }
          terminationGracePeriodSeconds: 30
    status:
      availableReplicas: 2
      conditions:
      - lastTransitionTime: 2018-04-09T04:57:27Z
        lastUpdateTime: 2018-04-09T04:57:27Z
        message: Deployment has minimum availability.
        reason: MinimumReplicasAvailable
        status: "True"
        type: Available
      observedGeneration: 4
      readyReplicas: 2
      replicas: 2
      updatedReplicas: 2

[root@171 nginx]# cat nginx_service.yaml 
    apiVersion: v1
    kind: Service
    metadata:
      creationTimestamp: 2018-04-09T11:34:09Z
      labels:
        run: nginx
      name: nginx
      namespace: default
      resourceVersion: "140236"
      selfLink: /api/v1/namespaces/default/services/nginx
      uid: eb57a21b-3be9-11e8-b75a-000c29858eab
    spec:
      clusterIP: 10.99.59.56
      ports:
      - port: 80
        protocol: TCP
        targetPort: 80
      selector:
        app: nginx
      sessionAffinity: None
      type: ClusterIP
    status:
      loadBalancer: { }

[root@171 nginx]# cat nginx_ingress.yaml 
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      creationTimestamp: 2018-04-09T11:39:48Z
      generation: 1
      name: test
      namespace: default
      resourceVersion: "140644"
      selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/test
      uid: b54bbda8-3bea-11e8-b75a-000c29858eab
    spec:
      rules:
      - host: test.nginx.com
        http:
          paths:
          - backend:
              serviceName: nginx
              servicePort: 80
    status:
      loadBalancer: { }

创建nginx的deploy、svc、ing资源，使用–record命令后面可以看到revision记录：

kubectl create -f nginx-deploy.yaml --record
    kubectl create -f nginx-service.yaml --record
    kubectl create -f nginx-ingress.yaml --record

过1分钟后查看状态：

[root@171 nginx]# kubectl get pods -o wide | grep nginx
    nginx-deploy-5964dfd755-b2xj5   1/1       Running   2          1d        10.244.0.16   170
    nginx-deploy-5964dfd755-v6tdq   1/1       Running   1          4h        10.244.0.17   170
    
    
    [root@171 nginx]# kubectl get ing | grep nginx
    test         test.nginx.com             80        23h
    
    [root@171 nginx]# kubectl get svc | grep nginx
    nginx        ClusterIP   10.99.59.56   <none>        80/TCP    23h

在本地curl pod ip和service 的clusterIP测试：

[root@171 nginx]# curl http://10.244.0.16
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
        body { 
            width: 35em;
            margin: 0 auto;
            font-family: Tahoma, Verdana, Arial, sans-serif;
        }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>
    
    
    [root@171 nginx]# curl http://10.244.0.17
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
        body { 
            width: 35em;
            margin: 0 auto;
            font-family: Tahoma, Verdana, Arial, sans-serif;
        }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>
    
    
    
    [root@171 nginx]# curl http://10.99.59.56
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    <style>
        body { 
            width: 35em;
            margin: 0 auto;
            font-family: Tahoma, Verdana, Arial, sans-serif;
        }
    </style>
    </head>
    <body>
    <h1>Welcome to nginx!</h1>
    <p>If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.</p>
    
    <p>For online documentation and support please refer to
    <a href="http://nginx.org/">nginx.org</a>.<br/>
    Commercial support is available at
    <a href="http://nginx.com/">nginx.com</a>.</p>
    
    <p><em>Thank you for using nginx.</em></p>
    </body>
    </html>

nginx资源已经创建好了，但是目前只能在本地和集群内访问，集群外部无法访问，需要把网络路由打通

## 三、网络配置 ##

**1.vmware网络配置：(无网络限制这一步可跳过)**  
在公司的电脑上搭的wmware虚拟机，公司网络上网认证限制外网，单人只允许使用单IP访问外网，因此虚拟机为了访问外网，使用的是NAT模式转接的物理机的网络，vmware NAT方式虚拟机通信配置：  
首先，每台vm都要配置：  
![这里写图片描述][70 2]  
vmware–编辑–虚拟网络编辑器：  
![这里写图片描述][70 3]  
![这里写图片描述][70 4]

为了直接访问vm网段，在windows物理机上配置路由：  
打开cmd，输入：

route add 192.168.0.0 MASK 255.255.255.0  192.168.0.1 IF 9

IF是接口编号，可以使用route print查看自己电脑上vmnet8网卡对应的网卡接口编号  
打开windows网络共享中心，找到物理网卡，右键–属性–共享，勾选开启网络连接共享（类似linux内核设置开启IP forward转发）：  
![这里写图片描述][70 5]

**2.k8s服务网段、pods网段路由，dns配置：**  
步骤1是只是打通了外部到vm之间的网络，如果没有网络限制，vm跟外部在同一网段的可以跳过上面的步骤1。

步骤2是外部添加k8s的服务网段、pods网段：  
windows cmd添加

route add 10.96.0.0  MASK 255.240.0.0  192.168.0.169 IF 9
    route add 10.244.0.0 MASK 255.255.0.0  192.168.0.169 IF 9

windows添加dns记录：  
C:\\Windows\\System32\\drivers\\etc\\hosts打开编辑，添加dns记录：

192.168.0.169       test.nginx.com
    192.168.0.169       ui.traefik.com

打开浏览器测试访问pod ip、服务的cluster ip：  
![这里写图片描述][70 6]  
![这里写图片描述][70 7]  
![这里写图片描述][70 8]  
可以直接访问！  
下面开始部署traefik，通过虚拟主机名访问.

## 四、traefik部署 ##

[root@171 traefik]# ls
    traefik-ds.yaml  traefik_ingress.yaml  traefik-rbac.yaml
    
    
    [root@171 traefik]# cat traefik-rbac.yaml 
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: traefik-ingress-controller
    rules:
      - apiGroups:
          - ""
        resources:
          - services
          - endpoints
          - secrets
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - extensions
        resources:
          - ingresses
        verbs:
          - get
          - list
          - watch
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: traefik-ingress-controller
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: traefik-ingress-controller
    subjects:
    - kind: ServiceAccount
      name: traefik-ingress-controller
      namespace: kube-system
    
    
    
    [root@171 traefik]# cat traefik-ds.yaml 
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: traefik-ingress-controller
      namespace: kube-system
    ---
    kind: DaemonSet
    apiVersion: extensions/v1beta1
    metadata:
      name: traefik-ingress-controller
      namespace: kube-system
      labels:
        k8s-app: traefik-ingress-lb
    spec:
      template:
        metadata:
          labels:
            k8s-app: traefik-ingress-lb
            name: traefik-ingress-lb
        spec:
          serviceAccountName: traefik-ingress-controller
          terminationGracePeriodSeconds: 60
          hostNetwork: true
          containers:
          - image: traefik
            name: traefik-ingress-lb
            ports:
            - name: http
              containerPort: 80
              hostPort: 80
            - name: admin
              containerPort: 8080
            securityContext:
              capabilities:
                drop:
                - ALL
                add:
                - NET_BIND_SERVICE
            args:
            - --api
            - --kubernetes
            - --logLevel=INFO
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: traefik-ingress-service
      namespace: kube-system
    spec:
      selector:
        k8s-app: traefik-ingress-lb
      ports:
        - protocol: TCP
          port: 80
          name: web
        - protocol: TCP
          port: 8080
          name: admin
      type: NodePort
    
    
    
    [root@171 traefik]# cat traefik_ingress.yaml 
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      creationTimestamp: 2018-04-09T11:39:48Z
      generation: 1
      name: traefik-ui
      namespace: default
      resourceVersion: "140644"
      selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/test
      uid: b54bbda8-3bea-11e8-b75a-000c29858eab
    spec:
      rules:
      - host: ui.traefik.com
        http:
          paths:
          - backend:
              serviceName: traefik-ingress-service
              servicePort: 80
    status:
      loadBalancer: { }

创建资源：

kubectl create -f traefik-rbac.yaml --record
    kubectl create -f traefik-ds.yaml --record
    kubectl create -f traefik-ingress.yaml --record

1分钟后查看创建情况：

[root@171 traefik]# kubectl get pods -o wide -n kube-system| grep trae
    traefik-ingress-controller-8lht4       1/1       Running   0          3h        192.168.0.171   171
    traefik-ingress-controller-ddvws       1/1       Running   0          3h        192.168.0.170   170
    
    
    [root@171 traefik]# kubectl get ds -n kube-system| grep trae
    traefik-ingress-controller   2         2         2         2            2           <none>          3h
    
    
    [root@171 traefik]# kubectl get ing 
    NAME         HOSTS            ADDRESS   PORTS     AGE
    test         test.nginx.com             80        1d
    traefik-ui   ui.traefik.com             80        3h

**打开浏览器测试：**  
![这里写图片描述][70 9]  
![这里写图片描述][70 10]

## 开启https ##

提前准备好证书和key:

root@h001238:~/traefik/ssl# ls
    cert.crt  cert.key

创建secret保存https证书:

kubectl create secret generic traefik-cert --from-file=./cert.crt --from-file=./cert.key -n kube-system

编辑traefik配置文件:  
traefik.toml

insecureSkipVerify = true
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
        entryPoint = "https"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]
          [[entryPoints.https.tls.certificates]]
          certFile = "/opt/conf/k8s/ssl/cert.crt"
          keyFile = "/opt/conf/k8s/ssl/cert.key"

创建ConfigMap:

kubectl create configmap traefik-conf --from-file=traefik.toml -n kube-system

修改DaemonSet部署文件,apply部署后直接删除原traefik pod进行配置更新.:

kind: DaemonSet
    apiVersion: extensions/v1beta1
    metadata:
      name: traefik-ingress-controller
      namespace: kube-system
      labels:
        k8s-app: traefik-ingress-lb
    spec:
      template:
        metadata:
          labels:
            k8s-app: traefik-ingress-lb
            name: traefik-ingress-lb
        spec:
          serviceAccountName: traefik-ingress-controller
          terminationGracePeriodSeconds: 60
          hostNetwork: true
          volumes:
          - name: ssl
            secret:
              secretName: traefik-cert
          - name: config
            configMap:
              name: traefik-conf
          containers:
          - image: traefik
            name: traefik-ingress-lb
            volumeMounts:
            - mountPath: "/opt/conf/k8s/ssl"
              name: "ssl"
            - mountPath: "/opt/conf/k8s/conf"
              name: "config"
            ports:
            - name: http
              containerPort: 80
              hostPort: 80
            - name: admin
              containerPort: 8080
            securityContext:
              capabilities:
                drop:
                - ALL
                add:
                - NET_BIND_SERVICE
            args:
            - --api
            - --kubernetes
            - --logLevel=INFO
            - --configFile=/opt/conf/k8s/conf/traefik.toml

查看更新后node上的端口监听:

root@h001238:~/traefik# ss -tnlp | grep trae
    LISTEN     0      128         :::8080                    :::*                   users:(("traefik",pid=18337,fd=6))
    LISTEN     0      128         :::80                      :::*                   users:(("traefik",pid=18337,fd=3))
    LISTEN     0      128         :::443                     :::*                   users:(("traefik",pid=18337,fd=5))

--------------------

## 开启session sticky ##

Traefik支持session sticky, 仅需添加一些简单的配置。有两种方法，一种是在traefik端配置文件中修改，一种是在svc端添加注解。  
第一种：  
在traefik配置文件中顶格添加如下几行，为backend1这个后端开启session sticky。这种方式需要traefik频繁重载配置文件，不适用于实际场景，所以这里不作演示了:

[backends]
          [backends.backend1]
            # Enable sticky session
            [backends.backend1.loadbalancer.stickiness]

完整ConfigMap如下

apiVersion: v1
    data:
      traefik.toml: | insecureSkipVerify = true defaultEntryPoints = ["http","https"] [entryPoints] [entryPoints.http] address = ":80" entryPoint = "https" [entryPoints.https] address = ":443" [entryPoints.https.tls] [[entryPoints.https.tls.certificates]] certFile = "/opt/conf/k8s/ssl/kokoerp.crt" keyFile = "/opt/conf/k8s/ssl/214417797960145.key" [metrics] [metrics.prometheus] entryPoint = "traefik" buckets = [0.1,0.3,1.2,5.0] [backends] [backends.backend1] # Enable sticky session [backends.backend1.loadbalancer.stickiness]
    kind: ConfigMap
    metadata:
      creationTimestamp: 2018-10-18T07:58:57Z
      name: traefik-conf
      namespace: kube-system

第二种：在需要添加session sticky功能的service资源上添加注解如下:

annotations:
        traefik.ingress.kubernetes.io/affinity: "true"
        # 可选，可以自定义
        traefik.ingress.kubernetes.io/session-cookie-name: backend

完成添加注解配置后即可实现该svc绑定的ingress的域名的session sticky功能。

**这里使用prometheus举例**  
将prometheus的deployment的replicas设置数量为2，生成两个pod后端实例：  
![在这里插入图片描述][20190124090246398.png]  
在prometheus的service上添加上方列出的annotation注解后，浏览器访问prometheus，即可看到traefik会为会话添加cookies，key为刚指定的值backend，value默认为后端pod实例：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3l3cTkzNQ_size_16_color_FFFFFF_t_70]  
重复刷新，backend值不会改变，手动清除此cookies后，再次刷新：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3l3cTkzNQ_size_16_color_FFFFFF_t_70 1]

#### session sticky配置完成 ####

更为详细的会话滞黏配置说明请参考官方文档：  
[配置文件][Link 1]  
[Service注解][Service]

--------------------

#### 11.10补充 ####

#### prometheus + traefik 监控 ####

traefik配置添加metric段:

[metrics]
      [metrics.prometheus]
        entryPoint = "traefik"
        buckets = [0.1,0.3,1.2,5.0]

更新configmap后重启traefik-ingress-lb容器.

在prometheus中添加job:

- job_name: 'traefik'
          scrape_interval: 5s
          static_configs: 
            - targets: ['traefik-ui']  #填服务名或任意node IP+指定ui端口

更新configmap后重启prometheus容器.

在prometheus-target中可以看到新job的状态:  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3l3cTkzNQ_size_16_color_FFFFFF_t_70 2]

在grafana中导入id为2870的模板,新版本的traefik metrics名称发生了一些小变化,因此默认模板获取不到数据,需自行对照job target详情页修改一下模板,熟悉prometheus和grafana的话也可以进行一些自定义修改::  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3l3cTkzNQ_size_16_color_FFFFFF_t_70 3]

Prometheus/grafana相关知识及使用可以参考专栏内的其他文章

## 总结 ##

k8s的各种功能组件、名词概念、资源类型（pod/svc/ds/rs/ingress/configmap/role…）、工作流程 等颇为复杂，理解起来不太容易，需要花时间阅读官方文档，因为k8s更新很快，有一些新功能或者即将淘汰的旧功能，官方的中文文档更新不太及时，建议直接阅读英文文档。

另外这里为了方便展示，用的windows查看效果，用linux curl工具测试是一样的，且网络的限制，使用的是vmware的NAT，需要多配置一步物理机网络到vm网络之间打通，再使用静态路由让k8s内部集群网络进行通信。有时间回家里电脑尝试下GNS3模拟路由器，搭建外部bgp，发布到集群bgp内部。

[k8s_1.9.0]: https://blog.csdn.net/ywq935/article/details/79856469
[70]: /images/20220527/0307275aff2b439f98dee93d4502e578.png
[70 1]: /images/20220527/4c76e7727e69437ca751b7dd6f0655f9.png
[70 2]: /images/20220527/6eef6dd306b94a9f84987d4cfe8bcb0b.png
[70 3]: /images/20220527/dbc069d2e201429da19d16ee26d5838f.png
[70 4]: /images/20220527/b8f3563372264c1e8d1821136d81c23b.png
[70 5]: /images/20220527/1cf97dbc815d47aea68df07bbd474707.png
[70 6]: /images/20220527/b47d3c416eac46c3bd78fb73b35a06c1.png
[70 7]: /images/20220527/7285b279926f4c709134742198299dce.png
[70 8]: /images/20220527/b297466bb0ff4ae09220cd32e40f6ba1.png
[70 9]: /images/20220527/5b03a90329514b3e92905c0fe37414de.png
[70 10]: /images/20220527/02f89d9c42f24f64bd2f55ee9050fc5d.png
[20190124090246398.png]: /images/20220527/f5df7d14eb544dcdae4e85850ee458c1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3l3cTkzNQ_size_16_color_FFFFFF_t_70]: /images/20220527/bea224ed1b2d49f2b0ae9bb3ea5dbff4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3l3cTkzNQ_size_16_color_FFFFFF_t_70 1]: /images/20220527/cad8d50cba924a6e804ed532597d327b.png
[Link 1]: https://docs.traefik.io/basics/#sticky-sessions
[Service]: https://docs.traefik.io/configuration/backends/kubernetes/#global-default-backend-ingresses
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3l3cTkzNQ_size_16_color_FFFFFF_t_70 2]: /images/20220527/668c80fb2fb34154a3f3f919e080d492.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3l3cTkzNQ_size_16_color_FFFFFF_t_70 3]: /images/20220527/07f4cd3f878345e78dad7226a903c0a4.png