[kubernetes]-kubernetes+nfs创建高可用mysql

迈不过友情╰ 2022-12-09 04:51 220阅读 0赞

安装mysql主从玩一下。

### 1 创建rabc rabc-1.yaml ###

---
    # 唯一需要修改的地方只有namespace,根据实际情况定义
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default        #根据实际环境设定namespace,下面类同
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: nfs-client-provisioner-runner
    rules:
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["create", "update", "patch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: run-nfs-client-provisioner
    subjects:
      - kind: ServiceAccount
        name: nfs-client-provisioner
        # replace with namespace where provisioner is deployed
        namespace: default
    roleRef:
      kind: ClusterRole
      name: nfs-client-provisioner-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: leader-locking-nfs-client-provisioner
        # replace with namespace where provisioner is deployed
      namespace: default
    rules:
      - apiGroups: [""]
        resources: ["endpoints"]
        verbs: ["get", "list", "watch", "create", "update", "patch"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: leader-locking-nfs-client-provisioner
    subjects:
      - kind: ServiceAccount
        name: nfs-client-provisioner
        # replace with namespace where provisioner is deployed
        namespace: default
    roleRef:
      kind: Role
      name: leader-locking-nfs-client-provisioner
      apiGroup: rbac.authorization.k8s.io

### 2 创建nfsStorageClass nfs-StorageClass-2.yaml ###

apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: managed-nfs-storage
    provisioner: mysql-nfs-storage #这里的名称要和provisioner配置文件中的环境变量PROVISIONER_NAME保持一致parameters:  archiveOnDelete: "false"

> 这里需要先创建好nfs server

### 3 创建nfs-provisioner nfs-provisioner-3.yaml ###

apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nfs-client-provisioner
      labels:
        app: nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default  #与RBAC文件中的namespace保持一致
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: nfs-client-provisioner
      strategy:
        type: Recreate
      selector:
        matchLabels:
          app: nfs-client-provisioner
      template:
        metadata:
          labels:
            app: nfs-client-provisioner
        spec:
          serviceAccountName: nfs-client-provisioner
          containers:
            - name: nfs-client-provisioner
              image: quay.io/external_storage/nfs-client-provisioner:latest
              volumeMounts:
                - name: nfs-client-root
                  mountPath: /persistentvolumes
              env:
                - name: PROVISIONER_NAME
                  value: mysql-nfs-storage  #provisioner名称,请确保该名称与 nfs-StorageClass.yaml文件中的provisioner名称保持一致
                - name: NFS_SERVER
                  value: 192.168.1.171   #NFS Server IP地址
                - name: NFS_PATH
                  value: /srv/data    #NFS挂载卷
          volumes:
            - name: nfs-client-root
              nfs:
                server: 192.168.1.171  #NFS Server IP地址
                path: /srv/data     #NFS 挂载卷

### 4 测试一下 创建test-claim-4.yaml ###

kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: test-claim
      annotations:
        volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"   #与nfs-StorageClass.yaml metadata.name保持一致
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Mi

### 5 测试一下 创建test-pod-5.yaml ###

kind: Pod
    apiVersion: v1
    metadata:
      name: test-pod
    spec:
      containers:
      - name: test-pod
        image: busybox:1.24
        command:
          - "/bin/sh"
        args:
          - "-c"
          - "touch /mnt/SUCCESS && exit 0 || exit 1"   #创建一个SUCCESS文件后退出
        volumeMounts:
          - name: nfs-pvc
            mountPath: "/mnt"
      restartPolicy: "Never"
      volumes:
        - name: nfs-pvc
          persistentVolumeClaim:
            claimName: test-claim  #与PVC名称保持一致

![36ff84ba73b14b2cb6fa98a5663b4a0a.png][]

在nfs 生成了SUCCESS文件

![8250b4be621d58be3f0d47e8d815953a.png][]

### 6 测试一下 nginx-statefulset-6.yaml ###

# 这个我自己没测试
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: nginx-headless
      labels:
        app: nginx
    spec:
      ports:
      - port: 80
        name: web
      clusterIP: None   #注意此处的值,None表示无头服务
      selector:
        app: nginx
    ---
    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: web
    spec:
      serviceName: "nginx"
      replicas: 2  #两个副本
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: ikubernetes/myapp:v1
            ports:
            - containerPort: 80
              name: web
            volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
      volumeClaimTemplates:
      - metadata:
          name: www
          annotations:
            volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"   #managed-nfs-storage为我们创建的storage-class名称
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 1Gi

![d25b56dfa3791b27ee947923ecfc312a.png][]

### 7 创建mysql配置文件 cm-mysql-7.yaml ###

apiVersion: v1
    kind: ConfigMap
    metadata:
      name: mysql
      labels:
        app: mysql
    data:
      master.cnf: |
        # Apply this config only on the master.
        [mysqld]
        log-bin
        log_bin_trust_function_creators=1
        lower_case_table_names=1
      slave.cnf: |
        # Apply this config only on slaves.
        [mysqld]
        super-read-only
        log_bin_trust_function_creators=1

### 8 创建mysql 的svc svc-mysql-8.yaml ###

# Headless service for stable DNS entries of StatefulSet members.
    apiVersion: v1
    kind: Service
    metadata:
      name: mysql
      labels:
        app: mysql
    spec:
      ports:
      - name: mysql
        port: 3306
      clusterIP: None
      selector:
        app: mysql
    ---
    # Client service for connecting to any MySQL instance for reads.
    # For writes, you must instead connect to the master: mysql-0.mysql.
    apiVersion: v1
    kind: Service
    metadata:
      name: mysql-read
      labels:
        app: mysql
    spec:
      ports:
      - name: mysql
        port: 3306
      selector:
        app: mysql

可以看到，这两个 Service 都代理了所有携带 app=mysql 标签的 Pod，也就是所有的 MySQL Pod。端口映射都是用 Service 的 3306 端口对应 Pod 的 3306 端口。

不同的是，第一个名叫“mysql”的 Service 是一个 Headless Service（即：clusterIP= None）。所以它的作用，是通过为 Pod 分配 DNS 记录来固定它的拓扑状态，比如“mysql-0.mysql”和“mysql-1.mysql”这样的 DNS 名字。其中，编号为 0 的节点就是我们的主节点。

而第二个名叫“mysql-read”的 Service，则是一个常规的 Service。

并且我们规定，所有用户的读请求，都必须访问第二个 Service 被自动分配的 DNS 记录，即：“mysql-read”（当然，也可以访问这个 Service 的 VIP）。这样，读请求就可以被转发到任意一个 MySQL 的主节点或者从节点上。

> 备注：Kubernetes 中的所有 Service、Pod 对象，都会被自动分配同名的 DNS 记录。并且读请求应该是分配到所有mysql节点，不是网上博客说的读是到read的slave是那个。

而所有用户的写请求，则必须直接以 DNS 记录的方式访问到 MySQL 的主节点，也就是：“mysql-0.mysql“这条 DNS 记录。

### 9 创建mysql的sts sts-mysql-9.yaml ###

apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: mysql
    spec:
      selector:
        matchLabels:
          app: mysql
      serviceName: mysql
      replicas: 3
      volumeClaimTemplates:
      - metadata:
          name: data
          annotations:
            volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 10Gi
      template:
        metadata:
          labels:
            app: mysql
        spec:
          initContainers:
          - name: init-mysql
            image: registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7
            command:
            - bash
            - "-c"
            - |
              set -ex
              [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
              ordinal=${BASH_REMATCH[1]}
              echo [mysqld] > /mnt/conf.d/server-id.cnf
              echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
              if [[ $ordinal -eq 0 ]]; then
                cp /mnt/config-map/master.cnf /mnt/conf.d/
              else
                cp /mnt/config-map/slave.cnf /mnt/conf.d/
              fi
            volumeMounts:
            - name: conf
              mountPath: /mnt/conf.d
            - name: config-map
              mountPath: /mnt/config-map
          - name: clone-mysql
            image: registry.cn-hangzhou.aliyuncs.com/xjm_public/xtrabackup:1.0
            command:
            - bash
            - "-c"
            - |
              set -ex
              [[ -d /var/lib/mysql/mysql ]] && exit 0
              [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
              ordinal=${BASH_REMATCH[1]}
              [[ $ordinal -eq 0 ]] && exit 0
              ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
              xtrabackup --prepare --target-dir=/var/lib/mysql
            volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
              subPath: mysql
            - name: conf
              mountPath: /etc/mysql/conf.d
          containers:
          - name: mysql
            image: registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7
            env:
            - name: MYSQL_ALLOW_EMPTY_PASSWORD
              value: "1"
            ports:
            - name: mysql
              containerPort: 3306
            volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
              subPath: mysql
            - name: conf
              mountPath: /etc/mysql/conf.d
            resources:
              requests:
                cpu: 500m
                memory: 1Gi
            livenessProbe:
              exec:
                command: ["mysqladmin", "ping"]
              initialDelaySeconds: 30
              periodSeconds: 10
              timeoutSeconds: 5
            readinessProbe:
              exec:
                command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
              initialDelaySeconds: 5
              periodSeconds: 2
              timeoutSeconds: 1
          - name: xtrabackup
            image: registry.cn-hangzhou.aliyuncs.com/xjm_public/xtrabackup:1.0
            ports:
            - name: xtrabackup
              containerPort: 3307
            command:
            - bash
            - "-c"
            - |
              set -ex
              cd /var/lib/mysql
              if [[ -f xtrabackup_slave_info ]]; then
                mv xtrabackup_slave_info change_master_to.sql.in
                rm -f xtrabackup_binlog_info
              elif [[ -f xtrabackup_binlog_info ]]; then
                [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
                rm xtrabackup_binlog_info
                echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
                      MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
              fi
              if [[ -f change_master_to.sql.in ]]; then
                echo "Waiting for mysqld to be ready (accepting connections)"
                until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
                echo "Initializing replication from clone position"
                mv change_master_to.sql.in change_master_to.sql.orig
                mysql -h 127.0.0.1 <<EOF
              $(<change_master_to.sql.orig),
                MASTER_HOST='mysql-0.mysql',
                MASTER_USER='root',
                MASTER_PASSWORD='',
                MASTER_CONNECT_RETRY=10;
              START SLAVE;
              EOF
              fi
              exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
                "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
            volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
              subPath: mysql
            - name: conf
              mountPath: /etc/mysql/conf.d
            resources:
              requests:
                cpu: 100m
                memory: 100Mi
          volumes:
          - name: conf
            emptyDir: {}
          - name: config-map
            configMap:
              name: mysql

![5589dded241eb6049008332e11233b90.png][]

测试服务 启动可能比较慢

kubectl run mysql-client --image=registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7 -it --rm --restart=Never -- mysql -h mysql-0.mysql.default

![dad76e0e209b344b084acae1ddcce5c0.png][]

CREATE DATABASE demo;
    CREATE TABLE demo.messages (message VARCHAR(250));
    INSERT INTO demo.messages VALUES ('hello');

查看主机名为mysql-read的数据

kubectl run mysql-client2 --image=registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7 -i -t --rm --restart=Never -- mysql -h mysql-read.default

select * from demo.messages;

![70f424dcfa8389516a272982a8d81945.png][]

最后参考博客里的有些错误描述 需要纠正一下

1 读请求 应该是0-2这3台机子都会请求。因为svc设置的是app:mysql 主从并没有区分

2 删除master节点 似乎并不会由slave 升级为master 依旧是重新生成了一个新的mysql-0

![8019914403d46a445a4ee9587840e60b.png][]

> 测试继续写入数据没有问题 后续进行更多测试
> 
> 但是slave不会升级成master的话 可能可用性还有待提高 更换为operater 主从用不同的sts来部署可能会好些

参考

mysql

https://my.oschina.net/u/3627276/blog/4528810

nfs storage

https://www.cnblogs.com/panwenbin-logs/p/12196286.html

[36ff84ba73b14b2cb6fa98a5663b4a0a.png]: /images/20221123/a09529ec2577465f9fb1179687973ac2.png
[8250b4be621d58be3f0d47e8d815953a.png]: /images/20221123/d79a0f3c333040debb4b846a6be5edb3.png
[d25b56dfa3791b27ee947923ecfc312a.png]: /images/20221123/2fb498c2859b46828ae447d4e2dca766.png
[5589dded241eb6049008332e11233b90.png]: /images/20221123/2f4789dfbbfe4e0cbdb03e9335156394.png
[dad76e0e209b344b084acae1ddcce5c0.png]: /images/20221123/315a358ceb6f48fd90274ada7005ebb4.png
[70f424dcfa8389516a272982a8d81945.png]: /images/20221123/3970fbfcb26c48cd961cd12168c47eab.png
[8019914403d46a445a4ee9587840e60b.png]: /images/20221123/28035aeb81ac45e8a2e89b778dbe75f6.png