在K8S集群中一步步构建一个复杂的MySQL数据库

桃扇骨 2022-03-24 18:48 46阅读 0赞

源：[https://segmentfault.com/a/1190000014966962][https_segmentfault.com_a_1190000014966962]

# 文档说明 #

本文面向容器初学者，作者先简单的用*MySQL*官方镜像搭建一个可运行的单实例数据库，而后考虑生产或现实需求，一步一步完善并揉合*K8S*多个技术，从而构建一个复杂且可供生产用的*MySQL*单实例库。

# 简单部署 #

如下所示，我们仅需设置*root*用户密码（环境变量*MYSQL\_ROOT\_PASSWORD*）， 便可轻松的使用*MySQL*官方镜像构建一个*MySQL*数据库。

# kubectl create -f - <<EOF
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      labels:
        app: mysql
      name: mysql
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: mysql
      template:
        metadata:
          labels:
            app: mysql
        spec:
          containers:
          - image: mysql
            name: mysql
            imagePullPolicy: IfNotPresent
            env:
            - name: MYSQL_ROOT_PASSWORD
              value: Changeme
    EOF

***注意***：若你的*K8S*集群是*minishift*、*openshift*、*origin*，因其为安全考虑，不允许容器以*root*用户运行，而官方*MySQL*镜像却需*root*权限，故为使其能顺利运行，我们需将*anyuid scc*赋予*default serviceaccount*：

# oc adm policy add-scc-to-user anyuid -z default

创建一*Service*以便集群内外均可访问数据库，其中集群外需通过*nodePort*设置的*30006*端口访问。

# kubectl create -f - <<EOF
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: mysql
      name: mysql
    spec:
      type: NodePort
      ports:
      - port: 3306
        nodePort: 30006
        protocol: TCP
        targetPort: 3306
      selector:
        app: mysql
    EOF

接着，访问数据库并验证其运行正常：

# kubectl get pod                                   # 当前Pod名称
    NAME                     READY     STATUS    RESTARTS   AGE
    mysql-5b5668c448-t44ml   1/1       Running   0          3h
    
    # 通过本机访问
    # kubectl exec -it mysql-5b5668c448-t44ml -- mysql -uroot -pChangeme
    
    mysql> select 1;
    +---+
    | 1 |
    +---+
    | 1 |
    +---+
    
    # 集群内部通过mysql service访问：
    # kubectl exec -it mysql-5b5668c448-t44ml -- mysql -uroot -pChangeme -hmysql
    
    mysql> select now();
    +---------------------+
    | now()               |
    +---------------------+
    | 2018-05-21 07:19:14 |
    +---------------------+
    
    # 集群外部，可通过任何一个K8S节点访问数据库：
    # mysql -uroot -pChangeme -horigin-lb-01 -P30006
    
    mysql> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | information_schema |
    | mysql              |
    | performance_schema |
    | sys                |
    +--------------------+

# 扩展部署 #

## 持久化存储 ##

若要确保*MySQL*重启后数据仍然存在，我们需为其配置可持久化存储，作者的实验环境配置了*GlusterFS*分布式存储，其支持*K8S*动态提供特性，故可执行如下命令创建*PVC*：

# kubectl create -f - <<EOF
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: mysql
    spec:
      accessModes:
        - ReadOnlyMany
      resources:
        requests:
          storage: 1Gi
      storageClassName: glusterfs-raid0
    EOF

而后，调整*Deploy*并挂载卷：

spec:
          containers:
          - image: mysql
    ...
            volumeMounts:
            - name: mysql-data
              mountPath: /var/lib/mysql
          volumes:
          - name: mysql-data
            persistentVolumeClaim:
              claimName: mysql

## 自定义配置文件 ##

通过创建*cm*并挂载到容器中，我们可自定义*MySQL*配置文件。如下所示，名为*mysql-config*的*cm*包含一个*custom.cnf*文件：

apiVersion: v1
    metadata:
      name: mysql-config
    data:
      custom.cnf: |
            [mysqld]
            default_storage_engine=innodb
            skip_external_locking
            lower_case_table_names=1
            skip_host_cache
            skip_name_resolve
    kind: ConfigMap

将*cm*挂载到容器内：

spec:
    ...
          containers:
          - image: mysql
    ...
            volumeMounts:
            - name: mysql-config
              mountPath: /etc/mysql/conf.d/
    ...
          volumes:
          - name: mysql-config
            configMap:
              name: mysql-config
    ...

## 加密铭感数据 ##

用户密码等铭感数据以*Secret*加密保存，而后被*Deployment*通过*volume*挂载或环境变量引用。如本例，我们创建*root*、*app*、*test*用户，将*3*个用户的密码加密保存：

# echo -n Changeme | base64
    Q2hhbmdlbWU=
    # kubectl create -f - <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      name: mysql-user-pwd
    data:
      mysql-root-pwd: Q2hhbmdlbWU=
      mysql-app-user-pwd: Q2hhbmdlbWU=
      mysql-test-user-pwd: Q2hhbmdlbWU=
    EOF

*Secret*创建完成后，我们将用户明文密码从*Deployment*去除，采用环境变量方式引用*Secret*数据，参见如下*Yaml*修改，做了*3*个调整：

*  镜像初始化时自动创建*MYSQL\_DATABASE*环境变量[1][]设置的数据库；
 *  镜像初始化时将*MYSQL\_DATABASE*数据库赋予*MYSQL\_USER*用户；
 *  *root*用户及*MYSQL\_USER*用户，其密码均通过*secretKeyRef*从*secret*获取；

spec:
    ...
          containers:
          - image: mysql
            name: mysql
            imagePullPolicy: IfNotPresent
            env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-user-pwd
                  key: mysql-root-pwd
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-user-pwd
                  key: mysql-app-user-pwd
            - name: MYSQL_USER
              value: app
            - name: MYSQL_DATABASE
              value: appdb

## 容器健康检查 ##

*K8S*镜像控制器可通过*livenessProbe*判断容器是否异常，进而决定是否重建容器；而*Service*服务可通过*readinessProbe*判断容器服务是否正常，从而确保服务可用性。

本例，作者配置的*livenessProbe*与*readinessProbe*是一样的，即连续3次查询数据库失败，则定义为异常。对*livenessProbe*与*readinessProbe*详细用法，不在本文的讨论范围内，可参考*K8S*官方文档：

*  [*Configure Liveness and Readiness Probes*][Configure Liveness and Readiness Probes]
 *  [*Pod Lifecycle*][Pod Lifecycle]

spec:
          containers:
          - image: mysql
    ...
            livenessProbe:
              exec:
                command:
                - /bin/sh
                - "-c"
                - MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
                - mysql -h 127.0.0.1 -u root -e "SELECT 1"
              initialDelaySeconds: 30
              timeoutSeconds: 5
              successThreshold: 1
              failureThreshold: 3
            readinessProbe:
              exec:
                command:
                - /bin/sh
                - "-c"
                - MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
                - mysql -h 127.0.0.1 -u root -e "SELECT 1"
              initialDelaySeconds: 10
              timeoutSeconds: 1
              successThreshold: 1
              failureThreshold: 3
    ...

## 容器初始化 ##

假设，我们有这样的需求：“初始部署*MySQL*时，其已包应用所需的数据库、用户、权限、表结构与数据”。研究*MySQL*官方镜像的*[Dockerfile][]*可知，数据库初始化时将自动执行目录*/docker-entrypoint-initdb.d*内的*.sh*、*.sql*、*.sql.gz*文件，鉴于此，我们可有如下两种方法：

1.  基于官方镜像重新编写*Dockerfile*，将脚本*copy*到新镜像*/docker-entrypoint-initdb.d*目录，因需编译新镜像，故此方法不灵活；
2.  初始化容器（*initContainers*）在常规容器（*containers*）前运行，故在初始化容器中可将脚本拷贝到共享目录，而后*MySQL*镜像挂载此目录到*/docker-entrypoint-initdb.d*，此方法灵活。

本例，作者采用初始化容器方案，功能如下：

*  初始化与常规容器共享名为*mysql-initdb*的*emptyDir*，均被挂载到*/docker-entrypoint-initdb.d*目录；
 *  初始化容器将*.sql*文件置于共享的*/docker-entrypoint-initdb.d*目录，其含初始化*testdb*与*appdb*数据库；
 *  为了避免*MySQL*数据库目录内的*lost+found*目录被误认为是数据库，初始化容器中将其删除；

spec:
          initContainers:
          - name: mysql-init
            image: busybox
            imagePullPolicy: IfNotPresent
            env:
            - name: MYSQL_TEST_USER_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-user-pwd
                  key: mysql-test-user-pwd
            command:  
              - sh
              - "-c"
              - |
                set -ex
                rm -fr /var/lib/mysql/lost+found
                cat > /docker-entrypoint-initdb.d/mysql-testdb-init.sql <<EOF
                create database testdb default character set utf8;
                grant all on testdb.* to 'test'@'%' identified by '$MYSQL_TEST_USER_PASSWORD';
                flush privileges;
                EOF
                cat > /docker-entrypoint-initdb.d/mysql-appdb-init.sql <<EOF
                create table app(id int);
                insert into app values(1);
                commit;
                EOF
            volumeMounts:
            - name: mysql-data
              mountPath: /var/lib/mysql
            - name: mysql-initdb
              mountPath: /docker-entrypoint-initdb.d
          containers:
          - image: mysql
            volumeMounts:
            - name: mysql-initdb
              mountPath: /docker-entrypoint-initdb.d
    ...
          volumes:
          - name: mysql-data
            persistentVolumeClaim:
              claimName: mysql
          - name: mysql-initdb
            emptyDir: {}
    ...

## 完整*Deployment* ##

通过如上多步调整，*MySQL*数据库的*Deplyment*如下所示：

--------------------

apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      labels:
        app: mysql
      name: mysql
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: mysql
      template:
        metadata:
          labels:
            app: mysql
        spec:
          initContainers:
          - name: mysql-init
            image: busybox
            imagePullPolicy: IfNotPresent
            env:
            - name: MYSQL_TEST_USER_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-user-pwd
                  key: mysql-test-user-pwd
            command:  
              - sh
              - "-c"
              - |
                set -ex
                rm -fr /var/lib/mysql/lost+found
                cat > /docker-entrypoint-initdb.d/mysql-testdb-init.sql <<EOF
                create database testdb default character set utf8;
                grant all on testdb.* to 'test'@'%' identified by '$MYSQL_TEST_USER_PASSWORD';
                flush privileges;
                EOF
                cat > /docker-entrypoint-initdb.d/mysql-appdb-init.sql <<EOF
                create table app(id int);
                insert into app values(1);
                commit;
                EOF
            volumeMounts:
            - name: mysql-data
              mountPath: /var/lib/mysql
            - name: mysql-initdb
              mountPath: /docker-entrypoint-initdb.d
          containers:
          - image: mysql
            name: mysql
            imagePullPolicy: IfNotPresent
            env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-user-pwd
                  key: mysql-root-pwd
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-user-pwd
                  key: mysql-app-user-pwd
            - name: MYSQL_USER
              value: app
            - name: MYSQL_DATABASE
              value: appdb
            volumeMounts:
            - name: mysql-data
              mountPath: /var/lib/mysql
            - name: mysql-initdb
              mountPath: /docker-entrypoint-initdb.d
            - name: mysql-config
              mountPath: /etc/mysql/conf.d/
            ports:
            - name: mysql
              containerPort: 3306
                command:
                - /bin/sh
                - "-c"
                - MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
                - mysql -h 127.0.0.1 -u root -e "SELECT 1"
              initialDelaySeconds: 30
              timeoutSeconds: 5
              successThreshold: 1
              failureThreshold: 3
            readinessProbe:
              exec:
                command:
                - /bin/sh
                - "-c"
                - MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
                - mysql -h 127.0.0.1 -u root -e "SELECT 1"
              initialDelaySeconds: 10
              timeoutSeconds: 1
              successThreshold: 1
              failureThreshold: 3
          volumes:
          - name: mysql-data
            persistentVolumeClaim:
              claimName: mysql
          - name: mysql-initdb
            emptyDir: {}
          - name: mysql-config
            configMap:
              name: mysql-config

创建此*Deployment*后，我们有如下组件：

# kubectl get all,pvc,cm,secret
    # MySQL Deployment：
    NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    deploy/mysql   1         1         1            1           1m
    
    # RS被Deployment调用，其是自动生成的
    NAME                 DESIRED   CURRENT   READY     AGE
    rs/mysql-998977cdd   1         1         1         1m
    
    # Pod：
    NAME                       READY     STATUS    RESTARTS   AGE
    po/mysql-998977cdd-v2ks2   1/1       Running   1          1m
    
    # Service：
    NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
    svc/mysql         NodePort    172.30.3.200    <none>        3306:30006/TCP   8h
    
    # Pvc：
    NAME            STATUS    VOLUME     CAPACITY   ACCESS MODES STORAGECLASS      AGE
    pvc/mysql       Bound     pvc-fe..   1Gi        ROX          glusterfs-raid0   2m
    
    # Configmap：
    NAME              DATA      AGE
    cm/mysql-config   1         6h
    
    # Secret：
    NAME                        TYPE                                  DATA      AGE
    secrets/mysql-user-pwd      Opaque                                3         1h

## 定期自动备份 ##

考虑到数据安全性，我们定期备份数据库，在*K8S*集群中，我们可配置[*CronJob*][CronJob]实现自动备份作业。首先，创建一个持久化存储供备份用：

# kubectl create -f - <<EOF
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: mysql-backup
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 2Gi
      storageClassName: glusterfs-raid0
    EOF

继而，配置实际的自动化作业任务，如下所示，每天凌晨*0*点将使用*mysqldump*备份*appdb*数据库。

# kubectl create -f - <<EOF
    apiVersion: batch/v1beta1
    kind: CronJob
    metadata:
      name: mysql-backup
    spec:
      schedule: "0 0 * * *"
      jobTemplate:
        spec:
          template:
            spec:
              containers:
              - name: mysql-backup
                imagePullPolicy: IfNotPresent
                image: mysql
                env:
                - name: MYSQL_BACKUP_USER
                  value: root
                - name: MYSQL_BACKUP_USER_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: mysql-user-pwd
                      key: mysql-root-pwd
                - name: MYSQL_HOST
                  value: mysql
                command:
                - /bin/sh
                - -c
                - |
                  set -ex
                  mysqldump --host=$MYSQL_HOST --user=$MYSQL_BACKUP_USER \
                            --password=$MYSQL_BACKUP_USER_PASSWORD \
                            --routines --databases appdb --single-transaction \
                            > /mysql-backup/mysql-`date +"%Y%m%d"`.sql
                volumeMounts:
                - name: mysql-backup
                  mountPath: /mysql-backup
              restartPolicy: OnFailure
              volumes:
              - name: mysql-backup
                persistentVolumeClaim:
                  claimName: mysql-backup
    EOF

# 结束语 #

本文揉合*K8S*多项技术，构建了一个复杂且可做生产使用的范例，当然，此库是单实例数据库，倘若需构建数据库高可用方案，需部署如*MySQL HA*、*PXC*集群，其中自动作业备份范例仅使用*mysqldump*备份，在生产环境不是很实用，我们需要考虑使用*xtrabackup*备份以及*mysqlbinlog*备份日志。

--------------------

1.  参见[*docker-entrypoint.sh*][docker-entrypoint.sh]。 [↩][Link 1]

[https_segmentfault.com_a_1190000014966962]: https://segmentfault.com/a/1190000014966962
[1]: https://segmentfault.com/a/1190000014966962#fn-1
[Configure Liveness and Readiness Probes]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
[Pod Lifecycle]: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
[Dockerfile]: https://github.com/docker-library/mysql/tree/fc3e856313423dc2d6a8d74cfd6b678582090fc7/8.0
[CronJob]: https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/
[docker-entrypoint.sh]: https://github.com/docker-library/mysql/blob/fc3e856313423dc2d6a8d74cfd6b678582090fc7/8.0/docker-entrypoint.sh
[Link 1]: https://segmentfault.com/a/1190000014966962#fnref-1