[kubernetes]-kubernetes中部署jenkins及slave并简单使用

爱被打了一巴掌 2022-12-02 01:12 215阅读 0赞

安装nfs做持久化存储，我这里是拿原来的jenkins服务器做了。因为有些插件 maven仓库和依赖包要挂载，复制来复制去不方便。如果是新的jenkins，就随意了。

# 安装nfs 并挂载共享目录/root/.m2  /usr/share/maven  /root/.jenkins 
    yum install -y nfs-utils
    echo "/root/.m2 192.168.0.0/16(rw,no_root_squash,sync)" >>/etc/exports
    echo "/usr/share/maven 192.168.0.0/16(rw,no_root_squash,sync)" >>/etc/exports
    echo "/root/.jenkins 192.168.0.0/16(rw,no_root_squash,sync)" >>/etc/exports
    systemctl enable rpcbind.service
    systemctl enable nfs-server.service
    
    systemctl start rpcbind.service
    systemctl start nfs-server.service
    
    systemctl  stop firewalld
    systemctl  disable firewalld
    rpcinfo -p
    
    exportfs -r
    exportfs

后面用到的镜像为jenkins/jenkins:2.199 和 cnych/jenkins:jnlp6 建议先下载好传到自己的仓库或者直接导入到node节点上。

![format_png][]

创建一个专门运维相关的namespace

kubectl create namespace devops

![format_png 1][]

创建3个yaml文件 jenkins-pvc.yaml jenkins-rbac.yaml jenkins-deploy.yaml

jenkins-pvc.yaml

# 给jenkins的workspace  maven地址等创建pv和pvc  ,为jenkins提供数据持久化
    # 挂载原jenkins的插件等到pod中 如果新建的话 直接挂干净的路径
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: jenkins-pv
      namespace: devops
    spec:
      capacity:
        storage: 200Gi
      accessModes:
      - ReadWriteMany
      persistentVolumeReclaimPolicy: Delete
      nfs:
        server: 192.168.1.171
        path: /root/.jenkins
    
    
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: jenkins-pvc
      namespace: devops
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 200Gi
    
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: mvn-pv
      namespace: devops
    spec:
      capacity:
        storage: 5Gi
      accessModes:
      - ReadWriteMany
      persistentVolumeReclaimPolicy: Delete
      nfs:
        server: 192.168.1.171
        path: /usr/share/maven
    
    
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: mvn-pvc
      namespace: devops
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 5Gi
    
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: m2-pv
      namespace: devops
    spec:
      capacity:
        storage: 200Gi
      accessModes:
      - ReadWriteMany
      persistentVolumeReclaimPolicy: Delete
      nfs:
        server: 192.168.1.171
        path: /root/.m2
    
    
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: m2-pvc
      namespace: devops
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 200Gi
    
    
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: jenkins-plugins-pv
      namespace: devops
    spec:
      capacity:
        storage: 200Gi
      accessModes:
      - ReadWriteMany
      persistentVolumeReclaimPolicy: Delete
      nfs:
        server: 192.168.1.171
        path: /root/.jenkins/plugins
    
    
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: jenkins-plugins-pvc
      namespace: devops
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 200Gi

jenkins-rbac.yaml

# 创建角色授权
    --- 
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: jenkins-sa
      namespace: devops
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
      name: jenkins-cr
    rules:
      - apiGroups: ["extensions", "apps"]
        resources: ["deployments"]
        verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
      - apiGroups: [""]
        resources: ["services"]
        verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["create","delete","get","list","patch","update","watch"]
      - apiGroups: [""]
        resources: ["pods/exec"]
        verbs: ["create","delete","get","list","patch","update","watch"]
      - apiGroups: [""]
        resources: ["pods/log"]
        verbs: ["get","list","watch"]
      - apiGroups: [""]
        resources: ["secrets"]
        verbs: ["get"]
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
      name: jenkins-crd
    roleRef:
      kind: ClusterRole
      name: jenkins-cr
      apiGroup: rbac.authorization.k8s.io
    subjects:
    - kind: ServiceAccount
      name: jenkins-sa
      namespace: devops

jenkins-deploy.yaml

# 部署一个deployment 跑jenkins 并且以nodeport的方式创建svc
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: jenkins
      namespace: devops
    spec:
      template:
        metadata:
          labels:
            app: jenkins
        spec:
          terminationGracePeriodSeconds: 10
          serviceAccount: jenkins-sa
          containers:
          - name: jenkins
            image: registry.cn-shanghai.aliyuncs.com/yjk-datag/jenkins:2.199
            env:
            - name: JAVA_OPTS
              value: -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
            ports:
            - containerPort: 8080
              name: web
              protocol: TCP
            - containerPort: 50000
              name: agent
              protocol: TCP
            resources:
              limits:
                cpu: 1000m
                memory: 1Gi
              requests:
                cpu: 500m
                memory: 512Mi
            livenessProbe:
              httpGet:
                path: /login
                port: 8080
              initialDelaySeconds: 60
              timeoutSeconds: 5
              failureThreshold: 12
            readinessProbe:
              httpGet:
                path: /login
                port: 8080
              initialDelaySeconds: 60
              timeoutSeconds: 5
              failureThreshold: 12
            volumeMounts:
    # 这里我测试的时候没敢把原jenkins的jenkins_home挂上去 所以先注释测试一下        
    #        - name: jenkinshome
    #          mountPath: /var/jenkins_home
            - name: pluginshome
              mountPath: /var/jenkins_home/plugins
            - name: mvnhome
              mountPath: /usr/share/maven
            - name: m2home
              mountPath: /root/.m2
          securityContext:
            fsGroup: 1000
          volumes:
          - name: jenkinshome
            persistentVolumeClaim:
              claimName: jenkins-pvc
          - name: mvnhome
            persistentVolumeClaim:
              claimName: mvn-pvc
          - name: m2home
            persistentVolumeClaim:
              claimName: m2-pvc
          - name: pluginshome
            persistentVolumeClaim:
              claimName: jenkins-plugins-pvc
    
          imagePullSecrets:
          - name: aliyun-ns-devops
    
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: jenkins
      namespace: devops
      labels:
        app: jenkins
    spec:
      selector:
        app: jenkins
      type: NodePort
      ports:
      - name: web
        port: 8080
        targetPort: web
        nodePort: 30002
      - name: agent
        port: 50000
        targetPort: agent

依次执行

kubectl apply -f jenkins-pvc.yaml
    kubectl apply -f jenkins-rbac.yaml
    kubectl apply -f jenkins-deploy.yaml
    kubectl get pods -n devops -o wide -w

这里启动比较慢 还重启了一次 不知道是否和我挂载了插件 需要加载有关系

![format_png 2][]

访问node节点的30002端口就可以访问jenkins服务。

\# 初始化登陆jenkins的密码可以通过登陆pod(kubectl exec)或者查看pod(kubectl log)日志获取。

\# 初始化插件我是全部没有安装 因为我自己挂载了插件文件夹。

\# 没有安装插件kubernetes的话 需要安装插件 我这里是直接挂载的插件包已经安装了这个

![format_png 3][]

直接进入系统配置 拉到最下面的云 配置填入如图所示参数

# https://kubernetes.default.svc.cluster.local
    # devops
    # http://jenkins.devops.svc.cluster.local:8080

点击连接测试 通过之后再继续往下配置

![format_png 4][]

添加pod模版

![format_png 5][]

添加pod label 继续填入参数

![format_png 6][]

\# pod的模板名称为jenkins-slave，Container的模板名称为jnlp。这里有非常重要的两点要注意：  
当Container的模板名称为jnlp的时候，jenkins-slave才会使用下面配置的docker镜像来启动pod，如果不为jnlp，则会使用默认的镜像jenkins/jnlp-slave:alpine

\# 当使用自定义的docker镜像来启动jenkins slave pod的时候，下面的command to run（默认值是 sh -c）和arguments to pass to the command(默认值是cat)两个值需要清空。否则会出现jenkins slave jnlp连接不上master的情况，尝试100次连接之后销毁pod，然后再创建一个pod继续尝试连接，无限循环。

![format_png 7][]

![format_png 8][]

需要挂载两个主机目录：

1.  /var/run/docker.sock：该文件是用于 Pod 中的容器能够共享宿主机的 Docker；
2.  /root/.kube：这个目录挂载到容器的/root/.kube目录下面这是为了让我们能够在 Pod 的容器中能够使用 kubectl 工具来访问我们的 Kubernetes 集群，方便我们后面在 Slave Pod 部署 Kubernetes 应用；

![format_png 9][]

![format_png 10][]

创建项目测试

注意:需要修改项目的标签 只在jenkins-jnlp的节点上运行

非pipeline项目

![format_png 11][]

pipeline项目

![format_png 12][]

构建测试

![format_png 13][]

查看此次构建结果

![format_png 14][]

到这里完成了第一步，后续在考虑安装java或者挂载文件夹。

遇到的问题

1 jenkins的slave 一直在更换node 这个是我没有用自己仓库的镜像，导致服务一直没有起来，超时之后就换机子继续下镜像，镜像下的又很慢。

![format_png 15][]

2 启动成功的slave 显示不在线 之后继续更换node

![format_png 16][]

参考并纠正

https://www.cnblogs.com/coolops/p/13129955.html

[format_png]: /images/20221123/9e5339aba0474d8ca90e91b178fec5d8.png
[format_png 1]: /images/20221123/775901307bdb49dab260c3ad51d7b99b.png
[format_png 2]: /images/20221123/c4e58ecbbf1042368b5f543d0db9c95d.png
[format_png 3]: /images/20221123/de4af655601c48f59a6c07ad24af8932.png
[format_png 4]: /images/20221123/b19bc22db2a24d67ab9817f711c5bf7e.png
[format_png 5]: /images/20221123/4de32d3b391b488091c8b25b39e2985a.png
[format_png 6]: /images/20221123/4071643416e8404b9ab5f8463b857e55.png
[format_png 7]: /images/20221123/316d76de2ab948e795e8da5fd25bc22c.png
[format_png 8]: /images/20221123/cb6fa6f4c41f4202869f9d72d8412006.png
[format_png 9]: /images/20221123/c924d75590c54504aa134c12505842ad.png
[format_png 10]: /images/20221123/82e033f0e87c424c9e90d773918697de.png
[format_png 11]: /images/20221123/b36116acf29744a7bdb37d45444a250a.png
[format_png 12]: /images/20221123/b41df560a40840aaa0fdda2c28ee1c52.png
[format_png 13]: /images/20221123/7c7d96aba8624a779a2261ab4416a294.png
[format_png 14]: /images/20221123/b8c385c885de45a8a0481a104e4b12e2.png
[format_png 15]: /images/20221123/03dac08cbfe24e5c935ee2516119c70c.png
[format_png 16]: /images/20221123/514f2ffd45b842d997c42169bf0923c7.png