【kubernetes/k8s源码分析】kubemark 解读源码

╰+哭是因爲堅強的太久メ 2023-05-31 06:36 11阅读 0赞

>     Kubernetes 社区提供了 SLI (服务等级指标) 和 SLO (服务等级目标) 系统性能测试、分析文档 [Kubernetes scalability and performance SLIs/SLOs][Kubernetes scalability and performance SLIs_SLOs]。模拟出一个 K8s cluster（Kubemark cluster），不受资源限制。cluster 中 master 是真实的机器，所有的 nodes 是 Hollow nodes。Hollow nodes 不会调用Docker，测试一套 K8s API 调用的完整流程，不会真正创建 pod。
> 
>  
> 
>     社区开发了 [perf-test/clusterloader2][perf-test_clusterloader2]，可配置性好，并且统计了相应的性能指标
> 
>     kubemark 不调用 CRI 接口之外，其它行为和 kubelet 基本一致

/bin/sh -c /kubemark --morph=kubelet --name=hollow-node-9dfp9 \{ \{hollow\_kubelet\_params\}\} --kubeconfig=/kubeconfig/kubelet.kubeconfig --logtostderr -v 4

使用了 command.Cobra 第三方命令行包，直接上重点 run 函数

# 1. run 函数 #

--morph 是该hollow 模拟哪个组建，可选的有 kubelet 和 kube-proxy

func run(config *hollowNodeConfig) {
    	// To help debugging, immediately log version
    	klog.Infof("Version: %+v", version.Get())
    
    	if !knownMorphs.Has(config.Morph) {
    		klog.Fatalf("Unknown morph: %v. Allowed values: %v", config.Morph, knownMorphs.List())
    	}

## ##

# 2. 如果指定 --morph 为 kubelet 的情况 #

GetHollowKubeletConfig 设置 root dir 为 /tmp/hollow-kubelet，pod 工作目录为 /tmp/hollow-kubelet/static-pods，

直接调用 kubelet 的 NewKubeletFlags 初始化 kubelet 的参数

### **    2.1 不直接调用 docker，而是调用 fake** ###

fakeDockerClientConfig := &dockershim.ClientConfig{
    	DockerEndpoint:    libdocker.FakeDockerEndpoint,
    	EnableSleep:       true,
    	WithTraceDisabled: true,
    }

### **    2.2 Run 函数启动 HollowKubelet 服务** ###

**     直接调用 kubelet 的 **RunKubelet 函数，所以和 kubelet 行为一模一样，只不过调用到 fake docker endpoint

// Starts this HollowKubelet and blocks.
    func (hk *HollowKubelet) Run() {
    	if err := kubeletapp.RunKubelet(&options.KubeletServer{
    		KubeletFlags:         *hk.KubeletFlags,
    		KubeletConfiguration: *hk.KubeletConfiguration,
    	}, hk.KubeletDeps, false); err != nil {
    		klog.Fatalf("Failed to run HollowKubelet: %v. Exiting.", err)
    	}
    	select {}
    }

# 3. 如果指定 --morph 为 kube-proxy 的情况 #

iptables 以及 sysctl 使用了 fake

if config.Morph == "proxy" {
    	client, err := clientset.NewForConfig(clientConfig)
    	if err != nil {
    		klog.Fatalf("Failed to create API Server client: %v", err)
    	}
    	iptInterface := fakeiptables.NewFake()
    	sysctl := fakesysctl.NewFake()
    	execer := &fakeexec.FakeExec{
    		LookPathFunc: func(_ string) (string, error) { return "", errors.New("fake execer") },
    	}

## A. 编译 kubemark ##

在 kubernetes 目录下执行，**make WHAT='test/e2e/e2e.test'   && make ginkgo****  **

*  编译二进制  
    下载kubernetes源码，执行`make WHAT='cmd/kubemark'`

## **B. 为 hollow 设置 secret 和 configmap** ##

**    **kubectl create secret generic kubeconfig -n default --from-file=kubelet.kubeconfig=/root/.kube/config

kubectl create configmap node-configmap -n default --from-literal=content.type="test-cluster"

## C. 部署 hollow，如下所示 hollow.yaml，执行 kubectl apply -f hollow.yaml  ##

注意

C.1.设置真实节点 Taint，主要是避免测试调度到真正的 node 节点上

kubectl taint nodes $\{node\} role=real:NoSchedule

C.2 为真实node 设置label，使 hollow 调度到其上运行

kubectl label nodes $\{node\} role=real

apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hollow-node
      namespace: default
      labels:
        name: hollow-node
    spec:
      replicas: 101
      selector:
        matchLabels:
          name: hollow-node
      template:
        metadata:
          labels:
            name: hollow-node
        spec:
          nodeSelector:
            noHollowNode: nodeHollowNode
          initContainers:
          - name: init-inotify-limit
            image: busybox
            imagePullPolicy: IfNotPresent
            command: ['sysctl', '-w', 'fs.inotify.max_user_instances=1000']
            securityContext:
              privileged: true
          volumes:
          - name: kubeconfig-volume
            secret:
              secretName: kubeconfig
          - name: kernelmonitorconfig-volume
            configMap:
              name: node-configmap
          - name: logs-volume
            hostPath:
              path: /var/log
          - name: no-serviceaccount-access-to-real-master
            emptyDir: {}
          - name: k8s-ssl
            hostPath:
              path: /opt/kubernetes/ssl
          containers:
          - name: hollow-kubelet
            image: zhangzhonglin/kubemark:v1.10.5
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 4194
            - containerPort: 10250
            - containerPort: 10255
            env:
            - name: CONTENT_TYPE
              valueFrom:
                configMapKeyRef:
                  name: node-configmap
                  key: content.type
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            command: [
              "/kubemark",
              "--morph=kubelet",
              "--name=$(NODE_NAME)",
              "--kubeconfig=/kubeconfig/kubelet.kubeconfig",
              "$(CONTENT_TYPE)",
              "--logtostderr=true",
            ]
            volumeMounts:
            - name: kubeconfig-volume
              mountPath: /kubeconfig
              readOnly: true
            - name: logs-volume
              mountPath: /var/log
            - name: k8s-ssl
              mountPath: /opt/kubernetes/ssl
            resources:
              requests:
                cpu: 5m
                memory: 10000Ki
            securityContext:
              privileged: true
          tolerations:
            - key: "role"
              operator: "Equal"
              value: "real"
              effect: "NoSchedule"

可以看到注册出 hollow 节点

![20191023112324144.png][]

[Kubernetes scalability and performance SLIs_SLOs]: https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md
[perf-test_clusterloader2]: https://links.jianshu.com/go?to=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fperf-tests%2Ftree%2Fmaster%2Fclusterloader2
[20191023112324144.png]: /images/20230531/9005ebf9dcf445098e2c053055d5f4ed.png