基於kubernetes構建動態Jenkins-slave
安裝配置 Master
-
創建pvc- 基於NFS的存儲類
--- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: jenkins-rbd-pvc spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 10Gi storageClassName: managed-nfs-storage -
創建RBAC
需要先創建namespace,這裏不寫在yaml裏，怕誤操作。
kubectl create ns devopsapiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: devops --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins rules: - apiGroups: ["extensions", "apps"] resources: ["deployments"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["services"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get","list","watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: jenkins namespace: devops roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins subjects: - kind: ServiceAccount name: jenkins namespace: devops -
創建Jenkins master的
jenkins-statefulset.yaml這裏我採用的是deployment，因爲我本地沒有存儲集羣，所以我這裏使用的是hostpath，也添加nodeSelector，只能調度到此節點，避免數據丟失。
--- apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: devops labels: app: jenkins spec: replicas: 1 selector: matchLabels: app: jenkins template: metadata: name: jenkins labels: app: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccountName: jenkins nodeSelector: jenkins: home containers: - name: jenkins image: jenkins/jenkins:lts imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: web protocol: TCP - containerPort: 50000 name: agent protocol: TCP resources: limits: cpu: 2000m memory: 4Gi requests: cpu: 500m memory: 512Mi volumeMounts: - name: jenkinshome subPath: jenkins mountPath: /var/jenkins_home env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai securityContext: fsGroup: 1000 volumes: - name: jenkinshome hostPath: path: /jenkins_home --- kind: Service apiVersion: v1 metadata: labels: app: jenkins name: jenkins namespace: devops spec: type: NodePort ports: - name: web port: 8080 targetPort: 8080 nodePort: 30086 - name: agent port: 50000 targetPort: 50000 nodePort: 30087 selector: app: jenkins如果使用了存儲類，可以參考如下配置文件。
--- apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: devops labels: app: jenkins spec: replicas: 1 selector: matchLabels: app: jenkins template: metadata: name: jenkins labels: app: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccountName: jenkins containers: - name: jenkins image: jenkins/jenkins:lts imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: web protocol: TCP - containerPort: 50000 name: agent protocol: TCP resources: limits: cpu: 2000m memory: 4Gi requests: cpu: 500m memory: 512Mi volumeMounts: - name: jenkinshome subPath: jenkins mountPath: /var/jenkins_home env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai securityContext: fsGroup: 1000 volumes: - name: jenkinshome persistentVolumeClaim: claimName: jenkins-rbd-pvc --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: jenkins-rbd-pvc namespace: devops spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 10Gi storageClassName: managed-nfs-storage --- kind: Service apiVersion: v1 metadata: labels: app: jenkins name: jenkins namespace: devops spec: type: NodePort ports: - name: web port: 8080 targetPort: 8080 nodePort: 30086 - name: agent port: 50000 targetPort: 50000 nodePort: 30087 selector: app: jenkins -
此時可以訪問k8s節點的nodePort端口，進行配置和驗證。
配置Jenkins cloud
需要安裝kubernetes相關插件。以及docker和pipeline相關的插件，可自行搜索。
在新版本的Jenkins當中，增加了Manage Nodes and Clouds，在此處配置我們的k8s集羣。配置如下圖所示：
添加Pod Labels
Jenkins在kubernetes集羣內部的話，是不需要進行證書配置的。
編寫測試的流水線
def label = "slave-${UUID.randomUUID().toString()}"
podTemplate(label: label, serviceAccount: 'jenkins', containers: [
containerTemplate(name: 'maven', image: 'maven:3.6-alpine', command: 'cat', ttyEnabled: true),
containerTemplate(name: 'docker', image: 'docker', command: 'cat', ttyEnabled: true),
containerTemplate(name: 'kubectl', image: 'cnych/kubectl', command: 'cat', ttyEnabled: true),
containerTemplate(name: 'helm', image: 'cnych/helm', command: 'cat', ttyEnabled: true)
], volumes: [
hostPathVolume(mountPath: '/root/.m2', hostPath: '/var/run/m2'),
hostPathVolume(mountPath: '/home/jenkins/.kube', hostPath: '/root/.kube'),
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock')
]) {
node(label) {
stage('單元測試') {
echo "測試階段"
}
stage('代碼編譯打包') {
container('maven') {
echo "打碼編譯打包階段"
}
}
stage('構建 Docker 鏡像') {
container('docker') {
echo "構建 Docker 鏡像階段"
}
}
stage('運行 Kubectl') {
container('kubectl') {
echo "查看 K8S 集羣 Pod 列表"
sh "whoami"
sh "echo $HOME"
sh "ls -l $HOME/.kube/config"
sh "sed -i 's/apiserver.k8s.local:8443/192.168.50.101:6443/g' $HOME/.kube/config"
sh "cat $HOME/.kube/config"
sh "ls -l $HOME/.kube/"
sh "kubectl get pods"
}
}
stage('運行 Helm') {
container('helm') {
echo "查看 Helm Release 列表"
sh "helm list"
}
}
}
}
PS: 這裏掛載宿主機的kubeconfig配置文件，可能需要所有的node節點 進行相關配置，因爲slave會動態的創建在隨機節點中。如果自行命令報錯。也可檢查Jenkins RBAC授權的權限是否足夠。