kubernetes集羣實現jenkins slave

一、安裝Jenkins

1. 安裝存儲服務器

找一臺服務器搭建一臺nfs服務器

系統：Ubuntu 16.04

IP：172.18.1.13

apt install nfs-common nfs-kernel-server -y

#配置掛載信息
cat /etc/exports
/data/k8s *(rw,sync,no_root_squash)
#給目錄添加權限
chmod -R 777 /data/k8s
#啓動
/etc/init.d/nfs-kernel-server start
#開機啓動
systemctl enable nfs-kernel-server

2. kubernetes集羣安裝Jenkins

#該目錄下是jenkins.mytest.io爲測試域名的自籤密鑰
ls jenkins.mytest.io/
cacerts.pem  cacerts.srl  cakey.pem  create_self-signed-cert.sh  jenkins.mytest.io.crt  jenkins.mytest.io.csr  jenkins.mytest.io.key  openssl.cnf  tls.crt  tls.key
---
cd jenkins.mytest.io 
#創建Jenkins所在的namespace
kubectl create namespace kube-ops

#將密文添加到kube-ops裏面
# 服務證書和私鑰密文
kubectl -n kube-ops create \
    secret tls tls-jenkins-ingress \
    --cert=./tls.crt \
    --key=./tls.key
# ca證書密文
kubectl -n kube-ops create secret \
    generic tls-ca \
    --from-file=cacerts.pem

3. 創建Jenkins

cat jenkins-pvc.yaml

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  capacity:
    storage: 20Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 172.18.1.13
    path: /data/k8s

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins-pvc
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi

cat rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: kube-ops

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: kube-ops

cat jenkins.yaml

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: kube-ops
spec:
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkinshome
          subPath: jenkins
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: jenkins-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: kube-ops
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 30002
  - name: agent
    port: 50000
    targetPort: agent

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-lb
  namespace: kube-ops
spec:
  tls:
    - secretName: tls-jenkins-ingress
  rules:
  - host: jenkins.mytest.io
    http:
      paths:
      -  backend:
          serviceName: jenkins
          servicePort: 8080

創建Jenkins

kubectl create -f pvc.yaml
kubectl create -f rbac.yaml
kubectl create -f jenkins.yaml

二、jenkins配置

在/etc/hosts配置域名解析

kube-ip jenkins.mytest.io

1. 初始化配置

打開https://jenkins.mytes.io

安裝插件，選擇默認即可

2. 插件配置

採用Jenkins裏面的kubernetes插件，讓Jenkins可以調用kubernetes生成Jenkins-slave
https://github.com/jenkinsci/kubernetes-plugin

2.1 安裝kubernetes插件

Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin 勾選安裝即可。

2.2 配置kubernetes插件功能

Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 選擇 Kubernetes，然後填寫 Kubernetes 和 Jenkins 配置信息。
kubernetes地址採用了kube的服務器發現https://kubernetes.default.svc.cluster.local
namespace填kube-ops，然後點擊Test Connection，如果出現 Connection test successful 的提示信息證明 Jenkins 已經可以和 Kubernetes 系統正常通信
Jenkins URL 地址：http://jenkins.kube-ops.svc.cluster.local:8080

2.3 配置 kubernetes Pod Template

其實就是配置 Jenkins Slave 運行的 Pod 模板，命名空間我們同樣是用 kube-ops，Labels 這裏也非常重要，對於後面執行 Job 的時候需要用到該值，然後我們這裏使用的是 cnych/jenkins:jnlp 這個鏡像，這個鏡像是在官方的 jnlp 鏡像基礎上定製的，加入了 kubectl 等一些實用的工具。

2.4 添加容器的掛載卷

另外需要注意我們這裏需要在下面掛載兩個主機目錄，一個是 /var/run/docker.sock，該文件是用於 Pod 中的容器能夠共享宿主機的 Docker，這就是大家說的 docker in docker 的方式，Docker 二進制文件我們已經打包到上面的鏡像中了，另外一個目錄下 /root/.kube 目錄，我們將這個目錄掛載到容器的 /home/jenkins/.kube 目錄下面這是爲了讓我們能夠在 Pod 的容器中能夠使用 kubectl 工具來訪問我們的 Kubernetes 集羣，方便我們後面在 Slave Pod 部署 Kubernetes 應用。

2.5 添加賬號

另外一些同學在配置了後運行 Slave Pod 的時候出現了權限問題，因爲 Jenkins Slave Pod 中沒有配置權限，所以需要配置上 ServiceAccount，在 Slave Pod 配置的地方點擊下面的高級，添加上對應的 ServiceAccount 即可：
測試的時候不添加賬號會告知沒有權限

在容器模板高級裏面添加kubernetes集羣中創建的jenkins賬號

三、測試

創建一個測試任務

在pipeline的框裏面添加一下內容

def label = "jnlp-slave"
podTemplate(inheritFrom: 'jnlp-slave', instanceCap: 0, label: 'jnlp-slave', name: '', namespace: 'kube-ops', nodeSelector: '', podRetention: always(), serviceAccount: '', workspaceVolume: emptyDirWorkspaceVolume(false), yaml: '') {
    node(label) {
        container('jnlp-slave'){
            stage('Run shell') {
                sh 'docker info'
                sh 'kubectl get pods -n kube-ops'
            }
        }
    }
}

開始構建任務

構建任務輸出

Started by user admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
‘Jenkins’ doesn’t have label ‘jnlp-slave’
Agent jnlp-slave-tbdnl is provisioned from template Kubernetes Pod Template
Agent specification [Kubernetes Pod Template] (jnlp-slave): 
* [jnlp-slave] cnych/jenkins:jnlp

Running on jnlp-slave-tbdnl in /home/jenkins/workspace/test-jnlp-slave
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Run shell)
[Pipeline] sh
+ docker info
Containers: 15
 Running: 12
 Paused: 0
 Stopped: 3
Images: 12
Server Version: 18.09.6
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-1049-azure
Operating System: Ubuntu 16.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.768GiB
Name: test-kube-node-04
ID: YFTJ:FVHK:TAF3:HTAJ:HJ2A:5SFW:73RW:VQY5:Y64U:UGIR:KMJ2:XPRL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://kv3qfp85.mirror.aliyuncs.com/
Live Restore Enabled: false

WARNING: No swap limit support
[Pipeline] sh
+ kubectl get pods -n kube-ops
NAME                      READY     STATUS    RESTARTS   AGE
jenkins-6b874b8d7-q28h4   1/1       Running   0          3h
jnlp-slave-tbdnl          2/2       Running   0          15s
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

結束語

上面步驟大部分都是根據陽明博客上的《基於 Jenkins 的 CI/CD(一)》思路跟進，但是由於環境和自己認知問題，會出現各種出錯，憋了一天，沒什麼進展。

後來只能根據kubectl -n kube-ops logs -f jenkins-xxxxx的命令一點點查出來的，搜過很多帖子大概思路一致，但是無法解決本質問題，如果跑不起來，再高端也是個沒有用，後來根據kubernetes-plugin的github，具體讀了遍結合自己報的錯一點一點調整過來，總算搞出來了。

具體參考了以下幾遍優秀的文章：

基於 Jenkins 的 CI/CD(一)（陽明老師的文章很棒）

kubernetes Jenkins gitlab搭建CI/CD環境 (二)

GitHub-Jenkins-kubernetes-plugin：jenkinsci/kubernetes-plugin

rancher官網: 在Kubernetes上部署和擴展Jenkins

kubernetes集羣實現jenkins slave

一、安裝Jenkins

1. 安裝存儲服務器

2. kubernetes集羣安裝Jenkins

3. 創建Jenkins

二、jenkins配置

1. 初始化配置

2. 插件配置

2.1 安裝kubernetes插件

2.2 配置kubernetes插件功能

2.3 配置 kubernetes Pod Template

2.4 添加容器的掛載卷

2.5 添加賬號

三、測試

結束語

測試人員都是畫畫大神，讓我看看誰還不會用代碼圖？

Object.values()對象遍歷

《將博客搬至CSDN》

kubernetes實踐：使用nfs-client實現storageclass

kubernetes實踐：內部dns域名解析服務器

kubernetes實踐：helm安裝harbor

kubernetes實踐：安裝nfs服務器

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結