K8S CICD環境搭建

一 CICD思路

• 提交代碼到gitlab，人工審查代碼
• jenkins拉去代碼，編譯、單元測試、打包、push、部署。
• 用戶體驗(開發、測試、用戶)

二 Gitlab搭建

docker run -d \
    --hostname 192.168.217.141 \
    -p 880:80 \
    -p 8443:443 \
    -p 822:22 \
    --name gitlab \
    --restart unless-stopped \
    -v /root/gitlab/gitlab-config:/etc/gitlab \
    -v /root/gitlab/gitlab-logs:/var/log/gitlab \
    -v /root/gitlab/gitlab-data:/var/opt/gitlab \
    twang2218/gitlab-ce-zh:latest

訪問地址 http://ip:880

三 Harbor搭建

• 安裝docker環境
• 安裝docker-compose環境
• 下載harbor離線安裝包 harbor-offline-installer-v1.9.1.tgz
• 解壓 harbor-offline-installer-v1.9.1.tgz，進入harbor目錄
• 更改harbor.yml中的hostname
• 執行./install.sh --with-chartmuseum，安裝成功

訪問地址 http://ip:port 賬號admin 密碼默認Harbor12345(我這裏把harbor改成了8080端口)

四 jenkins安裝

4.1 helm安裝jenkins master

• 目錄結構

# tree .
.
├── Chart.yaml
├── OWNERS
├── README.md
├── templates
│   ├── deployment.yaml
│   ├── pvc.yaml
│   ├── rbac.yaml
│   └── svc.yaml
└── values.yaml

• deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: {{ .Release.Namespace }}
  name: jenkins
  labels:
    name: jenkins
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      name: jenkins
  template:
    metadata:
      labels:
        name: jenkins
    spec:
      nodeSelector:
        kubernetes.io/hostname: k8s-master
      serviceAccountName: jenkins
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts
          imagePullPolicy: Always
          env:
          ports:
            - containerPort: 8080
              name: http
            - containerPort: 50000
              name: slavelistener
          livenessProbe:
            httpGet:
              path: /login
              port: http
            initialDelaySeconds: 120
          readinessProbe:
            httpGet:
              path: /login
              port: http
            initialDelaySeconds: 120
          resources:
            requests:
              cpu: 0.5
              memory: 500Mi
            limits:
              cpu: 1
              memory: 1Gi
          env:
            - name: JAVA_OPTS
              value: -Xmx1000m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MAGIN0=0.85
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
              readOnly: false
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-home

• svc.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: {{ .Release.Namespace }}
  name: jenkins
  labels:
    name: jenkins
spec:
  ports:
    - port: 8080
      name: http
      targetPort: 8080
      nodePort: 30008
    - port: 50000
      name: httpslave
      targetPort: 50000
      nodePort: 32117
  selector:
    name: jenkins
  type: NodePort

• pvc.yaml 使用自動供給存儲類 grafana-nfs(NFS章節創建的)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-home
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: grafana-nfs
  resources:
    requests:
      storage: 5Gi

• rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: {{ .Release.Namespace }}
  name: jenkins

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  namespace: {{ .Release.Namespace }}
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: {{ .Release.Namespace }}

• 安裝jenkins master

# kubectl create ns jenkins
# helm install jenkins -n jenkins

• 替換插件下載地址爲國內地址，提高插件下載速度把$jenkins_home/updates/default.json文件按下面替換

把文件中http://www.google.com 替換成 http://www.baidu.com

把文件中http://updates.jenkins-ci.org/download 替換成 https://mirrors.tuna.tsinghua.edu.cn/jenkin

進入vi編輯器：

:1,$s/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g
:1,$s/http:\/\/www.google.com/https:\/\/www.baidu.com/g

重啓jenkins即可，瀏覽器輸入下面地址可以重啓 $jenkins_url/restart

• 安裝kubernetes、pipeline、git、git parameter插件

• 使用helm uninstall jenkins -n jenkins，重新掛載之前爲Released狀態的pv

需要kubectl edit pv pvxxxx， 刪除pv的claimRef，pv會變成avaliable狀態

這個時候再創建pvc就會與pv綁定，注意這個期間別用其他的pvc綁定這個pv，從而導致數據被覆蓋了

4.2 jenkins動態pod

• 自定義jenkins-slave鏡像

# tree .
.
├── Dockerfile
├── helm
├── jenkins-slave
├── kubectl
├── settings.xml
└── slave.jar
# docker build -t jenkins-slave:jdk1.8 .
# docker tag jenkins-slave:jdk1.8 192.168.217.142:8080/library/jenkins-slave:jdk1.8
# docker push 192.168.217.142:8080/library/jenkins-slave:jdk1.8  # 保證登陸了harbor倉庫

• slave.jar獲取地址爲 http://jenkins-ip:port/jnlpJars/slave.jar
• jenkins-slave 腳本獲取的地址爲 https://github.com/diodonfrost/docker-jenkins-slave
• Dockerfile加入了基礎環境，內容如下

FROM centos:7
LABEL maintainer pengjunjie

RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \
    yum clean all && \
    rm -rf /var/cache/yum/* && \
    mkdir -p /usr/share/jenkins

COPY slave.jar /usr/share/jenkins/slave.jar
COPY jenkins-slave /usr/bin/jenkins-slave
COPY settings.xml /etc/maven/settings.xml
RUN chmod +x /usr/bin/jenkins-slave
COPY helm kubectl /usr/bin/

ENTRYPOINT ["jenkins-slave"]

• 新建一個流水線項目，執行下面的pipeline，可以在服務器上看到動態創建的jenkins-slave pod

pipeline {
  agent {
    kubernetes {
      label 'jenkins-slave'
      yaml """
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: jenkins-slave
spec:
  containers:
  - name: jnlp
    image: 192.168.217.142:8080/library/jenkins-slave:jdk1.8
"""
    }
  }
  stages {
    stage('1、代碼編譯') {
      steps {
        echo "build"
      }
    }
    
    stage('2、單元測試') {
      steps {
        echo "test"
      }
    }
    
   stage('3、上線部署') {
      steps {
        echo "deploy"
      }
    }
  }
}