安裝jenkins
1、創建一個命名空間
$ kubectl create namespace kube-ops
2、爲jenkins創建pvc（也可以使用存儲類創建）
apiVersion: v1
kind: PersistentVolume
metadata:
name: opspv
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Delete
nfs:
server: 192.168.1.244
path: /data/k8s
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: opspvc
namespace: kube-ops
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi3、創建jenkins需要的rbac權限
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins2
namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins2
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins2
namespace: kube-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins2
subjects:
- kind: ServiceAccount
name: jenkins2
namespace: kube-ops也可以爲ServiceAccoun綁定一個系統現有的 cluster-admin 集羣角色權限
4、創建jenkins pod
$ docker pull docker.io/jenkins/jenkins:lts
$ docker pull cnych/jenkins:jnlp6
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins2
namespace: kube-ops
spec:
template:
metadata:
labels:
app: jenkins2
spec:
terminationGracePeriodSeconds: 10
serviceAccount: jenkins2
containers:
- name: jenkins
image: docker.io/jenkins/jenkins:lts
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
volumeMounts:
- name: jenkinshome
subPath: jenkins2
mountPath: /var/jenkins_home
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvision
er.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai securityContext:
fsGroup: 1000
volumes:
- name: jenkinshome
persistentVolumeClaim:
claimName: opspvc
---
apiVersion: v1
kind: Service
metadata:
name: jenkins2
namespace: kube-ops
labels:
app: jenkins2
spec:
selector:
app: jenkins2
type: NodePort
ports:
- name: web
port: 8080
targetPort: web
nodePort: 30003
- name: agent
port: 50000
targetPort: agent$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 0/1 Running 0 #不能正常啓動
$ kubectl describe pod jenkins2-76644dbc9b-llcsp -n kube-ops
$ kubectl logs -f jenkins2-76644dbc9b-llcsp -n kube-ops
5、在nfs服務器上修改jenkins持久目錄的權限並重新創建jenkins pod
$ chown -R 1000 /data/k8s/jenkins2 #在192.168.1.244上
$ kubectl delete -f jenkins2.yaml
$ kubectl apply -f jenkins2.yaml
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0
爲什麼是1000？
上述鏡像的Dockerfile文件中定義的是：user=jenkins group=jenkins uid=1000 gid=1000
Dockerfile文件的地址：
https://github.com/jenkinsci/docker/blob/master/Dockerfile
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000
ARG http_port=8080
ARG agent_port=50000
ARG JENKINS_HOME=/var/jenkins_home
$ kubectl get svc -n kube-ops
jenkins2 NodePort 10.105.121.176 <none> 8080:30003/TCP,50000:30936/TCP
http://192.168.1.243:30003
初始密碼在nfs服務器上
$ cat /data/k8s/jenkins/secrets/initialAdminPassword
在jenkins上創建kubernetes雲
enkins Master 和 Jenkins Slave 以 Pod 形式運行在 Kubernetes 集羣的 Node 上，Master 運行在其中一個節點，並且將其配置數據存儲到一個 Volume 上去，Slave 運行在各個節點上，並且它不是一直處於運行狀態，它會按照需求動態的創建並自動刪除
這種方式的工作流程大致爲：當 Jenkins Master 接受到 Build 請求時，會根據配置的 Label 動態創建一個運行在 Pod 中的 Jenkins Slave 並註冊到 Master 上，當運行完 Job 後，這個 Slave 會被註銷並且這個 Pod 也會自動刪除，恢復到最初狀態。
1、安裝插件
安裝kubernetes plugin， 點擊 Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin
2、增加kubernetes雲
點擊 Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 選擇 Kubernetes，然後填寫 Kubernetes 和 Jenkins 配置信息----連接測試
name：kubernetes
Kubernetes 地址：https://kubernetes.default.svc.cluster.local
Kubernetes 命名空間：kube-ops
Jenkins 地址：http://jenkins2.kube-ops.svc.cluster.local:8080（jenkins2是svc）
3、添加pod模板
添加pod模板----Kubernetes Pod Template
名稱：jnlp
命名空間：kube-ops
標籤列表：dongyali-jnlp
4、添加容器模板
添加容器----Container Template
名稱：jnlp
Docker 鏡像：cnych/jenkins:jnlp6（Jenkins 版本在 2.176.x以下的鏡像名字去掉6）
工作目錄：/home/jenkins/agent
運行的命令：清空
命令參數：清空
5、添加兩個卷
添加捲----Host Path Volume
主機路徑：/var/run/docker.sock
掛載路徑：/var/run/docker.sock
主機路徑：/root/.kube
掛載路徑：/root/.kube
6、可能需要配置ServiceAccount
$ kubectl get sa -n kube-ops
jenkins2 1 14h
點擊添加捲下面的高級----Service Account----jenkins2
7、用shell測試 Kubernetes 動態生成 jenkins slave
新建任務----名字----自由風格
通用----勾選限制項目的運行節點----標籤表達式：dongyali-jnlp
構建----執行shell----輸入如下內容----保存立即構建
echo "測試 Kubernetes 動態生成 jenkins slave"
echo "==============docker in docker==========="
docker info
echo "=============kubectl============="
kubectl get pods觀察 Kubernetes 集羣中 Pod 的變化：
$ kubectl get pod -n kube-ops
jenkins2-76644dbc9b-llcsp 1/1 Running 0 3h49m
jnlp-tl1km 1/1 Running 0 44s
當任務運行完畢，jnlp這個slave pod就會自動消失。
8、用pipeline測試 Kubernetes 動態生成 jenkins slave
新建任務----名字----流水線
在流水線腳本中輸入如下內容----保存立即構建
node('dongyali-jnlp') {
stage('Clone') {
echo "1.Clone Stage"
}
stage('Test') {
echo "2.Test Stage"
}
stage('Build') {
echo "3.Build Stage"
}
stage('Deploy') {
echo "4. Deploy Stage"
}
}