參考文檔
https://blog.csdn.net/aixiaoyang168/article/details/79767649
https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/kubernetes
https://www.cnblogs.com/cocowool/p/kubernetes_statefulset.html
https://www.cnblogs.com/cocowool/p/kubernetes_storage.html
文檔目錄
- kubernetes1.13.1+etcd3.3.10+flanneld0.10集羣部署
- kubernetes1.13.1部署kuberneted-dashboard v1.10.1
- kubernetes1.13.1部署coredns
- kubernetes1.13.1部署ingress-nginx並配置https轉發dashboard
- kubernetes1.13.1部署metrics-server0.3.1
- kubernetes1.13.1集羣使用ceph rbd存儲塊
- kubernetes1.13.1集羣結合ceph rbd部署最新版本jenkins
簡介
jenkins-kubernetes-plugin
Jenkins plugin to run dynamic agents in a Kubernetes cluster.
Based on the Scaling Docker with Kubernetes article, automates the scaling of Jenkins agents running in Kubernetes.
The plugin creates a Kubernetes Pod for each agent started, defined by the Docker image to run, and stops it after each build.
Agents are launched using JNLP, so it is expected that the image connects automatically to the Jenkins master. For that some environment variables are automatically injected:
- JENKINS_URL: Jenkins web interface url
- JENKINS_SECRET: the secret key for authentication
- JENKINS_AGENT_NAME: the name of the Jenkins agent
- JENKINS_NAME: the name of the Jenkins agent (Deprecated. Only here for backwards compatibility)
基本環境
k81集羣1.13.1版本
[root@elasticsearch01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.2.8.34 Ready <none> 25d v1.13.1
10.2.8.65 Ready <none> 25d v1.13.1
ceph集羣 luminous版本
[root@ceph01 ~]# ceph -s
services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03
mgr: ceph03(active), standbys: ceph02, ceph01
osd: 24 osds: 24 up, 24 in
rgw: 3 daemons active
操作步驟
一、使用ceph rbd創建pv、pvc
官網使用的是自帶創建pv與pvc這裏使用的是手動創建
1、創建pv
[root@elasticsearch01 jenkins]# cat jenkins-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-home-pv
spec:
capacity:
storage: 40Gi
accessModes:
- ReadWriteOnce
rbd:
monitors:
- '10.0.4.10:6789'
- '10.0.4.13:6789'
- '10.0.4.15:6789'
pool: rbd-k8s
image: cephimage3
user: admin
secretRef:
name: ceph-secret
fsType: ext4
readOnly: false
persistentVolumeReclaimPolicy: Recycle
[root@elasticsearch01 jenkins]# kubectl create -f jenkins-pv.yaml
persistentvolume/jenkins-home-pv created
[root@elasticsearch01 jenkins]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ceph-rbd-pv 20Gi RWO Recycle Bound default/ceph-rbd-pv-claim 22h
jenkins-home-pv 40Gi RWO Recycle Available 4s
2、創建pvc
[root@elasticsearch01 jenkins]# cat jenkins-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-home-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
[root@elasticsearch01 jenkins]# kubectl create -f jenkins-pvc.yaml
persistentvolumeclaim/jenkins-home-pvc created
[root@elasticsearch01 jenkins]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-rbd-pv-claim Bound ceph-rbd-pv 20Gi RWO 22h
jenkins-home-pvc Bound jenkins-home-pv 40Gi RWO 3s
[root@elasticsearch01 jenkins]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ceph-rbd-pv 20Gi RWO Recycle Bound default/ceph-rbd-pv-claim 22h
jenkins-home-pv 40Gi RWO Recycle Bound default/jenkins-home-pvc 77s
二、跟進實際情況修改jenkins.yml文件
主要修改的配置從上到下分別是:
1、拉取鏡像策略
imagePullPolicy: IfNotPresent
2、自動存儲storage class改成voulumes的pvc方式實現
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins-home-pvc
3、ingress的host改成實際的
host: jenkins.minminmsn.com
4、ingres的tls證書改成實際的
tls:
- hosts:
- jenkins.minminmsn.com
secretName: ingress-secret
5、具體如下
[root@elasticsearch01 jenkins]# cat jenkins.yml
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: jenkins
labels:
name: jenkins
spec:
serviceName: jenkins
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: jenkins
labels:
name: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: jenkins
containers:
- name: jenkins
image: jenkins/jenkins:lts-alpine
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
- containerPort: 50000
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 0.5
memory: 500Mi
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
# value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
readOnly: false
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
securityContext:
fsGroup: 1000
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins-home-pvc
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
spec:
# type: LoadBalancer
selector:
name: jenkins
# ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s >=1.7)
#externalTrafficPolicy: Local
ports:
-
name: http
port: 80
targetPort: 8080
protocol: TCP
-
name: agent
port: 50000
protocol: TCP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: jenkins
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/tls-acme: "true"
# "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
# For nginx-ingress controller < 0.9.0.beta-18
ingress.kubernetes.io/ssl-redirect: "true"
# "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
ingress.kubernetes.io/proxy-body-size: 50m
ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: jenkins
servicePort: 80
host: jenkins.minminmsn.com
tls:
- hosts:
- jenkins.minminmsn.com
secretName: ingress-secret
三、創建狀態集、svc、pod、ingress
1、創建rbac認證角色
[root@elasticsearch01 jenkins]# kubectl create -f service-account.yml
serviceaccount/jenkins created
role.rbac.authorization.k8s.io/jenkins created
rolebinding.rbac.authorization.k8s.io/jenkins created
2、創建jenkins服務等
[root@elasticsearch01 jenkins]# kubectl create -f jenkins.yml
statefulset.apps/jenkins created
service/jenkins created
ingress.extensions/jenkins created 4s
[root@elasticsearch01 jenkins]# kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 454 18d
ceph-rbd-pv-pod1 1/1 Running 1 21h
jenkins-0 0/1 ContainerCreating 0 7s
[root@elasticsearch01 jenkins]# kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 454 18d
ceph-rbd-pv-pod1 1/1 Running 1 21h
jenkins-0 1/1 Running 0 4m52s
四、通過ingress訪問
獲取ingress-nginx對外端口,https://jenkins.minminmsn.com:47215/訪問即可,需要配置dns解析到pod所在node的ip
[root@elasticsearch01 jenkins]# kubectl get svc -n ingress-nginx|grep ingress-nginx
ingress-nginx LoadBalancer 10.254.125.151 <pending> 80:33003/TCP,443:47215/TCP 14d
五、初始化jenkins
1、查找密碼
[root@elasticsearch02 ~]# df -h|grep rbd
/dev/rbd0 493G 163G 306G 35% /data
/dev/rbd1 20G 45M 20G 1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage2
/dev/rbd2 40G 138M 40G 1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage3
[root@elasticsearch02 ~]# cd //var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage3
[root@elasticsearch02 rbd-k8s-image-cephimage3]# ls
config.xml init.groovy.d jobs nodes secrets war
copy_reference_file.log jenkins.CLI.xml logs plugins updates
hudson.model.UpdateCenter.xml jenkins.install.UpgradeWizard.state lost+found secret.key userContent
identity.key.enc jenkins.telemetry.Correlator.xml nodeMonitors.xml secret.key.not-so-secret users
[root@elasticsearch02 rbd-k8s-image-cephimage3]# cat secrets/initialAdminPassword
92c145b796cc48b0af8b5ef0f7afce28
2、選擇安裝插件
3、創建初始管理賬號
4、設置jenkins url默認https://jenkins.minminmsn.com:47215/
5、開始使用jenkins
6、jenkins控制檯界面,主要配置都在系統管理中
[圖片上傳失敗...(image-ce2d85-1548296164745)]
總結
使用ceph rbd 這種只能讀寫一次的設備不能用在線上,線上應該使用分佈式存儲例如nfs,cephfs,glusterfs等,這裏只是測試jenkins結合ceph,pv,pvc完成有狀態pod的測試