k8s環境部署及使用方式

安裝k8s+docker集羣環境（基於centos7系統）

集羣機器：
centos-master = 192.168.121.9
centos-minion-1 = 192.168.121.65
centos-minion-2 = 192.168.121.66
centos-minion-3 = 192.168.121.67

1.配置yum源
[centos-master]:cat /etc/yum.repos.d/virt7-docker-common-release.repo

[virt7-docker-common-release] name=virt7-docker-common-release baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/ gpgcheck=0

加載安裝包：
[centos-master]:yum repolist

virt7-docker-common-release virt7-docker-common-release 15

2.安裝集羣必要軟件—-etcd/flannel/kubernetes
Etcd服務在k8s集羣中用於配置共享和服務發現。

Flannel是針對k8s設計一個網絡規劃服務，讓集羣中的不同節點主機創建的Docker容器都具有全集羣唯一的虛擬IP地址。

[centos-master]:yum -y install –enablerepo=virt7-docker-common-release kubernetes etcd flannel

3.如果集羣中沒有使用DNS解析，那麼需要在master節點的/etc/hosts中添加node的主機名信息，比如：

echo "192.168.121.9 centos-master
192.168.121.65 centos-minion-1
192.168.121.66 centos-minion-2
192.168.121.67 centos-minion-3" >> /etc/hosts

4.修改配置master節點的kubernetes配置文件/etc/kubernetes/config

#表示錯誤日誌記錄到文件還是輸出到stderr
KUBE_LOGTOSTDERR="--logtostderr=true" #日誌等級
KUBE_LOG_LEVEL="--v=0" #允許運行特權容器
KUBE_ALLOW_PRIV="--allow-privileged=false" #apiserver的服務地址，controller-manager、scheduler及kubelet都會用到這個配置，這裏配置爲192.168.121.9
KUBE_MASTER="--master=http://192.168.121.9:8080"

5.k8s集羣中涉及的端口比較多，所以centos中的防火牆需要設置對應規則，並需關閉selinux。爲確保k8s集羣的正常運行，我們可以直接關閉iptables與seliinux服務。

setenforce 0
systemctl stop firewalld.service
systemctl stop firewalld.service

6.修改配置master節點的etcd配置文件/etc/etcd/etcd.conf
etcd服務的可調參數比較多，根據需求開啓對應功能，此處我們大概調整如下幾個功能：

# [member] #etcd名稱
ETCD_NAME=default #etcd數據存儲位置
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #監聽的端口
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" #[cluster] #集羣監聽的端口
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

7.修改配置master節點的apiserver配置文件/etc/kubernetes/apiserver

#監聽的接口，如果配置爲127.0.0.1則只監聽localhost，配置爲0.0.0.0會監聽所有接口，這裏配置爲0.0.0.0
KUBE_API_ADDRESS="--address=0.0.0.0" #apiserver的監聽端口，默認8080
KUBE_API_PORT="--port=8080" #minion上kubelet監聽的端口，默認10250
KUBELET_PORT="--kubelet-port=10250" #etcd服務地址，前面已經啓動了etcd服務,端口爲2379
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379" #kubernetes可以分配的ip的範圍，kubernetes啓動的每一個pod以及serveice都會分配一個ip地址，將從這個範圍分配
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" #需要額外添加的配置項，簡單地啓用一個集羣無需配置
KUBE_API_ARGS=""

8.啓動並且配置etcd的網段，此網段一定是未被使用的

systemctl start etcd
etcdctl mkdir /kube-centos/network
etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"

9.修改配置master節點的flanneld配置文件/etc/sysconfig/flanneld

#etcd的訪問地址及端口
FLANNEL_ETCD_ENDPOINTS="http://192.168.121.9:2379" #服務範圍
FLANNEL_ETCD_PREFIX="/kube-centos/network" #其他
FLANNEL_OPTIONS=""

10.啓動k8s集羣

for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler flanneld; do
 systemctl restart $SERVICES
 systemctl enable $SERVICES
 systemctl status $SERVICES
done

以上就是k8s，master節點的安裝及配置
接下來，配置nodes節點

11.修改配置nodes節點kubelet的配置文件/etc/kubernetes/kubelet

#minion監聽的地址，每個minion根據實際的ip配置,這裏配置爲0.0.0.0
KUBELET_ADDRESS="--address=0.0.0.0" #監聽的端口
KUBELET_PORT="--port=10250" #apiserver的訪問地址及端口
KUBELET_API_SERVER="--api-servers=http://192.168.121.9:8080" #額外增加的參數
KUBELET_ARGS="--logtostderr=false --v=0 --log-dir=/data/logs/kubernetes"

12.修改配置nodes節點flanneld的配置文件/etc/sysconfig/flanneld

#etcd的訪問地址及端口
FLANNEL_ETCD="http://192.168.121.9:2379" #etcd服務範圍
FLANNEL_ETCD_KEY="/kube-centos/network"

13.啓動k8s集羣服務

for SERVICES in kube-proxy kubelet flanneld docker; do
 systemctl restart $SERVICES
 systemctl enable $SERVICES
 systemctl status $SERVICES done

14.設置kubectl的配置文件

kubectl config set-cluster default-cluster --server=http://192.168.121.9:8080
kubectl config set-context default-context --cluster=default-cluster --user=default-admin
kubectl config use-context default-context

15.檢查集羣狀態
[centos-master]:kubectl get nodes

NAME STATUS AGE VERSION
centos-minion-1 Ready 3d v1.5.0+fff5156
centos-minion-2 Ready 3d v1.5.0+fff5156
centos-minion-3 Ready 3d v1.5.0+fff5156

至此，集羣構建完畢

搭建私有庫

私有庫用於系統內部存儲成品鏡像，能夠快速進行下載及被k8s調度。

1.下載並啓動私有庫

[centos-master]:docker run --name registry -v /etc/localtime:/etc/localtime -v /opt/registry:/var/lib/registry -p 5000:5000 -itd docker.io/registry

#--name 表示啓動的容器後名稱，此處爲registry #-v 表示掛載路徑 格式爲宿主機路徑:容器內路徑 #-p 表示映射端口 格式爲宿主機端口:容器內端口 #-itd docker的內部參數，此處聲明後臺運行容器並分配一個僞終端並綁定到容器的標準輸入上，後跟鏡像名稱此處爲docker.io/registry

2.創建一個secret服務，用於k8s調度私有庫容器時的“令牌”。簡單來說，secret服務就是一個存儲密碼的服務

[centos-master]:kubectl create secret docker-registry registrykey --docker-server=registry.evehicle.cn --docker-username=docker --docker-password=docker [email protected] 
[centos-master]:kubectl get secret
NAME TYPE DATA AGE
registrykey kubernetes.io/dockercfg 1 6s

此時登錄時會提示認證錯誤

[centos-master]:docker login -u docker -p docker -e [email protected] registry.evehicle.cn
Flag --email has been deprecated, will be removed in 1.13.
Error response from daemon: login attempt to https://registry.evehicle.cn/v2/ failed with status: 401 Unauthorized

這是因爲Docker官方是推薦採用Secure Registry的工作模式的，即transport採用tls。這樣我們就需要爲Registry配置tls所需的key和crt文件了

3.配置nginx反向代理
[centos-master]: cat registry.evehicle.cn.conf

# For versions of nginx > 1.3.9 that include chunked transfer encoding support # Replace with appropriate values where necessary upstream docker-registry {
 server 192.168.121.9:5000;
 #server 10.44.170.95:5000;
}

# uncomment if you want a 301 redirect for users attempting to connect # on port 80 # NOTE: docker client will still fail. This is just for convenience # server { # listen *:80; # server_name my.docker.registry.com; # return 301 https://$server_name$request_uri; # } server {
 listen 443;
 server_name registry.evehicle.cn;

 ssl on;
 ssl_certificate ssl/registry.evehicle.cn.crt;
 ssl_certificate_key ssl/registry.evehicle.cn.key;

 client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486) chunked_transfer_encoding on;

 location / {
 auth_basic "Restricted";
 auth_basic_user_file passwd;
 add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

 proxy_pass http://docker-registry;
 proxy_set_header Host $http_host; # required for docker client's sake proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_read_timeout 900;
 }

 location /_ping {
 auth_basic off;
 include docker-registry.conf;
 }

 location /v1/_ping {
 auth_basic off;
 include docker-registry.conf;
 }

 location /v2/_ping {
 auth_basic off;
 include docker-registry.conf;
 }
}

將key及crt證書文件放到../ssl目錄下。使用htpasswd生成密碼放於./上一級目錄

 htpasswd -bcm passwd docker docker
 #-c：創建一個加密文件 #-m：md5加密，默認可不填寫 #-b：表示用戶名密碼在命令行中一併輸入，不用分別填寫

4.再次登錄

[centos-master]:docker login -u docker -p docker -e [email protected] registry.evehicle.cn 
Login Succeeded
表示成功，此時再pull\push既在私有庫中進行

構建服務

docker的本意是將代碼包含在容器內製作成鏡像形成“產品”。但出於公司的（頻繁修改代碼及服務器資源受限）的特殊性，我們將代碼以“外掛”的形式運行在宿主機上。下面以部署官網（apache）服務爲例：
1.從docker的公有庫裏下載centos7的原生鏡像

[centos-master]:docker pull centos

Using default tag: latest
Trying to pull repository docker.io/library/centos ...
latest: Pulling from docker.io/library/centos
d9aaf4d82f24: Downloading [> ] 540 kB/73.39 MB
d9aaf4d82f24: Pulling fs layer
Digest: sha256:eba772bac22c86d7d6e72421b4700c3f894ab6e35475a34014ff8de74c10872e
Status: Downloaded newer image for centos:latest

2.編寫Dockerfile製造apache基礎鏡像

######httpd####
FROM centos
MAINTAINER lienhua lienhua@zhongchuangsanyou.com
RUN yum -y install epel-release
RUN yum -y install httpd php php-mysql php-memcache* php-mbstring
ADD httpd.conf /etc/httpd/conf/httpd.conf

EXPOSE 80

CMD ["/usr/sbin/apachectl", "-D", "FOREGROUND"]

其中httpd.conf文件需要在當前目錄下真實存在，此處其內容爲

ServerRoot "/etc/httpd" Listen 80
Listen 8080
Include conf.modules.d/*.conf
Include zcsy/*.conf
User apache
Group apache
ServerAdmin root@localhost
<Directory /> AllowOverride none
 Require all denied
</Directory> DocumentRoot "/var/www/html" <Directory "/var/www"> AllowOverride None
 Require all granted
</Directory> <Directory "/var/www/html"> Options Indexes FollowSymLinks
 AllowOverride None
 Require all granted
</Directory> <IfModule dir_module> DirectoryIndex index.html
</IfModule> <Files ".ht*"> Require all denied
</Files> ErrorLog "logs/error_log" LogLevel warn
<IfModule log_config_module> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
 LogFormat "%h %l %u %t \"%r\" %>s %b" common
 <IfModule logio_module> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
 </IfModule> CustomLog "logs/access_log" combined
</IfModule> <IfModule alias_module> ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" </IfModule> <Directory "/var/www/cgi-bin"> AllowOverride None
 Options None
 Require all granted
</Directory> <IfModule mime_module> TypesConfig /etc/mime.types
 AddType application/x-compress .Z
 AddType application/x-gzip .gz .tgz
 AddType application/x-httpd-php .php
 AddType text/html .shtml
 AddOutputFilter INCLUDES .shtml
</IfModule> AddDefaultCharset UTF-8
<IfModule mime_magic_module> MIMEMagicFile conf/magic
</IfModule> EnableSendfile off EnableMMAP off IncludeOptional conf.d/*.conf

執行[centos-master]:docker build -t registry.evehicle.cn/httpd . 命令製作名爲”registry.evehicle.cn/httpd”的鏡像（注意此處的點必須要有，並且其意義代表當前目錄下的Dockerfile文件）

3.將製作好的鏡像上傳到私有庫

docker push registry.evehicle.cn/httpd

4.編寫啓動apache服務的yaml文件

[centos-master]:cat 13-rc-httpd.yaml

apiVersion: v1
kind: ReplicationController
metadata:
 name: 13-rc-httpd
 labels:
 name: 13-rc-httpd
spec:
 replicas: 2
 selector:
 name: 13-rc-httpd
 template:
 metadata:
 labels:
 name: 13-rc-httpd
 spec:
 containers:
 - name: 13-rc-httpd
 image: registry.evehicle.cn/httpd
 env:
 - name: LANG  value: en_US.UTF-8
 ports:
 - containerPort: 80  hostPort: 80
 volumeMounts:
 - name: time
 mountPath: /etc/localtime
 - name: zcsy
 mountPath: /etc/httpd/zcsy
 - name: deploy
 mountPath: /docker/httpd/deploy
 - name: log
 mountPath: /var/log/httpd
 volumes:
 - name: time
 hostPath:
 path: /etc/localtime
 - name: zcsy
 hostPath:
 path: /docker/httpd/zcsy
 - name: deploy
 hostPath:
 path: /docker/httpd/deploy
 - name: log
 hostPath:
 path: /docker/httpd/log
 nodeSelector:
 slave: "13"
 imagePullSecrets:
 - name: registrykey

5.給其中一個node加上標籤爲“13”

kubectl label nodes centos-minion-1 slave=13

6.此時擁有標籤“13”的nodes應具備的條件

/docker/httpd/zcsy下需要有官網的配置文件

<VirtualHost *:80> ServerName www.evehicle.cn
 DocumentRoot /var/deploy/wordpress/
 RewriteEngine on RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
 RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
 RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !^.*\.(ico|pdf|flv|jpe?g|js|gif|png|html|shtml|zip|xml|gz|rar|swf|txt|apk|bmp|css|m4a|ogg|mp3|ipa|plist)$
 RewriteCond %{REQUEST_URI} !^/server-status$
 RewriteRule . /index.php [QSA,PT,L] </VirtualHost> <Directory /var/deploy/wordpress/> Options FollowSymLinks
 AllowOverride All Require all granted
</Directory>

以及/docker/httpd/deploy下需要有官網的代碼

7.運行yaml文件啓動容器

[centos-master]: kuberctl create -f 13-rc-httpd.yaml

8.查看服務

[centos-master]: kuberctl get rc

NAME DESIRED CURRENT AGE
13-rc-httpd 2 2 168d

9.程序中涉及的mysql\redis\memcache等服務也需使用容器運行起來

[centos-master]: docker pull redis
[centos-master]: docker tag registry.evehicle.cn/redis redis
[centos-master]: docker push registry.evehicle.cn/redis
[centos-master]: kubectl create -f rc-redis.yaml
[centos-master]: cat rc-redis.yaml

apiVersion: v1
kind: ReplicationController
metadata:
 name: redis
 labels:
 name: redis
spec:
 replicas: 2
 selector:
 name: redis
 template:
 metadata:
 labels:
 name: redis
 spec:
 containers:
 - name: redis
 image: registry.evehicle.cn/redis
 ports:
 - containerPort: 6379  hostPort: 6379
 volumeMounts:
 - name: data
 mountPath: /data
 - name: time
 mountPath: /etc/localtime
 volumes:
 - name: data
 hostPath:
 path: /docker/redis/6379
 - name: time
 hostPath:
 path: /etc/localtime
 nodeSelector:
 slave: "13"
 imagePullSecrets:
 - name: registrykey

啓動memcache
[centos-master]: docker pull memcache
[centos-master]: docker tag registry.evehicle.cn/memcached memcache
[centos-master]: docker push registry.evehicle.cn/memcached
[centos-master]: kubectl create -f rc-memcached.yaml
[centos-master]: cat rc-memcached.yaml

apiVersion: v1
kind: ReplicationController
metadata:
 name: memcached
 labels:
 name: memcached
spec:
 replicas: 3
 selector:
 name: memcached
 template:
 metadata:
 labels:
 name: memcached
 spec:
 containers:
 - name: memcached
 image: registry.evehicle.cn/memcached
 ports:
 - containerPort: 11211
 hostPort: 11211 #nodeSelector: # slave: "13"
 imagePullSecrets:
 - name: registrykey

製造mysql鏡像
[centos-master]: cat Dockerfile

FROM alpine


COPY startup.sh /startup.sh
RUN addgroup mysql && \
 adduser -H -D -s /bin/false -G mysql mysql && \
 apk add --update mysql mysql-client && rm -f /var/cache/apk/* && \
 mkdir /data && \
 chown -R mysql:mysql /data /etc/mysql && \
 chmod 755 /startup.sh \
 ;


WORKDIR /data VOLUME /data VOLUME /etc/mysql


EXPOSE 3306 CMD ["/startup.sh"]

啓動mysql（建議mysql在宿主機啓動）
[centos-master]: docker build -t registry.evehicle.cn/mysql
[centos-master]: docker push registry.evehicle.cn/mysql
[centos-master]: kubectl create -f rc-mysql.yaml
[centos-master]: cat rc-mysql.yaml

apiVersion: v1
kind: ReplicationController
metadata:
 name: 13-rc-mysql
 labels:
 name: 13-rc-mysql
spec:
 replicas: 2
 selector:
 name: 13-rc-mysql
 template:
 metadata:
 labels:
 name: 13-rc-mysql
 spec:
 containers:
 - name: 13-rc-mysql
 image: registry.evehicle.cn/mysql
 env:
 - name: MYSQL_DATABASE  value: admin
 - name: MYSQL_USER  value: tony
 - name: MYSQL_PASSWORD  value: 456
 - name: MYSQL_ROOT_PASSWORD  value: 123
 ports:
 - containerPort: 3306  hostPort: 3306
 volumeMounts:
 - name: time
 mountPath: /etc/localtime
 - name: data
 mountPath: /data
 - name: etc
 mountPath: /etc/mysql
 - name: run
 mountPath: /run/mysqld
 volumes:
 - name: time
 hostPath:
 path: /etc/localtime
 - name: data
 hostPath:
 path: /docker/mysql/data
 - name: etc
 hostPath:
 path: /docker/mysql/etc
 - name: run
 hostPath:
 path: /docker/mysql/run
 nodeSelector:
 slave: "13"
 imagePullSecrets:
 - name: registrykey

爲方便代碼編寫及統一管理，應提前做好內部DNS解析。將所負責的應用規整到對應的機器上。

本文轉自CSDN-k8s環境部署及使用方式

k8s環境部署及使用方式

搭建私有庫

構建服務

如何使用 JS 判斷用戶是否處於活躍狀態

通過HPA+CronHPA組合應對業務複雜彈性伸縮場景

❤️‍🔥 Solon Cloud Event 新的事務特性與應用

DockOne微信分享（一八三）：滴滴彈性雲Kubernetes實踐

Rook：基於Ceph的Kubernetes存儲解決方案

超越Kubernetes：值得關注的5大雲原生技術

選擇Serverless還是Kubernetes？這種爭辯並沒有意義

Docker和Kubernetes如何讓DevOps更具效力

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結