生產項目容器化改造(一)環境部署

主機列表:

192.168.1.248 CentOS7 DockerJenkins 4U8G 3*50G Disk
192.168.1.249 CentOS7 DockerGitlab  4U8G
192.168.1.250 CentOS7 DockerELK     4U8G
192.168.1.171 CentOS7 DockerServer1 8U16G
192.168.1.172 CentOS7 DockerServer2 8U16G
192.168.1.173 CentOS7 DockerServer3 8U16G
192.168.1.174 CentOS7 DockerServer4 8U16G
192.168.1.175 CentOS7 DockerServer5 8U16G

部署GitLab：

• 依據《CentOS7 gitlab安裝搭建簡單維護》和《CentOS7 gitlab支持https的改造》部署一臺Gitlab
• 相應的URL爲：https://gitlab.vincent.com，登陸 192.168.1.249 進行部署，全部步驟如下：

# 主機名和hosts解析
HOSTNAME=dockergitlab
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME gitlab.vincent.com">>/etc/hosts

# 創建密鑰證書目錄
mkdir -p /etc/gitlab/ssl && cd /etc/gitlab/ssl
openssl genrsa -out "/etc/gitlab/ssl/gitlab.vincent.com.key" 2048
openssl req -new \
  -key "/etc/gitlab/ssl/gitlab.vincent.com.key" \
  -out "/etc/gitlab/ssl/gitlab.vincent.com.csr"
openssl x509 -req -days 365 \
  -in "/etc/gitlab/ssl/gitlab.vincent.com.csr" \
  -signkey "/etc/gitlab/ssl/gitlab.vincent.com.key" \
  -out "/etc/gitlab/ssl/gitlab.vincent.com.crt"
openssl dhparam -out /etc/gitlab/ssl/dhparams.pem 2048
chmod 600 *

# 下載並安裝gitlab
cd /tmp
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-12.10.6-ce.0.el7.x86_64.rpm
yum -y localinstall gitlab-ce-12.10.6-ce.0.el7.x86_64.rpm
gitlab-ctl reconfigure
gitlab-ctl status|column -t
systemctl enable gitlab-runsvdir.service

# 配置gitlab
sed -i "s|^external_url.*$|# &\n\
external_url 'https://gitlab.vincent.com'|g" /etc/gitlab/gitlab.rb
sed -i "s|^# nginx\['enable'\] = true$|\
nginx['redirect_http_to_https'] = true\n\
nginx['ssl_certificate'] = \"/etc/gitlab/ssl/gitlab.vincent.com.crt\"\n\
nginx['ssl_certificate_key'] = \"/etc/gitlab/ssl/gitlab.vincent.com.key\"\
\n&|g" /etc/gitlab/gitlab.rb
sed -i "s|^# nginx\['ssl_dhparam'\] = nil|\
# nginx\['ssl_dhparam'\] = /etc/gitlab/ssl/dhparams.pem|g" /etc/gitlab/gitlab.rb
cat >>/etc/gitlab/gitlab.rb<<EOF
# mail alert setup
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'smtp.126.com'
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_user_name'] = '[email protected]'
gitlab_rails['smtp_password'] = 'xxxx'
gitlab_rails['smtp_authentication'] = 'login'
gitlab_rails['smtp_enable_starttls_auto']= true
gitlab_rails['gitlab_email_from']= '[email protected]'
gitlab_rails['gitlab_email_reply_to']= '[email protected]'
EOF
gitlab-ctl reconfigure
gitlab-ctl restart

# 客戶機添加hosts解析，使用瀏覽器訪問：https://gitlab.vincent.com，配置root密碼
# 導入各個maven項目

部署OPS主機：

提前將數據盤掛載到各個目錄

• 依據《CentOS7實驗機模板搭建部署》部署實驗機：192.168.1.248，分配3塊數據盤並掛載到相應目錄

echo -e 'n\np\n\n\n\nw\n'|fdisk /dev/sdb
echo -e 'n\np\n\n\n\nw\n'|fdisk /dev/sdc
echo -e 'n\np\n\n\n\nw\n'|fdisk /dev/sdd
mkfs.ext4 /dev/sdb1
mkfs.ext4 /dev/sdc1
mkfs.ext4 /dev/sdd1
mkdir -pv /usr/share/nginx/html /var/lib/jenkins /var/lib/docker
echo '/dev/sdb1 /usr/share/nginx/html ext4 defaults 0 0' >>/etc/fstab
echo '/dev/sdc1 /var/lib/jenkins ext4 defaults 0 0' >>/etc/fstab
echo '/dev/sdd1 /var/lib/docker ext4 defaults 0 0' >>/etc/fstab
mount -a

依據《CentOS7 Jenkins部署 Maven項目構建測試》部署jenkins

# 主機名和hosts解析
HOSTNAME=dockerjenkins
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts

# 部署java環境
cd /usr/local/
tar -xf /tmp/jdk-8u241-linux-x64.tar.gz
echo 'export JAVA_HOME=/usr/local/jdk1.8.0_241'>>/etc/profile
echo 'export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib'>>/etc/profile
echo 'export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH'>>/etc/profile
source /etc/profile
java -version

# 安裝git工具
yum -y install git

# 部署maven環境
cd /usr/local/
unzip /tmp/apache-maven-3.5.2-bin.zip
echo 'export MAVEN_HOME=/usr/local/apache-maven-3.5.2'>>/etc/profile
echo 'export PATH=$PATH:$MAVEN_HOME/bin'>>/etc/profile
source /etc/profile
mvn --version

# 部署安裝jenkins
cd /tmp
wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
yum -y install jenkins

# 配置啓動jenkins
useradd deploy
echo deploy|passwd --stdin deploy
sed -i 's|^\(JENKINS_JAVA_CMD=\).*$|\1"/usr/local/jdk1.8.0_241/bin/java"|g' /etc/sysconfig/jenkins
sed -i 's|^\(JENKINS_PORT=\).*$|\1"18080"|g' /etc/sysconfig/jenkins
sed -i 's|JENKINS_USER="jenkins"|JENKINS_USER="deploy"|g' /etc/sysconfig/jenkins
chown -R deploy: /var/log/jenkins
chown -R deploy: /var/lib/jenkins
chown -R deploy: /var/cache/jenkins
systemctl enable jenkins && systemctl start jenkins
cd /var/lib/jenkins
sed -i 's|https://updates.jenkins.io/update-center.json|https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json|g' hudson.model.UpdateCenter.xml
systemctl restart jenkins

# 瀏覽器登陸 http://192.168.1.248:18080，參見文檔繼續配置，最後添加一個登陸gitlab的jenkins憑據，ID爲：https.gitlab.root.pass

部署nginx作爲構建的項目war包共享服務，修改端口爲8080

cd /tmp
cat >/etc/yum.repos.d/nginx.repo<<EOF
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/\$basearch/
gpgcheck=0
enabled=1
EOF
yum -y install nginx

sed -i 's/80/8080/g' /etc/nginx/conf.d/default.conf
systemctl enable nginx && systemctl start nginx
chown -R deploy: /usr/share/nginx/html/

部署Harbor到OPS主機

• 依據《CentOS7部署安裝Docker和Docker Compose工具簡錄》部署Docker環境和Docker Compose工具

cd /tmp
mkdir -p /etc/docker
cat >/etc/docker/daemon.json<<EOF
{
  "registry-mirrors": ["https://cjw7u3gx.mirror.aliyuncs.com","https://docker.mirrors.ustc.edu.cn","http://hub-mirror.c.163.com"]
}
EOF
yum -y install yum-utils lvm2 device-mapper-persistent-data
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl start docker && systemctl enable docker
curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m) \
  -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

• 依據《CentOS7 Docker Harbor私有倉庫搭建使用簡錄》部署Harbor到該機器上
• 配置URL爲 harbor.vincent.com

# 生成一對ssl證書
yum -y install openssl
mkdir -pv /etc/harbor/ssl && cd /etc/harbor/ssl
openssl genrsa -out "/etc/harbor/ssl/harbor.vincent.com.key" 2048
openssl req -new -key "/etc/harbor/ssl/harbor.vincent.com.key" \
  -out "/etc/harbor/ssl/harbor.vincent.com.csr"
openssl x509 -req -days 365 -in "/etc/harbor/ssl/harbor.vincent.com.csr" \
  -signkey "/etc/harbor/ssl/harbor.vincent.com.key" \
  -out "/etc/harbor/ssl/harbor.vincent.com.crt"
chmod 600 *

# 下載離線安裝包
cd /opt/
wget https://github.com/goharbor/harbor/releases/download/v1.10.1/harbor-offline-installer-v1.10.1.tgz
tar -xf harbor-offline-installer-v1.10.1.tgz
cd harbor
echo "$(hostname -i) harbor.vincent.com">>/etc/hosts
sed -i "s/^hostname:.*$/hostname: harbor.vincent.com/g" harbor.yml
sed -i 's|/your/certificate/path|/etc/harbor/ssl/harbor.vincent.com.crt|g' harbor.yml
sed -i 's|/your/private/key/path|/etc/harbor/ssl/harbor.vincent.com.key|g' harbor.yml
./install.sh

# 網頁登陸 https://192.168.1.248，使用admin/Harbor12345登陸
# 無需創建項目和用戶，只需要對library項目添加tag保留策略和垃圾清理策略即可，建議保留3個tag

• 測試harbor可用性

# 配置本機對harbor的信任
cd /opt/harbor
docker-compose stop
sed -i "s/^.*registry-mirrors.*$/&\n  ,\"insecure-registries\": [\"harbor.vincent.com\"]/g" /etc/docker/daemon.json
sed -i 's|^\[Service\]$|&\nEnvironmentFile=-/etc/docker/daemon.json|g' /lib/systemd/system/docker.service
systemctl stop docker
systemctl daemon-reload
systemctl start docker
docker-compose start

# 下載鏡像，上傳到harbor
docker login harbor.vincent.com
docker pull centos:6
docker tag centos:6 harbor.vincent.com/library/centos:6
docker push harbor.vincent.com/library/centos:6
docker rmi centos:6
docker rmi harbor.vincent.com/library/centos:6

# 安裝jq工具
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum clean all && yum makecache faster
yum -y install jq

# 記錄查詢tag的url
curl -s -k -X GET "https://192.168.1.248/api/repositories/library%2Fcentos/tags" \
-H "accept: application/json" -H "X-Xsrftoken: wQS7eEff2UWUN0jCTKGwFiaPPmwpldVl" \
  | jq '.[]|{name:.name,digest:.digest}'

部署運行業務的5臺容器主機：

• 依據《CentOS7部署安裝Docker和Docker Compose工具簡錄》在5臺服務器上部署Docker環境，並做harbor信任

# 主機名和hosts解析
HOSTNAME=dockerserver1
# HOSTNAME=dockerserver2
# HOSTNAME=dockerserver3
# HOSTNAME=dockerserver4
# HOSTNAME=dockerserver5
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts

# Docker環境部署
cd /tmp
mkdir -p /etc/docker
cat >/etc/docker/daemon.json<<EOF
{
  "registry-mirrors": ["https://cjw7u3gx.mirror.aliyuncs.com","https://docker.mirrors.ustc.edu.cn","http://hub-mirror.c.163.com"]
}
EOF
yum -y install yum-utils lvm2 device-mapper-persistent-data
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl start docker && systemctl enable docker

# Harbor信任
echo '192.168.1.248 harbor.vincent.com'>>/etc/hosts
sed -i "s/^.*registry-mirrors.*$/&\n  ,\"insecure-registries\": [\"harbor.vincent.com\"]/g" /etc/docker/daemon.json
sed -i 's|^\[Service\]$|&\nEnvironmentFile=-/etc/docker/daemon.json|g' /lib/systemd/system/docker.service
systemctl daemon-reload
systemctl restart docker

• 配置OPS主機到各個DockerServer和harbor的ssh免密登陸

cat >>/etc/hosts<<EOF
192.168.1.171 dockerserver1
192.168.1.172 dockerserver2
192.168.1.173 dockerserver3
192.168.1.174 dockerserver4
192.168.1.175 dockerserver5
EOF
yum -y install sshpass
su - deploy
rm -rf ~/.ssh
ssh-keygen -qN '' -f ~/.ssh/id_rsa
sshpass -p 'deploy' ssh-copy-id -o StrictHostKeyChecking=no 127.0.0.1
sshpass -p 'vincent' ssh-copy-id -o StrictHostKeyChecking=no $(hostname -i)
sshpass -p 'vincent' ssh-copy-id -o StrictHostKeyChecking=no $(hostname)
sshpass -p 'vincent' ssh-copy-id -o StrictHostKeyChecking=no harbor.vincent.com
for i in $(cat /etc/hosts|grep dockerserver|awk '{print $2}')
do
    sshpass -p 'vincent' ssh-copy-id -o StrictHostKeyChecking=no root@$i
    ssh root@${i} hostname
done

部署單節點ELK：

• 依據《生產JAVA日誌的ELK歸集方案(一)》部署單節點ELK到 192.168.1.250

# 主機名和hosts解析
HOSTNAME=es1
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts

# 安裝java環境
yum -y install java-11-openjdk

# 安裝elasticsearch和kibana
cd /tmp
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.4.rpm
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.8.4-x86_64.rpm
yum -y localinstall elasticsearch-6.8.4.rpm kibana-6.8.4-x86_64.rpm

# 配置系統服務的ulimit，重啓生效
cat >>/etc/sysctl.conf<<EOF
vm.max_map_count=655360
EOF
sysctl -p
cat >>/etc/systemd/system.conf<<EOF
DefaultLimitNOFILE=100000
DefaultLimitNPROC=65535
DefaultLimitMEMLOCK=infinity
EOF
reboot

# 配置啓動 elasticsearch
cd /etc/elasticsearch
sed -i 's/^path.data/# &/g' elasticsearch.yml
sed -i 's/^path.logs/# &/g' elasticsearch.yml
cat >>elasticsearch.yml<<EOF
cluster.name: vincent-es
node.name: $(hostname)
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
path.data: /elasticsearch/data
path.logs: /elasticsearch/logs
# discovery.zen.ping.unicast.hosts: ["$(hostname)", "XXX", ...]
EOF
mkdir -pv /elasticsearch/{data,logs}
chown -R elasticsearch: /elasticsearch
# 修改jvm參數，設置 Xms=Xmx=物理內存*50%
MEM=$(free -g|grep Mem|awk '{printf "%d\n",$2/2}')
sed -i "s/-Xms1g/-Xms${MEM}g/g" jvm.options
sed -i "s/-Xmx1g/-Xmx${MEM}g/g" jvm.options
# 啓動並測試
systemctl start elasticsearch && systemctl enable elasticsearch
curl http://$(hostname -i):9200

# 配置index清理策略
# index的命名要符合 %{+YYYY.MM.dd} 規則
cat >/root/checkOS/elasticsearchCleanIndex.sh<<EOF
#!/bin/bash
source /etc/profile
DT=\$(date +%Y.%m.%d -d'3 day ago')
for index in \$(curl -s -XGET 'http://127.0.0.1:9200/_cat/indices/?v'|awk '{print \$3}'|grep \${DT})
do
  curl -XDELETE "http://127.0.0.1:9200/\${index}"
done
EOF
chmod +x /root/checkOS/elasticsearchCleanIndex.sh
crontab -l>/tmp/crontab.tmp
echo -e '\n# Elasticsearch Clean Index'>>/tmp/crontab.tmp
echo '0 0 * * * /bin/bash /root/checkOS/elasticsearchCleanIndex.sh'>>/tmp/crontab.tmp
cat /tmp/crontab.tmp |crontab
rm -rf /tmp/crontab.tmp

# 配置啓動kibana
cd /etc/kibana/
cat >>kibana.yml<<EOF
server.host: "0.0.0.0"
server.port: 5601
server.name: "$(hostname)"
elasticsearch.hosts: ["http://$(hostname -i):9200"]
EOF
systemctl start kibana && systemctl enable kibana
systemctl status kibana
netstat -lntup|grep 5601
# 瀏覽器訪問 http://192.168.1.250:5601/

[TOC]