生产项目容器化改造(一)环境部署

主机列表:

192.168.1.248 CentOS7 DockerJenkins 4U8G 3*50G Disk
192.168.1.249 CentOS7 DockerGitlab  4U8G
192.168.1.250 CentOS7 DockerELK     4U8G
192.168.1.171 CentOS7 DockerServer1 8U16G
192.168.1.172 CentOS7 DockerServer2 8U16G
192.168.1.173 CentOS7 DockerServer3 8U16G
192.168.1.174 CentOS7 DockerServer4 8U16G
192.168.1.175 CentOS7 DockerServer5 8U16G

部署GitLab：

• 依据《CentOS7 gitlab安装搭建简单维护》和《CentOS7 gitlab支持https的改造》部署一台Gitlab
• 相应的URL为：https://gitlab.vincent.com，登陆 192.168.1.249 进行部署，全部步骤如下：

# 主机名和hosts解析
HOSTNAME=dockergitlab
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME gitlab.vincent.com">>/etc/hosts

# 创建密钥证书目录
mkdir -p /etc/gitlab/ssl && cd /etc/gitlab/ssl
openssl genrsa -out "/etc/gitlab/ssl/gitlab.vincent.com.key" 2048
openssl req -new \
  -key "/etc/gitlab/ssl/gitlab.vincent.com.key" \
  -out "/etc/gitlab/ssl/gitlab.vincent.com.csr"
openssl x509 -req -days 365 \
  -in "/etc/gitlab/ssl/gitlab.vincent.com.csr" \
  -signkey "/etc/gitlab/ssl/gitlab.vincent.com.key" \
  -out "/etc/gitlab/ssl/gitlab.vincent.com.crt"
openssl dhparam -out /etc/gitlab/ssl/dhparams.pem 2048
chmod 600 *

# 下载并安装gitlab
cd /tmp
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-12.10.6-ce.0.el7.x86_64.rpm
yum -y localinstall gitlab-ce-12.10.6-ce.0.el7.x86_64.rpm
gitlab-ctl reconfigure
gitlab-ctl status|column -t
systemctl enable gitlab-runsvdir.service

# 配置gitlab
sed -i "s|^external_url.*$|# &\n\
external_url 'https://gitlab.vincent.com'|g" /etc/gitlab/gitlab.rb
sed -i "s|^# nginx\['enable'\] = true$|\
nginx['redirect_http_to_https'] = true\n\
nginx['ssl_certificate'] = \"/etc/gitlab/ssl/gitlab.vincent.com.crt\"\n\
nginx['ssl_certificate_key'] = \"/etc/gitlab/ssl/gitlab.vincent.com.key\"\
\n&|g" /etc/gitlab/gitlab.rb
sed -i "s|^# nginx\['ssl_dhparam'\] = nil|\
# nginx\['ssl_dhparam'\] = /etc/gitlab/ssl/dhparams.pem|g" /etc/gitlab/gitlab.rb
cat >>/etc/gitlab/gitlab.rb<<EOF
# mail alert setup
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'smtp.126.com'
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_user_name'] = '[email protected]'
gitlab_rails['smtp_password'] = 'xxxx'
gitlab_rails['smtp_authentication'] = 'login'
gitlab_rails['smtp_enable_starttls_auto']= true
gitlab_rails['gitlab_email_from']= '[email protected]'
gitlab_rails['gitlab_email_reply_to']= '[email protected]'
EOF
gitlab-ctl reconfigure
gitlab-ctl restart

# 客户机添加hosts解析，使用浏览器访问：https://gitlab.vincent.com，配置root密码
# 导入各个maven项目

部署OPS主机：

提前将数据盘挂载到各个目录

• 依据《CentOS7实验机模板搭建部署》部署实验机：192.168.1.248，分配3块数据盘并挂载到相应目录

echo -e 'n\np\n\n\n\nw\n'|fdisk /dev/sdb
echo -e 'n\np\n\n\n\nw\n'|fdisk /dev/sdc
echo -e 'n\np\n\n\n\nw\n'|fdisk /dev/sdd
mkfs.ext4 /dev/sdb1
mkfs.ext4 /dev/sdc1
mkfs.ext4 /dev/sdd1
mkdir -pv /usr/share/nginx/html /var/lib/jenkins /var/lib/docker
echo '/dev/sdb1 /usr/share/nginx/html ext4 defaults 0 0' >>/etc/fstab
echo '/dev/sdc1 /var/lib/jenkins ext4 defaults 0 0' >>/etc/fstab
echo '/dev/sdd1 /var/lib/docker ext4 defaults 0 0' >>/etc/fstab
mount -a

依据《CentOS7 Jenkins部署 Maven项目构建测试》部署jenkins

# 主机名和hosts解析
HOSTNAME=dockerjenkins
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts

# 部署java环境
cd /usr/local/
tar -xf /tmp/jdk-8u241-linux-x64.tar.gz
echo 'export JAVA_HOME=/usr/local/jdk1.8.0_241'>>/etc/profile
echo 'export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib'>>/etc/profile
echo 'export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH'>>/etc/profile
source /etc/profile
java -version

# 安装git工具
yum -y install git

# 部署maven环境
cd /usr/local/
unzip /tmp/apache-maven-3.5.2-bin.zip
echo 'export MAVEN_HOME=/usr/local/apache-maven-3.5.2'>>/etc/profile
echo 'export PATH=$PATH:$MAVEN_HOME/bin'>>/etc/profile
source /etc/profile
mvn --version

# 部署安装jenkins
cd /tmp
wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
yum -y install jenkins

# 配置启动jenkins
useradd deploy
echo deploy|passwd --stdin deploy
sed -i 's|^\(JENKINS_JAVA_CMD=\).*$|\1"/usr/local/jdk1.8.0_241/bin/java"|g' /etc/sysconfig/jenkins
sed -i 's|^\(JENKINS_PORT=\).*$|\1"18080"|g' /etc/sysconfig/jenkins
sed -i 's|JENKINS_USER="jenkins"|JENKINS_USER="deploy"|g' /etc/sysconfig/jenkins
chown -R deploy: /var/log/jenkins
chown -R deploy: /var/lib/jenkins
chown -R deploy: /var/cache/jenkins
systemctl enable jenkins && systemctl start jenkins
cd /var/lib/jenkins
sed -i 's|https://updates.jenkins.io/update-center.json|https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json|g' hudson.model.UpdateCenter.xml
systemctl restart jenkins

# 浏览器登陆 http://192.168.1.248:18080，参见文档继续配置，最后添加一个登陆gitlab的jenkins凭据，ID为：https.gitlab.root.pass

部署nginx作为构建的项目war包共享服务，修改端口为8080

cd /tmp
cat >/etc/yum.repos.d/nginx.repo<<EOF
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/\$basearch/
gpgcheck=0
enabled=1
EOF
yum -y install nginx

sed -i 's/80/8080/g' /etc/nginx/conf.d/default.conf
systemctl enable nginx && systemctl start nginx
chown -R deploy: /usr/share/nginx/html/

部署Harbor到OPS主机

• 依据《CentOS7部署安装Docker和Docker Compose工具简录》部署Docker环境和Docker Compose工具

cd /tmp
mkdir -p /etc/docker
cat >/etc/docker/daemon.json<<EOF
{
  "registry-mirrors": ["https://cjw7u3gx.mirror.aliyuncs.com","https://docker.mirrors.ustc.edu.cn","http://hub-mirror.c.163.com"]
}
EOF
yum -y install yum-utils lvm2 device-mapper-persistent-data
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl start docker && systemctl enable docker
curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m) \
  -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

• 依据《CentOS7 Docker Harbor私有仓库搭建使用简录》部署Harbor到该机器上
• 配置URL为 harbor.vincent.com

# 生成一对ssl证书
yum -y install openssl
mkdir -pv /etc/harbor/ssl && cd /etc/harbor/ssl
openssl genrsa -out "/etc/harbor/ssl/harbor.vincent.com.key" 2048
openssl req -new -key "/etc/harbor/ssl/harbor.vincent.com.key" \
  -out "/etc/harbor/ssl/harbor.vincent.com.csr"
openssl x509 -req -days 365 -in "/etc/harbor/ssl/harbor.vincent.com.csr" \
  -signkey "/etc/harbor/ssl/harbor.vincent.com.key" \
  -out "/etc/harbor/ssl/harbor.vincent.com.crt"
chmod 600 *

# 下载离线安装包
cd /opt/
wget https://github.com/goharbor/harbor/releases/download/v1.10.1/harbor-offline-installer-v1.10.1.tgz
tar -xf harbor-offline-installer-v1.10.1.tgz
cd harbor
echo "$(hostname -i) harbor.vincent.com">>/etc/hosts
sed -i "s/^hostname:.*$/hostname: harbor.vincent.com/g" harbor.yml
sed -i 's|/your/certificate/path|/etc/harbor/ssl/harbor.vincent.com.crt|g' harbor.yml
sed -i 's|/your/private/key/path|/etc/harbor/ssl/harbor.vincent.com.key|g' harbor.yml
./install.sh

# 网页登陆 https://192.168.1.248，使用admin/Harbor12345登陆
# 无需创建项目和用户，只需要对library项目添加tag保留策略和垃圾清理策略即可，建议保留3个tag

• 测试harbor可用性

# 配置本机对harbor的信任
cd /opt/harbor
docker-compose stop
sed -i "s/^.*registry-mirrors.*$/&\n  ,\"insecure-registries\": [\"harbor.vincent.com\"]/g" /etc/docker/daemon.json
sed -i 's|^\[Service\]$|&\nEnvironmentFile=-/etc/docker/daemon.json|g' /lib/systemd/system/docker.service
systemctl stop docker
systemctl daemon-reload
systemctl start docker
docker-compose start

# 下载镜像，上传到harbor
docker login harbor.vincent.com
docker pull centos:6
docker tag centos:6 harbor.vincent.com/library/centos:6
docker push harbor.vincent.com/library/centos:6
docker rmi centos:6
docker rmi harbor.vincent.com/library/centos:6

# 安装jq工具
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum clean all && yum makecache faster
yum -y install jq

# 记录查询tag的url
curl -s -k -X GET "https://192.168.1.248/api/repositories/library%2Fcentos/tags" \
-H "accept: application/json" -H "X-Xsrftoken: wQS7eEff2UWUN0jCTKGwFiaPPmwpldVl" \
  | jq '.[]|{name:.name,digest:.digest}'

部署运行业务的5台容器主机：

• 依据《CentOS7部署安装Docker和Docker Compose工具简录》在5台服务器上部署Docker环境，并做harbor信任

# 主机名和hosts解析
HOSTNAME=dockerserver1
# HOSTNAME=dockerserver2
# HOSTNAME=dockerserver3
# HOSTNAME=dockerserver4
# HOSTNAME=dockerserver5
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts

# Docker环境部署
cd /tmp
mkdir -p /etc/docker
cat >/etc/docker/daemon.json<<EOF
{
  "registry-mirrors": ["https://cjw7u3gx.mirror.aliyuncs.com","https://docker.mirrors.ustc.edu.cn","http://hub-mirror.c.163.com"]
}
EOF
yum -y install yum-utils lvm2 device-mapper-persistent-data
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl start docker && systemctl enable docker

# Harbor信任
echo '192.168.1.248 harbor.vincent.com'>>/etc/hosts
sed -i "s/^.*registry-mirrors.*$/&\n  ,\"insecure-registries\": [\"harbor.vincent.com\"]/g" /etc/docker/daemon.json
sed -i 's|^\[Service\]$|&\nEnvironmentFile=-/etc/docker/daemon.json|g' /lib/systemd/system/docker.service
systemctl daemon-reload
systemctl restart docker

• 配置OPS主机到各个DockerServer和harbor的ssh免密登陆

cat >>/etc/hosts<<EOF
192.168.1.171 dockerserver1
192.168.1.172 dockerserver2
192.168.1.173 dockerserver3
192.168.1.174 dockerserver4
192.168.1.175 dockerserver5
EOF
yum -y install sshpass
su - deploy
rm -rf ~/.ssh
ssh-keygen -qN '' -f ~/.ssh/id_rsa
sshpass -p 'deploy' ssh-copy-id -o StrictHostKeyChecking=no 127.0.0.1
sshpass -p 'vincent' ssh-copy-id -o StrictHostKeyChecking=no $(hostname -i)
sshpass -p 'vincent' ssh-copy-id -o StrictHostKeyChecking=no $(hostname)
sshpass -p 'vincent' ssh-copy-id -o StrictHostKeyChecking=no harbor.vincent.com
for i in $(cat /etc/hosts|grep dockerserver|awk '{print $2}')
do
    sshpass -p 'vincent' ssh-copy-id -o StrictHostKeyChecking=no root@$i
    ssh root@${i} hostname
done

部署单节点ELK：

• 依据《生产JAVA日志的ELK归集方案(一)》部署单节点ELK到 192.168.1.250

# 主机名和hosts解析
HOSTNAME=es1
hostnamectl set-hostname "$HOSTNAME"
echo "$HOSTNAME">/etc/hostname
echo "$(grep -E '127|::1' /etc/hosts)">/etc/hosts
echo "$(ip a|grep "inet "|grep -v 127|awk -F'[ /]' '{print $6}') $HOSTNAME">>/etc/hosts

# 安装java环境
yum -y install java-11-openjdk

# 安装elasticsearch和kibana
cd /tmp
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.4.rpm
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.8.4-x86_64.rpm
yum -y localinstall elasticsearch-6.8.4.rpm kibana-6.8.4-x86_64.rpm

# 配置系统服务的ulimit，重启生效
cat >>/etc/sysctl.conf<<EOF
vm.max_map_count=655360
EOF
sysctl -p
cat >>/etc/systemd/system.conf<<EOF
DefaultLimitNOFILE=100000
DefaultLimitNPROC=65535
DefaultLimitMEMLOCK=infinity
EOF
reboot

# 配置启动 elasticsearch
cd /etc/elasticsearch
sed -i 's/^path.data/# &/g' elasticsearch.yml
sed -i 's/^path.logs/# &/g' elasticsearch.yml
cat >>elasticsearch.yml<<EOF
cluster.name: vincent-es
node.name: $(hostname)
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
path.data: /elasticsearch/data
path.logs: /elasticsearch/logs
# discovery.zen.ping.unicast.hosts: ["$(hostname)", "XXX", ...]
EOF
mkdir -pv /elasticsearch/{data,logs}
chown -R elasticsearch: /elasticsearch
# 修改jvm参数，设置 Xms=Xmx=物理内存*50%
MEM=$(free -g|grep Mem|awk '{printf "%d\n",$2/2}')
sed -i "s/-Xms1g/-Xms${MEM}g/g" jvm.options
sed -i "s/-Xmx1g/-Xmx${MEM}g/g" jvm.options
# 启动并测试
systemctl start elasticsearch && systemctl enable elasticsearch
curl http://$(hostname -i):9200

# 配置index清理策略
# index的命名要符合 %{+YYYY.MM.dd} 规则
cat >/root/checkOS/elasticsearchCleanIndex.sh<<EOF
#!/bin/bash
source /etc/profile
DT=\$(date +%Y.%m.%d -d'3 day ago')
for index in \$(curl -s -XGET 'http://127.0.0.1:9200/_cat/indices/?v'|awk '{print \$3}'|grep \${DT})
do
  curl -XDELETE "http://127.0.0.1:9200/\${index}"
done
EOF
chmod +x /root/checkOS/elasticsearchCleanIndex.sh
crontab -l>/tmp/crontab.tmp
echo -e '\n# Elasticsearch Clean Index'>>/tmp/crontab.tmp
echo '0 0 * * * /bin/bash /root/checkOS/elasticsearchCleanIndex.sh'>>/tmp/crontab.tmp
cat /tmp/crontab.tmp |crontab
rm -rf /tmp/crontab.tmp

# 配置启动kibana
cd /etc/kibana/
cat >>kibana.yml<<EOF
server.host: "0.0.0.0"
server.port: 5601
server.name: "$(hostname)"
elasticsearch.hosts: ["http://$(hostname -i):9200"]
EOF
systemctl start kibana && systemctl enable kibana
systemctl status kibana
netstat -lntup|grep 5601
# 浏览器访问 http://192.168.1.250:5601/

[TOC]