jenkins 持續集成 - harbor企業級倉庫替代docker倉庫

軟 件 下 載 : h t t p s : / / g i t h u b . c o m / g o h a r b o r / h a r b o r / re l e a s e s

爲什麼使用harbor

原文鏈接：https://blog.csdn.net/jessise_zhan/article/details/80130104

一、Harbor的安全機制

harbor 提供了圖形界面，非常易於管理。
企業中的軟件研發團隊往往劃分爲諸多角色，這些角色對於鏡像的使用需求是不一樣的。從安全的角度，也是需要通過某種機制來進行權限控制的。
Harbor爲這種需求提供了用戶和成員兩種管理概念。
在Harbor中，用戶主要分爲兩類。一類爲管理員，另一類爲普通用戶。兩類用戶都可以成爲項目的成員。而管理員可以對用戶進行管理。

• 成員是對應於項目的概念，分爲三類：管理員、開發者、訪客。管理員可以對開發者和訪客作權限的配置和管理。測試和運維人員可以訪客身份讀取項目鏡像，或者公共鏡像庫中的文件。
• 從項目的角度出發，顯然項目管理員擁有最大的項目權限，如果要對用戶進行禁用或限權等，可以通過修改用戶在項目中的成員角色來實現，甚至將用戶移除出這個項目。

二、Harbor的鏡像同步

Harbor提供了更加靈活的方案來處理鏡像的同步，其核心是三個概念：

• 用Harbor自己的API來進行鏡像下載和傳輸，作到與底層存儲環境解耦。
• 利用任務調度和監控機制進行復制任務的管理，保障複製任務的健壯性。在同步過程中，如果源鏡像已刪除，Harbor會自動同步刪除遠端的鏡像。在鏡像同步複製的過程中，Harbor會監控整個複製過程，遇到網絡等錯誤，會自動重試。
• 提供複製策略機制保證項目級的複製需求。在Harbor中，可以在項目中創建複製策略，來實現對鏡像的同步。與Docker Registry的不同之處在於，Harbor的複製是推（PUSH）的策略，由源端發起，而Docker Registry的複製是拉（PULL）的策略，由目標端發起。

三、Harbor與K8s的集成實踐

可以再 k8s 的文章中查看詳情。

創建harbor倉庫

開啓server3虛擬機，安裝docker（和server2的方式一樣），安裝harbor：

[root@server3 ~]# ls
containerd.io-1.2.5-3.1.el7.x86_64.rpm   docker-ce-19.03.8-3.el7.x86_64.rpm      harbor	#這是解壓後的harbor目錄
container-selinux-2.77-1.el7.noarch.rpm  docker-ce-cli-19.03.8-3.el7.x86_64.rpm

[root@server2 yum.repos.d]# scp /etc/sysctl.d/bridge.conf server3:/etc/sysctl.d/

[root@server3 ~]# sysctl --system
[root@server3 harbor]# ls
common.sh  harbor.v1.10.1.tar.gz  harbor.yml  install.sh  LICENSE  prepare
[root@server3 harbor]# vim harbor.yml 		# 更改配置文件
hostname: reg.caoaoyuan.org
harbor_admin_password: caoaoyuan		# 暫時只更改這兩個，先進行測試。

[root@server3 harbor]# ./install.sh 

[Step 0]: checking if docker is installed ...

Note: docker version: 19.03.8

[Step 1]: checking docker-compose is installed ...
✖ Need to install docker-compose(1.18.0+) by yourself first and run this script again.
		# 它需要docker-compose的支持，可以同時操作多個容器，並把多個容器和成一個應用
[root@server3 ~]# mv docker-compose-Linux-x86_64-1.24.1 /usr/local/bin/docker-compose
[root@server3 ~]# chmod +x /usr/local/bin/docker-compose
# 放到環境變量，方便調用。
[root@server3 ~]# docker-compose --help
Define and run multi-container applications with Docker.

Usage:
  docker-compose [-f <arg>...] [options] [COMMAND] [ARGS...]
  docker-compose -h|--help

然後我們進入harbor目錄，再次進行安裝。
[root@server3 harbor]# systemctl start docker
[root@server3 harbor]# ./install.sh 
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-db     ... done
Creating redis         ... done
Creating registry      ... done
Creating harbor-portal ... done
Creating registryctl   ... done
Creating harbor-core   ... done
Creating nginx             ... done			# 這些是他創建管理的容器
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----

[root@server3 harbor]# ls
 docker-compose.yml 		#生成這個文件
 
[root@server3 harbor]# docker-compose ps		# 從上面的文件中讀取內容
      Name                     Command                       State                     Ports          
------------------------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up (health: starting)                            
harbor-db           /docker-entrypoint.sh            Up (health: starting)   5432/tcp                 
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (health: starting)                            
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)            127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (health: starting)   8080/tcp                 
nginx               nginx -g daemon off;             Up (health: starting)   0.0.0.0:80->8080/tcp     
redis               redis-server /etc/redis.conf     Up (health: starting)   6379/tcp                 
registry            /home/harbor/entrypoint.sh       Up (health: starting)   5000/tcp                 
registryctl         /home/harbor/start.sh            Up (health: starting)
[root@server3 harbor]# netstat -tnlp
       # 打開80端口，我們就可以訪問了
tcp6       0      0 :::80                   :::*                    LISTEN      16180/docker-proxy 

現在我們測試上傳鏡像：

[root@server3 harbor]# vim /etc/hosts
172.25.254.1	server1 
172.25.254.2	server2
172.25.254.3	server3 reg.caoaoyuan.org	# 加上解析

#由於我們還沒有認證，只能走80端口，所以寫localhost，不寫reg.caoaoyuan.org
[root@server3 ~]# docker tag game2048:latest localhost/library/game2048:latest
[root@server3 ~]# docker login localhost		# 本地需要先登陸，再上傳
Username: admin
Password: 
[root@server3 ~]# docker push localhost/library/game2048
The push refers to repository [localhost/library/game2048]
88fca8ae768a: Pushed 
6d7504772167: Pushed 
192e9fad2abc: Pushed 
36e9226e74f8: Pushed 
011b303988d2: Pushed 
latest: digest: sha256:8a34fb9cb168c420604b6e5d32ca6d412cb0d533a826b313b190535c03fe9390 size: 1364

上傳上去了。刪掉它，並垃圾清理。

[root@server3 ~]# docker logout localhost
Removing login credentials for localhost
[root@server3 harbor]# docker-compose stop	#同時關閉開啓的容器

現在我們去做加密的方式：

[root@server3 harbor]# vim harbor.yml 
# https related config
https:
  # https port for harbor, default is 443
  port: 443		# 打開443端口
  # The path of cert and key files for nginx
  certificate: /data/certs/caoaoyuan.org.crt			# 證書位置
  private_key: /data/certs/caoaoyuan.org.key

# 創建證書
[root@server3 data]# mkdir certs
[root@server3 data]# openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/caoaoyuan.org.key -x509 -days 365 -out certs/caoaoyuan.org.crt
Country Name (2 letter code) [XX]:cn
State or Province Name (full name) []:shaanxi
Locality Name (eg, city) [Default City]:xi'an
Organization Name (eg, company) [Default Company Ltd]:westos
Organizational Unit Name (eg, section) []:linux
Common Name (eg, your name or your server's hostname) []:reg.caoaoyuan.org
Email Address []:[email protected]

[root@server3 data]# cd certs/
[root@server3 certs]# ls
caoaoyuan.org.crt  caoaoyuan.org.key
[root@server3 certs]# cd /root/harbor/
[root@server3 harbor]# ls
common  common.sh  docker-compose.yml  harbor.v1.10.1.tar.gz  harbor.yml  install.sh  LICENSE  prepare
[root@server3 harbor]# ./prepare 		# 更新文件
[root@server3 harbor]# ./install.sh 		# 重新啓動
[root@server3 harbor]# netstat -tnlp |grep 443
tcp6       0      0 :::443                  :::*                    LISTEN      11580/docker-proxy 

再次訪問：

就會自動跳轉到https上了。
這次我們從server2上上傳一個鏡像：

[root@server2 ~]# docker tag nginx:latest reg.caoaoyuan.org/library/nginx:latest	# 表示上傳到reg.caoaoyuan.org主機上
[root@server2 ~]# vim /etc/hosts

172.25.254.3	server3	reg.caoaoyuan.org		# 做個本地解析
配置證書：
[root@server2 ~]# mkdir -p /etc/docker/certs.d/reg.caoaoyuan.org/

[root@server3 certs]# scp caoaoyuan.org.crt server2:/etc/docker/certs.d/reg.caoaoyuan.org/ca.crt

[root@server2 reg.caoaoyuan.org]# ll
total 4
-rw-r--r-- 1 root root 2114 Jun 16 14:07 ca.crt
[root@server2 reg.caoaoyuan.org]# docker login reg.caoaoyuan.org		# 認證
Username: admin
Password: 

Login Succeeded
[root@server2 reg.caoaoyuan.org]# docker push reg.caoaoyuan.org/library/nginx

就傳過來了。
那拉取是什麼樣的，拉取是不需要認證的。

[root@server2 reg.caoaoyuan.org]# docker logout reg.caoaoyuan.org
Removing login credentials for reg.caoaoyuan.org
[root@server2 reg.caoaoyuan.org]# docker pull reg.caoaoyuan.org/library/nginx
Using default tag: latest
latest: Pulling from library/nginx
Digest: sha256:0efad4d09a419dc6d574c3c3baacb804a530acd61d5eba72cb1f14e1f5ac0c8f
Status: Image is up to date for reg.caoaoyuan.org/library/nginx:latest
reg.caoaoyuan.org/library/nginx:latest		

再harbor上我們可以看到：

匿名用戶進行了拉取。

我們在網上拉取的時候，官方鏡像往往不用加上前面的域名，直接拉取nginx就行，而我們再則需要加上reg.caoaoyuan.org/library/nginx 才能拉取，這是什麼原理:

[root@server2 docker]# vim daemon.json
[root@server2 docker]# cat daemon.json 
{
	"registry-mirrors": ["https://reg.caoaoyuan.org"]		# 裏面寫上自己的倉庫
}
[root@server2 docker]# systemctl restart docker  		# 需要重啓
[root@server2 docker]# docker rmi nginx
[root@server2 docker]# docker rmi reg.caoaoyuan.org/library/nginx:latest 	# 刪除所有的nginx鏡像
[root@server2 docker]# docker images | grep nginx
[root@server2 docker]# docker pull nginx	# 不加reg.caoaoyuan.org 拉取

直接從我們的私有倉庫拉取了。
在試一個game2048鏡像;

[root@server3 harbor]# cd /etc/docker/
[root@server3 docker]# mkdir certs.d/reg.caoaoyuan.org -p
[root@server3 docker]# cd certs.d/reg.caoaoyuan.org/
[root@server3 reg.caoaoyuan.org]# cp /data/certs/caoaoyuan.org.crt ca.crt
[root@server3 reg.caoaoyuan.org]# ls
ca.crt
[root@server3 ~]# docker login reg.caoaoyuan.org
Username: admin
Password: 
Login Succeeded
[root@server3 ~]# docker push  reg.caoaoyuan.org/library/game2048
The push refers to repository [reg.caoaoyuan.org/library/game2048]
88fca8ae768a: Pushed 
6d7504772167: Pushed 
192e9fad2abc: Pushed 
36e9226e74f8: Pushed 
011b303988d2: Pushed 
latest: digest: sha256:8a34fb9cb168c420604b6e5d32ca6d412cb0d533a826b313b190535c03fe9390 size: 1364
[root@server3 ~]#

現在我們倉庫裏有game2048.

[root@server2 docker]# docker pull game2048

看到了吧。就直接從我們指定的倉庫下載了，如果我們倉庫沒有的話，纔會去從網上拉取。

私有harbor倉庫

新建一個項目，不勾公開。

新建一個用戶，以開發人員的身份加入westos項目中。

[root@server2 docker]# docker login reg.caoaoyuan.org
Username: cay
Password: 

Login Succeeded
[root@server2 docker]# docker push reg.caoaoyuan.org/westos/game2048

開發人員可以上傳下載，我們登陸進harbor中。

可以看出沒有任何的配置權限。不能進行系統管理。

[root@server3 harbor]# docker pull reg.caoaoyuan.org/westos/game2048
Using default tag: latest
Error response from daemon: pull access denied for reg.caoaoyuan.org/westos/game2048, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
# 私有倉庫不登陸是無法下載的。登陸後纔可以下載

[root@server3 harbor]# docker login reg.caoaoyuan.org
Username: cay
Password: 

Login Succeeded
[root@server3 harbor]# docker pull reg.caoaoyuan.org/westos/game2048
Using default tag: latest
latest: Pulling from westos/game2048
Digest: sha256:8a34fb9cb168c420604b6e5d32ca6d412cb0d533a826b313b190535c03fe9390
Status: Downloaded newer image for reg.caoaoyuan.org/westos/game2048:latest
reg.caoaoyuan.org/westos/game2048:latest

在這裏勾上公開，就可以不登陸下載了。我們先勾上。

漏洞掃描，內容信任。

我們還可以通過漏洞掃描，鏡像信任等方式提升倉庫的安全：

[root@server3 harbor]# docker-compose stop

[root@server3 harbor]# ./install.sh --help
Note: Please set hostname and other necessary attributes in harbor.yml first. DO NOT use localhost or 127.0.0.1 for hostname, because Harbor needs to be accessed by external clients.
Please set --with-notary 	# 內容信任
Please set --with-clair if needs enable Clair in Harbor		# 漏洞掃描
Please set --with-chartmuseum 	# 支持chart，在k8s會用到
[root@server3 harbor]# ./install.sh --with-notary --with-clair --with-chartmuseum

出現了helm charts。後面會用到。

在鏡像這裏出現了掃描和簽名的選項，我們手動掃描一下。



也可以在項目配置這裏勾上自動掃描。

[root@server3 ~]# docker login reg.caoaoyuan.org
Username: admin
Password: 

Login Succeeded
[root@server3 ~]# docker tag busybox:latest reg.caoaoyuan.org/library/busybox
[root@server3 ~]# docker push reg.caoaoyuan.org/library/busybox
The push refers to repository [reg.caoaoyuan.org/library/busybox]
8a788232037e: Pushed 
latest: digest: sha256:915f390a8912e16d4beb8689720a17348f3f6d1a7b659697df850ab625ea29d5 size: 527

內容信任：

打開library的內容信任，

[root@server3 ~]# docker pull reg.caoaoyuan.org/library/game2048
Using default tag: latest
Error response from daemon: unknown: The image is not signed in Notary.	# 拉取失敗，禁止沒有簽名的鏡像部署到生產環境中。

那我們怎麼去部署哪：

• 部署根證書:

[root@server3 ~]# cd .docker

[root@server3 .docker]# mkdir tls/reg.caoaoyuan.org:4443 -p
cd [root@server3 .docker]# cd tls/reg.caoaoyuan.org\:4443/
[root@server3 reg.caoaoyuan.org:4443]# cp /etc/docker/certs.d/reg.caoaoyuan.org/ca.crt  .
[root@server3 reg.caoaoyuan.org:4443]# ls
ca.crt
# /etc/docker/cert/下的證書是給docker使用的，這裏是給操作系統使用的

• 啓 用 d o c ke r 內 容 信 任 :

[root@server3 reg.caoaoyuan.org:4443]# export DOCKER_CONTENT_TRUST=1
[root@server3 reg.caoaoyuan.org:4443]# export DOCKER_CONTENT_TRUST_SERVER=https://reg.caoaoyuan.org:4443
[root@server3 reg.caoaoyuan.org:4443]# docker port nginx
8443/tcp -> 0.0.0.0:443
4443/tcp -> 0.0.0.0:4443		# 4443端口就是做內容信任的
8080/tcp -> 0.0.0.0:80

[root@server3 ~]# docker tag ikubernetes/myapp:v1 reg.caoaoyuan.org/westos/myapp:v1
[root@server3 ~]# docker push reg.caoaoyuan.org/westos/myapp:v1
The push refers to repository [reg.caoaoyuan.org/westos/myapp]
a0d2c4392b06: Pushed 
05a9e65e2d53: Pushed 
68695a6cfd7d: Pushed 
c1dc81a64903: Pushed 
8460a579ab63: Pushed 
d39d92664027: Pushed 
v1: digest: sha256:9eeca44ba2d410e54fccc54cbe9c021802aa8b9836a0bcf3d3229354e4c8870e size: 1569
Signing and pushing trust metadata
You are about to create a new root signing key passphrase. This passphrase
will be used to protect the most sensitive key in your signing system. Please
choose a long, complex passphrase and be careful to keep the password and the
key file itself secure and backed up. It is highly recommended that you use a
password manager to generate the passphrase and keep it safe. There will be no
way to recover this key. You can find the key in your config directory.
Enter passphrase for new root key with ID 659fb4e: 
Repeat passphrase for new root key with ID 659fb4e: 
Enter passphrase for new repository key with ID f7f16c1: Cc990718
Repeat passphrase for new repository key with ID f7f16c1: 
Finished initializing "reg.caoaoyuan.org/westos/myapp"
Successfully signed reg.caoaoyuan.org/westos/myapp:v1

這時我們再上傳的時候它要求我們創建一個新的根簽名密鑰密碼，我們保持一致就行。

已簽名。

[root@server3 ~]# docker tag ikubernetes/myapp:v2 reg.caoaoyuan.org/westos/myapp:v2
[root@server3 ~]# docker push reg.caoaoyuan.org/westos/myapp:v2
The push refers to repository [reg.caoaoyuan.org/westos/myapp]

v2: digest: sha256:5f4afc8302ade316fc47c99ee1d41f8ba94dbe7e3e7747dd87215a15429b9102 size: 1362
Signing and pushing trust metadata
Enter passphrase for repository key with ID f7f16c1:  #我們只是換了個標籤，這裏只要倉庫的密碼
Successfully signed reg.caoaoyuan.org/westos/myapp:v2

我們去遠程主機上進行拉取：

[root@server2 docker]# docker pull reg.caoaoyuan.org/westos/myapp:v1
v1: Pulling from westos/myapp
Digest: sha256:9eeca44ba2d410e54fccc54cbe9c021802aa8b9836a0bcf3d3229354e4c8870e
Status: Downloaded newer image for reg.caoaoyuan.org/westos/myapp:v1
reg.caoaoyuan.org/westos/myapp:v1
[root@server2 docker]# docker pull reg.caoaoyuan.org/westos/game2048
Using default tag: latest
Error response from daemon: unknown: The image is not signed in Notary.
	# 簽名的myapp可以拉取，沒有簽名的game2048不可以拉取部署。

[root@server3 ~]# docker tag busybox:latest reg.caoaoyuan.org/westos/busybox
# 我們在這裏不打標籤（：v1），但是簽名是和標籤綁定的
docker[root@server3 ~]# docker push reg.caoaoyuan.org/westos/busybox
The push refers to repository [reg.caoaoyuan.org/westos/busybox]
8a788232037e: Mounted from library/busybox 
latest: digest: sha256:915f390a8912e16d4beb8689720a17348f3f6d1a7b659697df850ab625ea29d5 size: 527
No tag specified, skipping trust metadata push

No tag specified, skipping trust metadata push
46/5000

# 這時告訴我們未指定標籤，跳過信任推送

就沒有簽名了。
此時我們想刪除鏡像是不允許的，因爲帶有簽名，我們可以進行回收:

[root@server3 ~]# docker trust inspect reg.caoaoyuan.org/westos/myapp:v1
......				# 這裏就是他們的信任證書
                "Name": "Repository",
                "Keys": [
                    {
                        "ID": "f7f16c151fa84357260bc4af7e46b2ca77d471b96b720e70026c28725ba7fcb6"

[root@server3 ~]# docker trust revoke reg.caoaoyuan.org/westos/myapp:v1		# 回收
Enter passphrase for repository key with ID f7f16c1: 
Successfully deleted signature for reg.caoaoyuan.org/westos/myapp:v1
[root@server3 ~]# docker trust revoke reg.caoaoyuan.org/westos/myapp:v2
Enter passphrase for repository key with ID f7f16c1: 
Successfully deleted signature for reg.caoaoyuan.org/westos/myapp:v2

他們的簽名就沒有了，可以刪除了

連項目都可以刪除了。我們去掉內容信任，開始接下來的實驗。更改觸發jenkins方式的實驗。