實現方式
複製同步
多harbor實例共享後端存儲
主從複製同步
harbor官方默認提供主從複製的方案來解決鏡像同步問題，通過複製的方式，我們可以實時將測試環境harbor倉庫的鏡像同步到生產環境harbor，類似於如下流程：
在實際生產運維的中，往往需要把鏡像發佈到幾十或上百臺集羣節點上。這時，單個Registry已經無法滿足大量節點的下載需求，因此要配置多個Registry實例做負載均衡。手工維護多個Registry實例上的鏡像，將是十分繁瑣的事情。Harbor可以支持一主多從的鏡像發佈模式，可以解決大規模鏡像發佈的難題：
只要往一臺Registry上發佈，鏡像就像“仙女散花”般地同步到多個Registry中，高效可靠。
如果是地域分佈較廣的集羣，還可以採用層次型發佈方式，如從集團總部同步到省公司，從省公司再同步到市公司：
然而單靠主從同步，仍然解決不了harbor主節點的單點問題。
雙主複製同步
所謂的雙主複製其實就是複用主從同步實現兩個harbor節點之間的雙向同步，來保證數據的一致性，然後在兩臺harbor前端頂一個負載均衡器將進來的請求分流到不同的實例中去，只要有一個實例中有了新的鏡像，就是自動的同步複製到另外的的實例中去，這樣實現了負載均衡，也避免了單點故障，在一定程度上實現了Harbor的高可用性：
這個方案有一個問題就是有可能兩個Harbor實例中的數據不一致。假設如果一個實例A掛掉了，這個時候有新的鏡像進來，那麼新的鏡像就會在另外一個實例B中，後面即使恢復了掛掉的A實例，Harbor實例B也不會自動去同步鏡像，這樣只能手動的先關掉Harbor實例B的複製策略，然後再開啓複製策略，才能讓實例B數據同步，讓兩個實例的數據一致。
在實際生產使用中，主從複製十分的不靠譜。
所以這裏推薦使用下面要說的這種方案。
多harbor實例共享後端存儲
共享後端存儲算是一種比較標準的方案，就是多個Harbor實例共享同一個後端存儲，任何一個實例持久化到存儲的鏡像，都可被其他實例中讀取。通過前置LB進來的請求，可以分流到不同的實例中去處理，這樣就實現了負載均衡，也避免了單點故障：
這個方案在實際生產環境中部署需要考慮三個問題：
共享存儲的選取，Harbor的後端存儲目前支持AWS S3、Openstack Swift, Ceph等，在我們的實驗環境裏，就直接使用nfs
Session在不同的實例上共享，這個現在其實已經不是問題了，在最新的harbor中，默認session會存放在redis中，我們只需要將redis獨立出來即可。可以通過redis sentinel或者redis cluster等方式來保證redis的可用性。在我們的實驗環境裏，仍然使用單臺redis
Harbor多實例數據庫問題，這個也只需要將harbor中的數據庫拆出來獨立部署即可。讓多實例共用一個外部數據庫，數據庫的高可用也可以通過數據庫的高可用方案保證。
環境說明
實驗環境
Ip | Role |
172.16.251.131 | harbor01.dc.com |
172.16.251.132 | harbor02.dc.com |
172.16.251.133 | Storage.dc.com |
172.16.251.134 | Lvs+keepalived |
172.16.251.135 | Lvs+keepalived |
配置說明
安裝配置NFS
1、安裝NFS
root@storage:~# apt install nfs-kernel-server nfs-common
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 python python-minimal python2.7 python2.7-minimal rpcbind
Suggested packages:
watchdog python-doc python-tk python2.7-doc binutils binfmt-support
The following NEW packages will be installed:
keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 nfs-common nfs-kernel-server python python-minimal python2.7
python2.7-minimal rpcbind
0 upgraded, 13 newly installed, 0 to remove and 4 not upgraded.
Need to get 4,344 kB of archives.
After this operation, 18.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.aliyun.com/ubuntu xenial/main amd64 libnfsidmap2 amd64 0.25-5 [32.2 kB]
Get:2 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libpython2.7-minimal amd64 2.7.12-1ubuntu0~16.04.4 [339 kB]
Get:3 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python2.7-minimal amd64 2.7.12-1ubuntu0~16.04.4 [1,261 kB]
Get:4 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python-minimal amd64 2.7.12-1~16.04 [28.1 kB]
Get:5 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libpython2.7-stdlib amd64 2.7.12-1ubuntu0~16.04.4 [1,880 kB]
Get:6 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python2.7 amd64 2.7.12-1ubuntu0~16.04.4 [224 kB]
Get:7 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libpython-stdlib amd64 2.7.12-1~16.04 [7,768 B]
Get:8 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 python amd64 2.7.12-1~16.04 [137 kB]
Get:9 http://mirrors.aliyun.com/ubuntu xenial/main amd64 keyutils amd64 1.5.9-8ubuntu1 [47.1 kB]
Get:10 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libtirpc1 amd64 0.2.5-1ubuntu0.1 [75.4 kB]
Get:11 http://mirrors.aliyun.com/ubuntu xenial/main amd64 rpcbind amd64 0.2.3-0.2 [40.3 kB]
Get:12 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 nfs-common amd64 1:1.2.8-9ubuntu12.1 [184 kB]
Get:13 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 nfs-kernel-server amd64 1:1.2.8-9ubuntu12.1 [88.0 kB]
Fetched 4,344 kB in 0s (5,046 kB/s)
Selecting previously unselected package libnfsidmap2:amd64.
(Reading database ... 60656 files and directories currently installed.)
Preparing to unpack .../libnfsidmap2_0.25-5_amd64.deb ...
Unpacking libnfsidmap2:amd64 (0.25-5) ...
Selecting previously unselected package libpython2.7-minimal:amd64.
Preparing to unpack .../libpython2.7-minimal_2.7.12-1ubuntu0~16.04.4_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.4) ...
Selecting previously unselected package python2.7-minimal.
Preparing to unpack .../python2.7-minimal_2.7.12-1ubuntu0~16.04.4_amd64.deb ...
Unpacking python2.7-minimal (2.7.12-1ubuntu0~16.04.4) ...
Selecting previously unselected package python-minimal.
Preparing to unpack .../python-minimal_2.7.12-1~16.04_amd64.deb ...
Unpacking python-minimal (2.7.12-1~16.04) ...
Selecting previously unselected package libpython2.7-stdlib:amd64.
Preparing to unpack .../libpython2.7-stdlib_2.7.12-1ubuntu0~16.04.4_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.4) ...
Selecting previously unselected package python2.7.
Preparing to unpack .../python2.7_2.7.12-1ubuntu0~16.04.4_amd64.deb ...
Unpacking python2.7 (2.7.12-1ubuntu0~16.04.4) ...
Selecting previously unselected package libpython-stdlib:amd64.
Preparing to unpack .../libpython-stdlib_2.7.12-1~16.04_amd64.deb ...
Unpacking libpython-stdlib:amd64 (2.7.12-1~16.04) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Setting up libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.4) ...
Setting up python2.7-minimal (2.7.12-1ubuntu0~16.04.4) ...
Linking and byte-compiling packages for runtime python2.7...
Setting up python-minimal (2.7.12-1~16.04) ...
Selecting previously unselected package python.
(Reading database ... 61412 files and directories currently installed.)
Preparing to unpack .../python_2.7.12-1~16.04_amd64.deb ...
Unpacking python (2.7.12-1~16.04) ...
Selecting previously unselected package keyutils.
Preparing to unpack .../keyutils_1.5.9-8ubuntu1_amd64.deb ...
Unpacking keyutils (1.5.9-8ubuntu1) ...
Selecting previously unselected package libtirpc1:amd64.
Preparing to unpack .../libtirpc1_0.2.5-1ubuntu0.1_amd64.deb ...
Unpacking libtirpc1:amd64 (0.2.5-1ubuntu0.1) ...
Selecting previously unselected package rpcbind.
Preparing to unpack .../rpcbind_0.2.3-0.2_amd64.deb ...
Unpacking rpcbind (0.2.3-0.2) ...
Selecting previously unselected package nfs-common.
Preparing to unpack .../nfs-common_1%3a1.2.8-9ubuntu12.1_amd64.deb ...
Unpacking nfs-common (1:1.2.8-9ubuntu12.1) ...
Selecting previously unselected package nfs-kernel-server.
Preparing to unpack .../nfs-kernel-server_1%3a1.2.8-9ubuntu12.1_amd64.deb ...
Unpacking nfs-kernel-server (1:1.2.8-9ubuntu12.1) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.15) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up libnfsidmap2:amd64 (0.25-5) ...
Setting up libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.4) ...
Setting up python2.7 (2.7.12-1ubuntu0~16.04.4) ...
Setting up libpython-stdlib:amd64 (2.7.12-1~16.04) ...
Setting up python (2.7.12-1~16.04) ...
Setting up keyutils (1.5.9-8ubuntu1) ...
Setting up libtirpc1:amd64 (0.2.5-1ubuntu0.1) ...
Setting up rpcbind (0.2.3-0.2) ...
Setting up nfs-common (1:1.2.8-9ubuntu12.1) ...
Creating config file /etc/idmapd.conf with new version
Creating config file /etc/default/nfs-common with new version
Adding system user `statd' (UID 111) ...
Adding new user `statd' (UID 111) with group `nogroup' ...
Not creating home directory `/var/lib/nfs'.
nfs-utils.service is a disabled or a static unit, not starting it.
Setting up nfs-kernel-server (1:1.2.8-9ubuntu12.1) ...
Creating config file /etc/exports with new version
Creating config file /etc/default/nfs-kernel-server with new version
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.15) ...
Processing triggers for ureadahead (0.100.0-19) ...
2、配置NFS
root@storage:~# mkdir /data
root@storage:~# vim /etc/exports
root@storage:~/redis-mysql# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/data *(rw,no_root_squash)
root@storage:~# chmod 777 -R /data/
root@storage:~# systemctl restart nfs-server.service
root@storage:~# showmount -e
Export list for storage.dc.com:
/data *
安裝redis和mysql
這裏我們就直接通過docker安裝，docker-compose.yml文件內容如下：
root@storage:~/redis-mysql# cat docker-compose.yml
version: '3'
services:
mysql-server:
hostname: mysql-server
container_name: mysql-server
image: mysql:5.7
network_mode: host
volumes:
- /mysql57/data:/var/lib/mysql
command: --character-set-server=utf8
environment:
MYSQL_ROOT_PASSWORD: 123456
redis:
hostname: redis-server
container_name: redis-server
image: redis:3
network_mode: host
啓動運行
root@storage:~/redis-mysql# docker-compose up -d
Pulling mysql-server (mysql:5.7)...
5.7: Pulling from library/mysql
177e7ef0df69: Pull complete
cac25352c4c8: Pull complete
8585afabb40a: Pull complete
1e4af4996053: Pull complete
c326522894da: Pull complete
9020d6b6b171: Pull complete
55eb37ec6e5f: Pull complete
1a9d2f77e0e7: Pull complete
d7e648ad64aa: Pull complete
4120d828ea6b: Pull complete
3b39dc5451af: Pull complete
Pulling redis (redis:3)...
3: Pulling from library/redis
f17d81b4b692: Pull complete
b32474098757: Pull complete
8980cabe8bc2: Pull complete
58af19693e78: Pull complete
a977782cf22d: Pull complete
9c1e268980b7: Pull complete
Creating redis-server ... done
Creating mysql-server ... done
root@storage:~/redis-mysql# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e6533520c245 mysql:5.7 "docker-entrypoint.s…" 28 minutes ago Up 28 minutes mysql-server
a26c846816b0 redis:3 "docker-entrypoint.s…" 28 minutes ago Up 28 minutes redis-server
導入registry數據庫
配置好了mysql以後，還需要往mysql數據庫中導入harbor registry庫。安裝了一個單機版harbor，啓動了一個mysql，裏面有一個registry數據庫，直接導出來，然後再導入到新數據庫中：
# 導出數據庫：
docker exec -it harbor_db /bin/bash
mysqldump -uroot -p --databases registry > registry.dump
# 在宿主機上將registry.dump複製出來
docker cp harbor_db:/registry.dump ./
# 將宿主機上的registry.dump複製到獨立的mysql容器中
docker cp ./registry.dump <mysql-server-container>:/registry.dump
# 在獨立的mysql容器將將registry數據庫導入
docker exec -it <mysql-server-container> /bin/bash
mysql -uroot -p
mysql> source /registry.dump
配置harbor
掛載NFS目錄
root@harbor01:~# showmount -e 172.16.251.133
Export list for 172.16.251.133:
/data *
root@harbor01:~# mkdir /data
root@harbor01:~# mount 172.16.251.133:/data /data
root@harbor01:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 8.9M 1.6G 1% /run
/dev/sda3 42G 2.0G 38G 5% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda1 464M 59M 377M 14% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/0
172.16.251.133:/data 18G 2.6G 15G 16% /data
生成證書
root@harbor01:~# openssl genrsa -des3 -out server.key 1024
Generating RSA private key, 1024 bit long modulus
............++++++
..................++++++
e is 65537 (0x10001)
Enter pass phrase for server.key:
Verifying - Enter pass phrase for server.key:
root@harbor01:~# openssl rsa -in server.key -out server.key
Enter pass phrase for server.key:
root@harbor01:~# openssl req -new -key server.key -out server.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CN
State or Province Name (full name) [Some-State]:Shanghai
Locality Name (eg, city) []:Pudong
Organization Name (eg, company) [Internet Widgits Pty Ltd]:BL
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:harbor.dc.com
Email Address []:[email protected]
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
root@harbor01:~# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=CN/ST=Shanghai/L=Pudong/O=BL/OU=IT/CN=harbor.dc.com/[email protected]
Getting Private key
配置harbor.cfg
解壓harbor-offline-installer-v1.5.2.tgz進入harbor目錄
root@harbor01:~/harbor# cat harbor.cfg |grep -v '^$' |grep -v '^#'
_version = 1.5.0
hostname = harbor.dc.com
ui_url_protocol = https
max_job_workers = 50
customize_crt = on
ssl_cert = /data/certs/server.crt
ssl_cert_key = /data/certs/server.key
secretkey_path = /data
admiral_url = NA
log_rotate_count = 50
log_rotate_size = 200M
http_proxy =
https_proxy =
no_proxy = 127.0.0.1,localhost,ui
email_identity =
email_server = smtp.mydomain.com
email_server_port = 25
email_username = [email protected]
email_password = abc
email_from = admin <[email protected]>
email_ssl = false
email_insecure = false
harbor_admin_password = Harbor12345
auth_mode = db_auth
ldap_url = ldaps://ldap.mydomain.com
ldap_basedn = ou=people,dc=mydomain,dc=com
ldap_uid = uid
ldap_scope = 2
ldap_timeout = 5
ldap_verify_cert = true
ldap_group_basedn = ou=group,dc=mydomain,dc=com
ldap_group_filter = objectclass=group
ldap_group_gid = cn
ldap_group_scope = 2
self_registration = on
token_expiration = 30
project_creation_restriction = everyone
db_host = 172.16.251.133
db_password = 123456
db_port = 3306
db_user = root
redis_url = 172.16.251.133:6379
clair_db_host = postgres
clair_db_password = password
clair_db_port = 5432
clair_db_username = postgres
clair_db = postgres
uaa_endpoint = uaa.mydomain.org
uaa_clientid = id
uaa_clientsecret = secret
uaa_verify_cert = true
uaa_ca_cert = /path/to/ca.pem
registry_storage_provider_name = filesystem
registry_storage_provider_config =
部署harbor
root@harbor01:~/harbor# cp ha/docker-compose.yml .
root@harbor01:~/harbor# ./prepare
Clearing the configuration file: ./common/config/jobservice/config.yml
Clearing the configuration file: ./common/config/jobservice/env
Clearing the configuration file: ./common/config/db/env
Clearing the configuration file: ./common/config/adminserver/env
Clearing the configuration file: ./common/config/ui/app.conf
Clearing the configuration file: ./common/config/ui/env
Clearing the configuration file: ./common/config/ui/private_key.pem
Clearing the configuration file: ./common/config/registry/root.crt
Clearing the configuration file: ./common/config/registry/config.yml
Clearing the configuration file: ./common/config/log/logrotate.conf
Clearing the configuration file: ./common/config/nginx/cert/server.key
Clearing the configuration file: ./common/config/nginx/cert/server.crt
Clearing the configuration file: ./common/config/nginx/nginx.conf
loaded secret from file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/log/logrotate.conf
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.
root@harbor01:~/harbor# ./install.sh
[Step 0]: checking installation environment ...
Note: docker version: 18.09.0
Note: docker-compose version: 1.23.2
[Step 1]: loading Harbor images ...
Loaded image: vmware/registry-photon:v2.6.2-v1.5.2
Loaded image: vmware/photon:1.0
b9a9313d1f80: Loading layer [==================================================>] 457.3MB/457.3MB
a17318dff4d5: Loading layer [==================================================>] 9.216kB/9.216kB
11b0dc760d85: Loading layer [==================================================>] 9.216kB/9.216kB
ab253bfa0789: Loading layer [==================================================>] 7.68kB/7.68kB
7c41921b687e: Loading layer [==================================================>] 1.536kB/1.536kB
Loaded image: vmware/mariadb-photon:v1.5.2
Loaded image: vmware/harbor-log:v1.5.2
6c8870489188: Loading layer [==================================================>] 61.24MB/61.24MB
Loaded image: vmware/nginx-photon:v1.5.2
d6e1f5663df8: Loading layer [==================================================>] 79.23MB/79.23MB
1e7414733b92: Loading layer [==================================================>] 10.95MB/10.95MB
740974ce05ad: Loading layer [==================================================>] 17.3MB/17.3MB
8e5664ede767: Loading layer [==================================================>] 15.87kB/15.87kB
86e94116f829: Loading layer [==================================================>] 3.072kB/3.072kB
173729674517: Loading layer [==================================================>] 28.24MB/28.24MB
Loaded image: vmware/notary-signer-photon:v0.5.1-v1.5.2
22c53afb5f7e: Loading layer [==================================================>] 151MB/151MB
2b4cfcba43e3: Loading layer [==================================================>] 6.656kB/6.656kB
11d44832002d: Loading layer [==================================================>] 2.048kB/2.048kB
f805651773a5: Loading layer [==================================================>] 7.68kB/7.68kB
Loaded image: vmware/postgresql-photon:v1.5.2
b60d71996f91: Loading layer [==================================================>] 11.78kB/11.78kB
9f7d20f3523b: Loading layer [==================================================>] 2.56kB/2.56kB
1928f43498e2: Loading layer [==================================================>] 3.072kB/3.072kB
Loaded image: vmware/harbor-db:v1.5.2
74bdb194304b: Loading layer [==================================================>] 79.23MB/79.23MB
97251db4b4f7: Loading layer [==================================================>] 20.91MB/20.91MB
7c3dce375b3a: Loading layer [==================================================>] 20.91MB/20.91MB
Loaded image: vmware/harbor-jobservice:v1.5.2
3e7aac7aae19: Loading layer [==================================================>] 212.2MB/212.2MB
8e7d884dc74c: Loading layer [==================================================>] 10.93MB/10.93MB
12ed92f1f090: Loading layer [==================================================>] 2.048kB/2.048kB
81a629ba5751: Loading layer [==================================================>] 48.13kB/48.13kB
2171b5a262b0: Loading layer [==================================================>] 10.97MB/10.97MB
Loaded image: vmware/clair-photon:v2.0.4-v1.5.2
b5630b8d91ab: Loading layer [==================================================>] 79.23MB/79.23MB
89643b845018: Loading layer [==================================================>] 15.37MB/15.37MB
7b6fa7dbcef0: Loading layer [==================================================>] 15.37MB/15.37MB
Loaded image: vmware/harbor-adminserver:v1.5.2
bae5fd08868a: Loading layer [==================================================>] 79.24MB/79.24MB
ca9e737c9888: Loading layer [==================================================>] 24.41MB/24.41MB
5630a0e582d3: Loading layer [==================================================>] 7.168kB/7.168kB
68eedf2397cc: Loading layer [==================================================>] 10.56MB/10.56MB
74ac61aa9a2b: Loading layer [==================================================>] 24.4MB/24.4MB
Loaded image: vmware/harbor-ui:v1.5.2
1b5a1160526a: Loading layer [==================================================>] 136.2MB/136.2MB
24d9672d5b26: Loading layer [==================================================>] 3.072kB/3.072kB
fe4d52841f3d: Loading layer [==================================================>] 59.9kB/59.9kB
bffe30bd1941: Loading layer [==================================================>] 61.95kB/61.95kB
Loaded image: vmware/redis-photon:v1.5.2
dccbbe2d4eff: Loading layer [==================================================>] 12.16MB/12.16MB
d1291eaee2a3: Loading layer [==================================================>] 17.3MB/17.3MB
610d6a304e63: Loading layer [==================================================>] 15.87kB/15.87kB
2dabac4548fb: Loading layer [==================================================>] 3.072kB/3.072kB
46189122932e: Loading layer [==================================================>] 29.46MB/29.46MB
Loaded image: vmware/notary-server-photon:v0.5.1-v1.5.2
454c81edbd3b: Loading layer [==================================================>] 135.2MB/135.2MB
e99db1275091: Loading layer [==================================================>] 395.4MB/395.4MB
051e4ee23882: Loading layer [==================================================>] 9.216kB/9.216kB
6cca4437b6f6: Loading layer [==================================================>] 9.216kB/9.216kB
1d48fc08c8bc: Loading layer [==================================================>] 7.68kB/7.68kB
0419724fd942: Loading layer [==================================================>] 1.536kB/1.536kB
543c0c1ee18d: Loading layer [==================================================>] 655.2MB/655.2MB
4190aa7e89b8: Loading layer [==================================================>] 103.9kB/103.9kB
Loaded image: vmware/harbor-migrator:v1.5.0
[Step 2]: preparing environment ...
Clearing the configuration file: ./common/config/jobservice/config.yml
Clearing the configuration file: ./common/config/jobservice/env
Clearing the configuration file: ./common/config/db/env
Clearing the configuration file: ./common/config/adminserver/env
Clearing the configuration file: ./common/config/ui/app.conf
Clearing the configuration file: ./common/config/ui/env
Clearing the configuration file: ./common/config/ui/private_key.pem
Clearing the configuration file: ./common/config/registry/root.crt
Clearing the configuration file: ./common/config/registry/config.yml
Clearing the configuration file: ./common/config/log/logrotate.conf
Clearing the configuration file: ./common/config/nginx/cert/server.key
Clearing the configuration file: ./common/config/nginx/cert/server.crt
Clearing the configuration file: ./common/config/nginx/nginx.conf
loaded secret from file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/log/logrotate.conf
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.
[Step 3]: checking existing instance of Harbor ...
[Step 4]: starting Harbor ...
Creating harbor-log ... done
Creating registry ... done
Creating harbor-adminserver ... done
Creating harbor-ui ... done
Creating nginx ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----
Now you should be able to visit the admin portal at https://harbor01.dc.com.
For more details, please visit https://github.com/vmware/harbor .
root@harbor01:~/harbor# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cdf3685c2d94 vmware/harbor-jobservice:v1.5.2 "/harbor/start.sh" 13 seconds ago Up 12 seconds harbor-jobservice
5de16bad4f2d vmware/nginx-photon:v1.5.2 "nginx -g 'daemon of…" 13 seconds ago Up 12 seconds (health: starting) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp nginx
15406c90e365 vmware/harbor-ui:v1.5.2 "/harbor/start.sh" 14 seconds ago Up 13 seconds (health: starting) harbor-ui
9e7330ce8130 vmware/harbor-adminserver:v1.5.2 "/harbor/start.sh" 14 seconds ago Up 13 seconds (health: starting) harbor-adminserver
d344a7ae027e vmware/registry-photon:v2.6.2-v1.5.2 "/entrypoint.sh serv…" 14 seconds ago Up 13 seconds (health: starting) 5000/tcp registry
3c84cf5271f6 vmware/harbor-log:v1.5.2 "/bin/sh -c /usr/loc…" 15 seconds ago Up 14 seconds (health: starting) 127.0.0.1:1514->10514/tcp harbor-log
root@harbor01:~/harbor#
harbor測試
安裝配置LVS
root@Userver05:~# apt install ipvsadm keepalived
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libnl-route-3-200 libsensors4 libsnmp-base libsnmp30
Suggested packages:
heartbeat ldirectord lm-sensors snmp-mibs-downloader
The following NEW packages will be installed:
ipvsadm keepalived libnl-route-3-200 libsensors4 libsnmp-base libsnmp30
0 upgraded, 6 newly installed, 0 to remove and 9 not upgraded.
Need to get 1,429 kB of archives.
After this operation, 5,570 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libnl-route-3-200 amd64 3.2.27-1ubuntu0.16.04.1 [124 kB]
Get:2 http://mirrors.aliyun.com/ubuntu xenial/main amd64 libsensors4 amd64 1:3.4.0-2 [28.4 kB]
Get:3 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libsnmp-base all 5.7.3+dfsg-1ubuntu4.2 [224 kB]
Get:4 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 libsnmp30 amd64 5.7.3+dfsg-1ubuntu4.2 [813 kB]
Get:5 http://mirrors.aliyun.com/ubuntu xenial-updates/main amd64 keepalived amd64 1:1.2.24-1ubuntu0.16.04.1 [205 kB]
Get:6 http://mirrors.aliyun.com/ubuntu xenial/main amd64 ipvsadm amd64 1:1.28-3 [35.5 kB]
Fetched 1,429 kB in 0s (2,824 kB/s)
Selecting previously unselected package libnl-route-3-200:amd64.
(Reading database ... 60370 files and directories currently installed.)
Preparing to unpack .../libnl-route-3-200_3.2.27-1ubuntu0.16.04.1_amd64.deb ...
Unpacking libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) ...
Selecting previously unselected package libsensors4:amd64.
Preparing to unpack .../libsensors4_1%3a3.4.0-2_amd64.deb ...
Unpacking libsensors4:amd64 (1:3.4.0-2) ...
Selecting previously unselected package libsnmp-base.
Preparing to unpack .../libsnmp-base_5.7.3+dfsg-1ubuntu4.2_all.deb ...
Unpacking libsnmp-base (5.7.3+dfsg-1ubuntu4.2) ...
Selecting previously unselected package libsnmp30:amd64.
Preparing to unpack .../libsnmp30_5.7.3+dfsg-1ubuntu4.2_amd64.deb ...
Unpacking libsnmp30:amd64 (5.7.3+dfsg-1ubuntu4.2) ...
Selecting previously unselected package keepalived.
Preparing to unpack .../keepalived_1%3a1.2.24-1ubuntu0.16.04.1_amd64.deb ...
Unpacking keepalived (1:1.2.24-1ubuntu0.16.04.1) ...
Selecting previously unselected package ipvsadm.
Preparing to unpack .../ipvsadm_1%3a1.28-3_amd64.deb ...
Unpacking ipvsadm (1:1.28-3) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.15) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) ...
Setting up libsensors4:amd64 (1:3.4.0-2) ...
Setting up libsnmp-base (5.7.3+dfsg-1ubuntu4.2) ...
Setting up libsnmp30:amd64 (5.7.3+dfsg-1ubuntu4.2) ...
Setting up keepalived (1:1.2.24-1ubuntu0.16.04.1) ...
Setting up ipvsadm (1:1.28-3) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.15) ...
Processing triggers for ureadahead (0.100.0-19) ...
root@Userver05:~# cat /etc/keepalived/keepalived.conf
global_defs {
router_id haborlb
}
vrrp_sync_groups VG1 {
group {
VI_1
}
}
#Please change "ens160" to the interface name on you loadbalancer hosts.
#In some case it will be eth0, ens16xxx etc.
vrrp_instance VI_1 {
interface ens32
track_interface {
ens32
}
state BACKUP
virtual_router_id 51
priority 8
virtual_ipaddress {
172.16.251.136/32
}
advert_int 1
authentication {
auth_type PASS
auth_pass d0cker
}
}
########## Section for Harbor use HTTP protocol ######################
#Please change <change_to_VIP_address>, <harbor_node1_ip>, <harbor_node2_ip> to real ip address
#virtual_server <change_to_VIP_address> 80 {
# delay_loop 15
# lb_algo rr
# lb_kind DR
# protocol TCP
# nat_mask 255.255.255.0
# persistence_timeout 10
#
# real_server <harbor_node1_ip> 80 {
# weight 10
# MISC_CHECK {
# misc_path "/usr/local/bin/check.sh <harbor_node1_ip>"
# misc_timeout 5
# }
# }
#
# real_server <harbor_node2_ip> 80 {
# weight 10
# MISC_CHECK {
# misc_path "/usr/local/bin/check.sh <harbor_node2_ip>"
# misc_timeout 5
# }
# }
#}
#########################End of HTTP############################
##########################HTTPS#################################
#Please uncomment the follow when harbor running under https
virtual_server 172.16.251.136 443 {
delay_loop 15
lb_algo rr
lb_kind DR
protocol TCP
nat_mask 255.255.255.0
persistence_timeout 10
real_server 172.16.251.131 443 {
weight 10
MISC_CHECK {
misc_path "/usr/local/bin/check.sh 172.16.251.131"
misc_timeout 5
}
}
real_server 172.16.251.132 443 {
weight 10
MISC_CHECK {
misc_path "/usr/local/bin/check.sh 172.16.251.132"
misc_timeout 5
}
}
}
#########################End of HTTPS Section#################
root@Userver05:~# cat /usr/local/bin/check.sh
#!/bin/bash
set -e
#get protocol
#LOG=/var/log/keepalived_check.log
nodeip=$1
nodeaddress="http://${nodeip}"
http_code=`curl -s -o /dev/null -w "%{http_code}" ${nodeaddress}`
if [ $http_code == 200 ] ; then
protocol="http"
elif [ $http_code == 301 ]
then
protocol="https"
else
# echo "`date +"%Y-%m-%d %H:%M:%S"` $1, CHECK_CODE=$http_code" >> $LOG
exit 1
fi
systeminfo=`curl -k -o - -s ${protocol}://${nodeip}/api/systeminfo`
echo $systeminfo | grep "registry_url"
if [ $? != 0 ] ; then
exit 1
fi
#TODO need to check Clair, but currently Clair status api is unreachable from LB.
# echo $systeminfo | grep "with_clair" | grep "true"
# if [ $? == 0 ] ; then
# clair is enabled
# do some clair check
# else
# clair is disabled
# fi
#check top api
http_code=`curl -k -s -o /dev/null -w "%{http_code}\n" ${protocol}://${nodeip}/api/repositories/top`
set +e
if [ $http_code == 200 ] ; then
exit 0
else
exit 1
fi
root@Userver04:~# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.251.136:https rr persistent 10
-> 172.16.251.131:https Route 10 2 0
-> 172.16.251.132:https Route 10 0 0
root@Userver04:~#
測試
root@Userver04:~# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.251.136:https rr persistent 10
-> 172.16.251.132:https Route 10 6 1
root@Userver04:~#
