k8s(十三)、企業級docker倉庫Harbor在kubernetes上搭建使用

前言

Harbor是Docker鏡像倉庫管理系統，用於企業級管理容器鏡像，支持對接ldap進行權限管理,由VMVare中國團隊開發。
項目地址:
https://github.com/goharbor/harbor/
安裝方式:
1.Docker Compose方式部署:
https://github.com/goharbor/harbor/blob/master/docs/installation_guide.md
2.通過kubernetes直接部署:
新版本已廢棄
3.通過helm/kubernetes方式部署:
https://github.com/goharbor/harbor-helm/blob/master/README.md
推薦使用,本文將採用此方式

Helm安裝

helm簡介

Helm是一個用於Kubernetes的包管理工具,每一個安裝包構成一個chart基本單位.想象一下,一個完整的部署,可能包含前端\後端\中間件\DB的部署實例,以及k8s管理對應的secret/configmap/svc/ing/pv/pvc等衆多資源類型,在部署管理或者卸載之前,如果逐一編輯管理這些資源的yaml文件,工作量比較龐大且容易疏漏.helm的出現很好的解決了這個問題,使用模板以及變量替換的方式,按需生成部署yaml文件包,並且向kubernetes api發送調度請求,雖然額外地引入了一些學習成本,但對於部署包的管理無疑是帶來了很大的便利性的.

helm組件

Helm
Helm是一個cli客戶端，可以完成如下內容：

1.創建/打包/調試Chart
2.創建本地Chart倉庫,管理本地和遠程Chart倉庫
3.與Tiller交互並完成Chart的安裝，升級，刪除，回滾，查看等操作

Tiller
1.監聽helm客戶端的請求,根據chart生成相應的release
2.將release的資源信息發送給kubernetes api,實現資源增刪改查.
3.追蹤release的狀態,實現資源的更新/回滾操作

安裝:
安裝較爲簡單,參考官方文檔,不做複述:
https://docs.helm.sh/using_helm/#installing-helm

Harbor安裝

這裏採用次新版本的v1.5.3版本

root@h1:# wget https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz
root@h1:# tar -xf https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz 
root@h1:# cd harbor-1.5.3/contrib/helm/harbor/

helm方式安裝的harbor默認是https的,因此需要k8s集羣ingress網關開啓https協議,traefik開啓https參考此前文章:
k8s（二）、對外服務

更新helm dependency:

harbor的helm部署依賴了postgresql的helm，在官方的安裝文檔沒有明確說明，直接按照官方文檔說明安裝，就會缺失postgresql的部署，導致helm install failed

helm dependency update

修改value.yaml文件,添加pvc持久存儲並將相應組件上下文中的volume部分註釋打開:

mysql:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    
adminserver:
  volumes:
    config:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 1G

registry:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 5G

完整value.yaml文件如下:

# Configure persisten Volumes per application
## Applications that require storage have a `volumes` definition which will be used
## when `persistence.enabled` is set to true.
## example
# mysql:
#   volumes:
#     data:
## Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
##   set, choosing the default provisioner.  (gp2 on AWS, standard on
##   GKE, AWS & OpenStack)
##
#     storageClass: "-"
#     accessMode: ReadWriteOnce
#     size: 1Gi

mysql:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
adminserver:
  volumes:
    config:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 1G

registry:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 5G

## Configure resource requests and limits per application
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
# mysql:
#   resources:
#     requests:
#       memory: 256Mi
#       cpu: 100m

persistence:
  enabled: true

# The tag for Harbor docker images.
harborImageTag: &harbor_image_tag v1.4.0

# The FQDN for Harbor service.
externalDomain: harbor.my.domain
# If set to true, you don't need to set tlsCrt/tlsKey/caCrt, but must add
# Harbor FQDN as insecure-registries for your docker client.
insecureRegistry: false
#insecureRegistry: true
# The TLS certificate for Harbor. The common name of tlsCrt must match the externalDomain above.
tlsCrt: 
#tlsCrt: root/traefik/ssl/kokoerp.crt
tlsKey:
#tlsKey: root/traefik/ssl/kokoerp.key
caCrt:

# The secret key used for encryption. Must be a string of 16 chars.
secretKey: not-a-secure-key

# These annotations allow the registry to work behind the nginx
# ingress controller.
ingress:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/body-size: "0"
    ingress.kubernetes.io/proxy-body-size: "0"

adminserver:
  image:
    repository: vmware/harbor-adminserver
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  emailHost: "smtp.mydomain.com"
  emailPort: "25"
  emailUser: "[email protected]"
  emailSsl: "false"
  emailFrom: "admin <[email protected]>"
  emailIdentity: ""
  emailInsecure: "False"
  emailPwd: not-a-secure-password
  adminPassword: Harbor12345
  authenticationMode: "db_auth"
  selfRegistration: "on"
  ldap:
    url: "ldaps://ldapserver"
    searchDN: ""
    baseDN: ""
    filter: "(objectClass=person)"
    uid: "uid"
    scope: "2"
    timeout: "5"
    verifyCert: "True"
  ## Persist data to a persistent volume
  volumes:
    config:
      storageClass: "cephrbd"
      accessMode: ReadWriteOnce
      size: 1Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

## jobservice
#
jobservice:
  image:
    repository: vmware/harbor-jobservice
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  secret: not-a-secure-secret
# resources:
#   requests:
#     memory: 256Mi
#     cpu: 100m

## UI
#
ui:
  image:
    repository: vmware/harbor-ui
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  secret: not-a-secure-secret
  privateKeyPem: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIJKAIBAAKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnM
    bBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPR
    AiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96
    /zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZz
    e0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9Iq
    YXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/
    GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw2
    2YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbT
    aKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZ
    qVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGm
    Ib0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEA
    AQKCAgBEUigO8/4UJse6xKr3APHv7E94NjKtjMqPT8RhDCLhqAH/lRuClTVb8k0Y
    RILi6oHggsKGDvkS1vJEESCU5LfYBjDAX/r/M0I7gp6TU1AukAXKMdETvkfoMbg/
    9j7W/G152hF4KztvjwmcHyUd7aay+SDh0n1taPm/FzaXfgONwmQFmo40uQ2SfwhX
    I3tD6iMWjASLV4eRfe5w88WpJQ3r5IGYMNuKFF1RcV7MNL3xMHBAwl1kudmRWY4w
    p6+83Gc0m+2AQbY70TkQuRbeUFkIBsWn99yEqXC+7h2us+JLm57iGN1ByQvVnEwL
    Zs7Pl0Hge4leSxeZWhv+aE1R/jm/VdG4dglInuhED0ug8WAJg58IkDYfMKOOALHx
    +0CNHE02XqqUIFwboZJSYTjMYvFL1i14L30FWnqH/0kDs4whXHbnGWhVustsMSK9
    iyIGepuGhMnvtUF1wa/SrBd12qfDj68QHDXsKKbs6eTNYHfn3QL9uisrfMIa5HAt
    nX2YOsAVxg+yvxkWD6n1DU+a/+pAu6iAgiwyxSZiyn6vJUE2zO6pJNbk1kJW6jU3
    A69srtbO4jQn4EM859XYSqdqwXgJL+XJEYNbBcHalmiIOvRg9CCvDSKS7M5rJ0M1
    L7oCzl6EW+zUb4JHkSO7V5uxIZu2sEduw5gofQ3OT9L/qDhDIQKCAQEA8T/8okF2
    Q7SOj3su6KKX6H/ab31SvHECf/oeJtH8ZfLBYL55Yof0pZwq8iXQ26d8cH7FPKBo
    hz0RZ9i2S3bYkzEVCPv9ISFg1NACxL3dU0PMBnmbmg2vPhMzEuQI2JOUu6ILOXEN
    mImvfjZXps/b8OjQgzicH0skBBcbUlXT3a4fF52ktC8FiXgBG9JYg5CsXmfPRxci
    ITa4w4ZLEuECmtJieS0MdKXPLwUVv3e2BlNS6c1JzXyp6EyX/euJ8cCe3n/GHbTY
    2j1OO+xTQfQJVf6S9f2mSzjdHe9KZwWKgyxQ9dZ9Qtho2z/gUN9/UkL52fdljjlw
    ++b/z9Ppcl9K0QKCAQEA7y4Fv8dPFLLnr0R/S7eoAKa0S95xVe97EJHVUhWyOI09
    K9VdZHp6be8W0Yd9h/Ks8Zi4EPRiTTaF3yA3iADwdKFeZt49jGzeM+Gl7Q2Ll98W
    I5gOdJkHSVAP2uK7qSjZ8lPCu4iUYRsae+Psam7Yd6X17RP0M966PlUFj1nnrJjQ
    EN4zeh/m01q9vqebB9C1W/ZiJ6rpt6VVHAcOQQ69F/lKdTif4XCvbMIhIXTYNifk
    1oIv2qTDnfzzv+bgrlvpBJPpPYR0Oc7WoEpyd1Y9IzienLZi8RnujV//FXEmJ45E
    F9GE1HOmoERdEWA1bMYhOO5OfRY1HSMuFMA4+5ojSwKCAQEAmwubio/1uMemw3HQ
    kPRGGsdolDR/4tniWGtfy2UzCDY+r7Vaf8eOpIy8UQmatEBsykO+8RrKcvf9Yrc1
    WUSVJevqb+67HPq9p6fTz6uSPXwZ+KNZLGXVFVjzfxWM1dvrP7eB7TXKHhmG7t9v
    76Yw3SBTObI9LCN3jyVmisDcO+E23E+VVbPOpC260K2b81ocXUPsQ+0LIztu/UIm
    p4hyyxug6+3WznTttXNYKch+9IvCgr5Ly0NuUvw+xpMFAZjgwXBu3BKpN4Ek8YAN
    dhqnkVveCTguErQF78IlGBbIkUr+8TAbKsW4hggEWxV4V17yAnJsEz65bTtldqTj
    qHyzsQKCAQBGhv6g/2d9Rgf1cbBLpns+vel6Wbx3x6c1SptpmgY0kMlR7JeeclM5
    qX/EBzzn4pJGp27XaQi3lfVBxyE41HYTHiZVFQF3L/8Rs18XGKBqBxljI4pXrWwt
    nRMfyy3lAqvJvhM082A1hiV4FMx40fi4x1JON00SIoIusSlzjOI4zdLEtpDdWRza
    g+5hktCvLEbeODfXVJmYUoNXQWldm7f8osDm8eyLMIw5+MCGOgsrZPYgnsD3qxAX
    vSgvFSh5oZaDiA4F2tHe3fQBzhIUyHQ8t4xlz447ZBcozv7L1tKWZWgE0f5mGzgu
    GBqNbh4y1fWj8Plp/ytoTSBgdBIZdukjAoIBAELJPSVFnlf/gv6OWRCHyKxquGjv
    fEn/E8bw5WSqMcj/7wiSJozr0Y8oyWjtWXObliLRQXcEhC8w3lLMjNqnFzQOAI7s
    Oa6BQPigqyXZPXG5GK+V0TlUYvZQn9sfCq4YCxUBNtQ4GHbKKl3FGQL3rJiuFr6G
    fVcetuDFNCiIGYbUF+giJ2cEN3a/Q+7fR6V4xC7VDdL+BqM09wZ6R98G48XzCKKp
    ekNpEfmvJiuk9tFFQwDPWcQ6uyHqesK/Wiweo5nh5y2ZPipwcb0uBoYOQH60NqEL
    6MXRVNdtKujjl1XZkG053Nvcz/YfF6lFjDekwgfd9m49b/s0EGTrl7z9z8Y=
    -----END RSA PRIVATE KEY-----
# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m

## MySQL Settings. Currently Harbor does not support an external
## MySQL server, only their own image. Until this is fixed, do not
## Change the settings below.
#
mysql:
  image:
    repository: vmware/harbor-db
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  # If left blank will use the included mysql service name.
  host: ~
  port: 3306
  user: "root"
  pass: "registry"
  database: "registry"
  volumes:
    data:
      storageClass: "cephrbd"
      accessMode: ReadWriteOnce
      size: 1Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

registry:
  image:
    repository: vmware/registry-photon
    tag: v2.6.2-v1.4.0
    pullPolicy: IfNotPresent
  httpSecret: not-a-secure-secret
  logLevel:
# comment out one of the below to use your cloud's object storage.
#  objectStorage:
#    gcs:
#      keyfile: ""
#      bucket: ""
#      chunksize: "5242880"
#    s3:
#      region: ""
#      accesskey: ""
#      secretkey: ""
#      bucket: ""
#      encrypt: "true"
#    azure:
#      accountname: ""
#      accountkey: ""
#      container: ""
  rootCrt: |
    -----BEGIN CERTIFICATE-----
    MIIE0zCCArugAwIBAgIJAIgs3S+hsjhmMA0GCSqGSIb3DQEBCwUAMAAwHhcNMTcx
    MTA5MTcyNzQ5WhcNMjcxMTA3MTcyNzQ5WjAAMIICIjANBgkqhkiG9w0BAQEFAAOC
    Ag8AMIICCgKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnM
    bBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPR
    AiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96
    /zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZz
    e0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9Iq
    YXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/
    GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw2
    2YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbT
    aKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZ
    qVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGm
    Ib0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEA
    AaNQME4wHQYDVR0OBBYEFCMYYMOL0E/Uyj5wseDfIl7o4ELsMB8GA1UdIwQYMBaA
    FCMYYMOL0E/Uyj5wseDfIl7o4ELsMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEL
    BQADggIBABG8fPvrrR+erpwQFuB/56j2i6sO+qoOJPpAMYwkzICrT0eerWAavwoy
    f0UAKN7cUeEJXjIR7s7CogGFijWdaWaQsXUD0zJq5aotLYZLimEc1O0uAmJEsfYC
    v7mG07eU6ge22sSo5hxhVplGt52hnXnT0DdgSRZpq2mvgd9lcopAidM+KHlaasXk
    IecHKM99KX9D8smr0AcQ6M/Ygbf2qjO9YRmpBIjyQWEake4y/4LWm+3+v08ecg4B
    g+iMC0Rw1QcPqgwaGaWu71RtYhyTg7SnAknb5nBcHIbLb0hdLgQTa3ZdtXgqchIi
    GuFlEBmHFZP6bLJORRUQ0ari5wpXIsYfrB4T8PybTzva3OCMlEsMjuysFr9ewhzM
    9UGLiSQNDyKA10J8WwlzbeD0AAW944hW4Dbg6SWv4gAo51T+6AukRdup5y6lfQ5a
    h4Lbo6pzaA369IsJBntvKvia6hUf/SghnbG7pCHX/AEilcgTb13HndF/G+7aZgKR
    mi9qvNRSDsE/BrgZawovp81+j6aL4y6UtXYspHr+SuWsKYsaH7pl5HspNCyJ5vV6
    dpJAwosFBqSEnI333wAunpMYmi/jKHH/j4WqjLnCInp0/wouzYu42l8Pmz591BSp
    Jag500bEBxqI2RLELgMt/bUdjp4N2M7mrxdrN+2579HTzb6Hviu9
    -----END CERTIFICATE-----
  ## Persist data to a persistent volume
  volumes:
    data:
      storageClass: "cephrbd"
      accessMode: ReadWriteOnce
      size: 5Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

clair:
  enabled: true
  image:
    repository: vmware/clair-photon
    tag: v2.0.1-v1.4.0
    pullPolicy: IfNotPresent
## The following needs to match the credentials
## in the `postgresql` configuration under the
## `postgresql` namespace below.
  postgresPassword: not-a-secure-password
  postgresUser: clair
  postgresDatabase: clair
# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m
# pgResources:
#  requests:
#    memory: 256Mi
#    cpu: 100m
#  volumes:
#    pgData:
#      storageClass: "cephrbd"
#      accessMode: ReadWriteOnce
#      size: 1Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

## Notary support is not yet fully implemented in the Helm Charts
## Enabling it will just break things.
#
notary:
  enabled: false

## Settings for postgresql dependency.
## see https://github.com/kubernetes/charts/tree/master/stable/postgresql
## for further configurables.
postgresql:
  postgresUser: clair
  postgresPassword: not-a-secure-password
  postgresDatabase: clair
  persistence:
    # enabled: false
    enabled: true
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 8Gi

部署:

helm install . --debug --name hub --set externalDomain=hub.test.com

等待一段時間的鏡像拉取後,查看pod:

hub-harbor-adminserver-0                 1/1       Running            1          3h
hub-harbor-clair-6c7d9dcdb7-q4lv4        1/1       Running            2          3h
hub-harbor-jobservice-75f7fbcc9c-ggwp4   1/1       Running            2          3h
hub-harbor-mysql-0                       1/1       Running            0          3h
hub-harbor-registry-0                    1/1       Running            1          3h
hub-harbor-ui-57b4674ff9-kcfq6           1/1       Running            0          3h
hub-postgresql-ccf8d56d5-jg4wq           1/1       Running            1          3h

添加dns,指向ingress gateway(即traefik的node ip):

echo "192.168.1.238  hub.test.com"  >> /etc/hosts

瀏覽器打開測試,默認登錄口令爲admin/Harbor12345

可以看到默認有一個公開項目library,嘗試往這裏推送鏡像

Docker推送測試

找一臺安裝了docker的機器,修改docker服務腳本:

~# cat /lib/systemd/system/docker.service

#ExecStart=/usr/bin/dockerd -H fd://
ExecStart=/usr/bin/dockerd  --insecure-registry hub.kokoerp.com

重啓docker並登錄測試:

~# systemctl daemon-reload
~# systemctl restart docker

root@h2:~# docker login hub.test.com
Username (admin): admin
Password: 
Login Succeeded

推送鏡像測試:

root@h2:~# docker tag busybox hub.test.com/library/busybox:test1
root@h2:~# docker push hub.test.com/library/busybox:test1
The push refers to a repository [hub.test.com/library/busybox]
8a788232037e: Pushed 
test1: digest: sha256:915f390a8912e16d4beb8689720a17348f3f6d1a7b659697df850ab625ea29d5 size: 527

可以看到推送成功

ldap配置:

harbor可以基於企業內部的ldap來獲取人員信息,可以在ui上創建非公開項目,將ldap中獲取的人員加入項目並賦予相應pull push權限,ui操作界面比較友好,這裏就不展示了.