harbor 2.0 搭建docker私有仓库

harbor

Harbor 是一个CNCF基金会托管的开源的可信的云原生docker registry项目，可以用于存储、签名、扫描镜像内容，Harbor 通过添加一些常用的功能如安全性、身份权限管理等来扩展 docker registry 项目，此外还支持在 registry 之间复制镜像，还提供更加高级的安全功能，如用户管理、访问控制和活动审计等，在新版本中还添加了Helm仓库托管的支持。

Harbor最核心的功能就是给 docker registry 添加上一层权限保护的功能，要实现这个功能，就需要我们在使用 docker login、pull、push 等命令的时候进行拦截，先进行一些权限相关的校验，再进行操作，其实这一系列的操作 docker registry v2 就已经为我们提供了支持，v2 集成了一个安全认证的功能，将安全认证暴露给外部服务，让外部服务去实现。

环境准备

linux 3.10.0-957.5.1.el7.x86_64 centos 7.6.1810 配置 2c2g500g

安装

docker 安装

docker 为centos用户提供了三种安装方式，我们选择第一种，也是官网推荐的安装方式

移除旧的docker 依赖 sudo yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine 设置docker 源 sudo yum install -y yum-utils sudo yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo 安装docker引擎，默认最新版本 sudo yum install docker-ce docker-ce-cli containerd.io 启动 systemctl enable docker systemctl start docker 测试 docker -v Docker version 19.03.12, build 48a66213fe

docker-compose 安装

下载可执行文件 sudo curl -L "https://github.com/docker/compose/releases/download/1.26.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose 修改权限 sudo chmod +x /usr/local/bin/docker-compose 测试 docker-compose -v docker-compose version 1.26.0, build d4451659

harbor安装

下载在线安装包 harbor可通过在线和离线安装，由于离线安装包下载失败，所以选择在线安装 wget https://github.com/goharbor/harbor/releases/download/v2.0.0/harbor-online-installer-v2.0.0.tgz 解压 tar xvf harbor-online-installer-v2.0.0.tgz 修改配置 harbor通过docker-compose 管理，harbor.yml 是compose 配置文件 备份配置文件：cp harbor.yml.tmpl harbor.yml 修改配置：vim harbor.yml # Configuration file of Harbor # The IP address or hostname to access admin UI and registry service. # DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients. hostname: xx.xx.xx.xx # 对外开发访问的 ip或域名 # http related config http: # port for http, default is 80. If https enabled, this port will redirect to https port port: 80 # 注释掉https,当然你也可以通过设置证书，开放https，建议生产使用https # https related config #https: # https port for harbor, default is 443 # port: 443 # The path of cert and key files for nginx # certificate: /your/certificate/path # private_key: /your/private/key/path # # Uncomment following will enable tls communication between all harbor components # internal_tls: # # set enabled to true means internal tls is enabled # enabled: true # # put your cert and key files on dir # dir: /etc/harbor/tls/internal # Uncomment external_url if you want to enable external proxy # And when it enabled the hostname will no longer used # external_url: https://reg.mydomain.com:8433 # The initial password of Harbor admin # It only works in first time to install harbor # Remember Change the admin password from UI after launching Harbor. harbor_admin_password: Harbor12345 # Harbor DB configuration database: # The password for the root user of Harbor DB. Change this before any production use. password: root123 # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained. max_idle_conns: 50 # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections. # Note: the default number of connections is 100 for postgres. max_open_conns: 100 # The default data volume data_volume: /data/harbor # Harbor Storage settings by default is using /data dir on local filesystem # Uncomment storage_service setting If you want to using external storage # storage_service: # # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore # # of registry's and chart repository's containers. This is usually needed when the user hosts a internal storage with self signed certificate. # ca_bundle: # # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss # # for more info about this configuration please refer https://docs.docker.com/registry/configuration/ # filesystem: # maxthreads: 100 # # set disable to true when you want to disable registry redirect # redirect: # disabled: false # Clair configuration clair: # The interval of clair updaters, the unit is hour, set to 0 to disable the updaters. updaters_interval: 12 # Trivy configuration trivy: # ignoreUnfixed The flag to display only fixed vulnerabilities ignore_unfixed: false # skipUpdate The flag to enable or disable Trivy DB downloads from GitHub # # You might want to enable this flag in test or CI/CD environments to avoid GitHub rate limiting issues. # If the flag is enabled you have to manually download the `trivy.db` file and mount it in the # /home/scanner/.cache/trivy/db/trivy.db path. skip_update: false # # insecure The flag to skip verifying registry certificate insecure: false # github_token The GitHub access token to download Trivy DB # # Trivy DB contains vulnerability information from NVD, Red Hat, and many other upstream vulnerability databases. # It is downloaded by Trivy from the GitHub release page https://github.com/aquasecurity/trivy-db/releases and cached # in the local file system (/home/scanner/.cache/trivy/db/trivy.db). In addition, the database contains the update # timestamp so Trivy can detect whether it should download a newer version from the Internet or use the cached one. # Currently, the database is updated every 12 hours and published as a new release to GitHub. # # Anonymous downloads from GitHub are subject to the limit of 60 requests per hour. Normally such rate limit is enough # for production operations. If, for any reason, it's not enough, you could increase the rate limit to 5000 # requests per hour by specifying the GitHub access token. For more details on GitHub rate limiting please consult # https://developer.github.com/v3/#rate-limiting # # You can create a GitHub token by following the instuctions in # https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line # # github_token: xxx jobservice: # Maximum number of job workers in job service max_job_workers: 10 notification: # Maximum retry count for webhook job webhook_job_max_retry: 10 chart: # Change the value of absolute_url to enabled can enable absolute url in chart absolute_url: disabled # Log configurations log: # options are debug, info, warning, error, fatal level: info # configs for logs in local storage local: # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated. rotate_count: 50 # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes. # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G # are all valid. rotate_size: 200M # The directory on your host that store log location: /var/log/harbor # Uncomment following lines to enable external syslog endpoint. # external_endpoint: # # protocol used to transmit log to external endpoint, options is tcp or udp # protocol: tcp # # The host of external endpoint # host: localhost # # Port of external endpoint # port: 5140 #This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY! _version: 2.0.0 # Uncomment external_database if using external database. # external_database: # harbor: # host: harbor_db_host # port: harbor_db_port # db_name: harbor_db_name # username: harbor_db_username # password: harbor_db_password # ssl_mode: disable # max_idle_conns: 2 # max_open_conns: 0 # clair: # host: clair_db_host # port: clair_db_port # db_name: clair_db_name # username: clair_db_username # password: clair_db_password # ssl_mode: disable # notary_signer: # host: notary_signer_db_host # port: notary_signer_db_port # db_name: notary_signer_db_name # username: notary_signer_db_username # password: notary_signer_db_password # ssl_mode: disable # notary_server: # host: notary_server_db_host # port: notary_server_db_port # db_name: notary_server_db_name # username: notary_server_db_username # password: notary_server_db_password # ssl_mode: disable # Uncomment external_redis if using external Redis server # external_redis: # host: redis # port: 6379 # password: # # db_index 0 is for core, it's unchangeable # registry_db_index: 1 # jobservice_db_index: 2 # chartmuseum_db_index: 3 # clair_db_index: 4 # trivy_db_index: 5 # idle_timeout_seconds: 30 # Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert. # uaa: # ca_file: /path/to/ca # Global proxy # Config http proxy for components, e.g. http://my.proxy.com:3128 # Components doesn't need to connect to each others via http proxy. # Remove component from `components` array if want disable proxy # for it. If you want use proxy for replication, MUST enable proxy # for core and jobservice, and set `http_proxy` and `https_proxy`. # Add domain to the `no_proxy` field, when you want disable proxy # for some special registry. proxy: http_proxy: https_proxy: no_proxy: components: - core - jobservice - clair - trivy 安装 cd /home/admin/harbor/ ./install.sh

脚本执行完成后查看harbor组成 和运行状态

docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3c73fc1bdf49 goharbor/nginx-photon:v2.0.0 "nginx -g 'daemon of…" 44 hours ago Up 44 hours (healthy) 0.0.0.0:80->8080/tcp nginx cc58e0fd2347 goharbor/harbor-jobservice:v2.0.0 "/harbor/entrypoint.…" 44 hours ago Up 44 hours (healthy) harbor-jobservice 7f5929dba6bc goharbor/harbor-core:v2.0.0 "/harbor/entrypoint.…" 44 hours ago Up 44 hours (healthy) harbor-core 001167acd8fc goharbor/harbor-db:v2.0.0 "/docker-entrypoint.…" 44 hours ago Up 44 hours (healthy) 5432/tcp harbor-db 1684303e0782 goharbor/harbor-registryctl:v2.0.0 "/home/harbor/start.…" 44 hours ago Up 44 hours (healthy) registryctl aa4b212f6595 goharbor/redis-photon:v2.0.0 "redis-server /etc/r…" 44 hours ago Up 44 hours (healthy) 6379/tcp redis 79189cffe12a goharbor/registry-photon:v2.0.0 "/home/harbor/entryp…" 44 hours ago Up 44 hours (healthy) 5000/tcp registry c1f45a3e3131 goharbor/harbor-portal:v2.0.0 "nginx -g 'daemon of…" 44 hours ago Up 44 hours (healthy) 8080/tcp harbor-portal 4c85974a6c25 goharbor/harbor-log:v2.0.0 "/bin/sh -c /usr/loc…" 44 hours ago Up 44 hours (healthy) 127.0.0.1:1514->10514/tcp harbor-log docker-compose ps Name Command State Ports --------------------------------------------------------------------------------------------- harbor-core /harbor/entrypoint.sh Up (healthy) harbor-db /docker-entrypoint.sh Up (healthy) 5432/tcp harbor-jobservice /harbor/entrypoint.sh Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp nginx nginx -g daemon off; Up (healthy) 0.0.0.0:80->8080/tcp redis redis-server /etc/redis.conf Up (healthy) 6379/tcp registry /home/harbor/entrypoint.sh Up (healthy) 5000/tcp registryctl /home/harbor/start.sh Up (healthy) 访问 harbor 使用默认用户名密码登录，登录后可修改 admin : Harbor12345

Harbor 组成

Harbor 配置

配置认证方式 harbor 支持 数据库,ldap/ad, oidc等认证方式，这里配置ldap的认证方式，如图，选择LDAP，并填写 LDAP相关信息，关于LDAP和AD可考虑内网协议学习LDAP篇之组和OU介绍，配置好后，点击测试，通过即可

注意：选择ldap后，用户认证管理通过ldap控制，不能通过harbor 新增，删除，管理用户组权限

Harbor 功能

项目管理 Harbor中的一个项目包含了一个应用程序所需要的所有repositories。在工程创建之前，并不允许推送镜像到Harbor中。Harbor对于project采用基于角色的访问控制。在Harbor中projects有两种类型： Public: 所有的用户都具有读取public project的权限， 其主要是为了方便你分享一些repositoriesPrivate: 一个私有project只能被特定用户以适当的权限进行访问

如图：一个项目下，包含这个对项目的镜像仓库管理（项目下的镜像），成员管理（可访问项目的成员），配置管理（白名单，可访问程度等），标签管理（分类表示），机器人账户（通过机器人账户自动pull ， push 镜像）管理等功能

系统管理

基础功能都集中在上图，左侧菜单栏提供了系统管理的功能，包括用户管理，组管理, 仓库管理，复制管理，标签，项目定额，审查服务，垃圾清理等功能

用户管理 启用ldap后，初次登录后的用户，就会成为系统可展示的用户 组管理 接入ldap后，组管理由ldap的域控决定，由于公司暂无现成的域控分类，所以此处不添加，如果使用数据库认证，这里可自行管理组，方便项目中统一成员管理 仓库管理 ldap 2.0 不仅支持 多个harbor系统之间的镜像同步，也支持向第三方docker仓库同步镜像，这个仓库可供复制管理选择 复制管理 复制管理允许通过一定的策略，向其他仓库同步镜像，比如镜像集群之间的数据同步 标签 这个标签是全局标签 项目定额 项目定额可以设置项目可存储镜像的大小，默认不限制 审查服务 为了镜像安全，审查服务提供了扫描器，扫描镜像是否有漏洞，2.0 以Trivy作为默认扫描器，也可以配置其他扫描器，如clair等，harbor提供prepare脚本支持一键修改 ./prepare --help prepare base dir is set to /home/admin/harbor/harbor Usage: main.py prepare [OPTIONS] Options: --conf TEXT the path of Harbor configuration file --with-notary the Harbor instance is to be deployed with notary --with-clair the Harbor instance is to be deployed with clair --with-trivy the Harbor instance is to be deployed with Trivy --with-chartmuseum the Harbor instance is to be deployed with chart repository supporting --help Show this message and exit. Clean up the input dir

垃圾清理 垃圾清理用来删除指定规则下的镜像

存储

Harbor支持远程挂载存储，以解决本地存储不足的情况，支持NFS，OSS，分布式存储等

docker client 配置

Harbor搭建好之后，就可以通过docker client上传镜像了，docker安装同上

配置镜像加速 配置阿里云镜像加速器 vim /etc/docker/daemon.json { "data-root": "/data/docker", # docker数据目录，可不修改 "insecure-registries": ["0.0.0.0/0"], # 非安全地址 你的harbor地址 或者0.0.0.0/0 （由于未使用https认证） "registry-mirrors": ["https://xxxx.mirror.aliyuncs.com"] # 阿里云加速地址 自己找 } 重启docker service docker restart

systemd 系统使用如下命令

systemctl restart docker

上传镜像

login docker login -u username http://xx.xx.xx.xx:port

登录私有镜像仓库，如果没有配置上面的非安全地址，会出现401错误

tag 我们从dockerhub pull一个node镜像到本地，然后对这个镜像打tag，可直接复制此处命令做修改 查看拉下来的官网镜像 docker images REPOSITORY TAG IMAGE ID CREATED SIZE node latest 37ad18cd8bd1 2 days ago 943MB

对其打 tag，私有需要在tag中加入仓库地址，如下

docker tag node:latest xx.xx.xx.xx/library/zm-node

查看镜像

docker images REPOSITORY TAG IMAGE ID CREATED SIZE xx.xx.xx.xx/library/zm-node latest 37ad18cd8bd1 2 days ago 943MB node latest 37ad18cd8bd1 2 days ago 943MB 推送 docker push xx.xx.xx.xx/library/zm-node 查看镜像 至此私有仓库的搭建和测试完成

问题归纳

过程中在ldap 和 docker 认证问题上耗时较长，最诡异的问题是 login成功了，push有问题 ，提示 unauthorized to access repository

参考资料

harbor官方文档 harbor使用手册 k8s harbor 内网协议学习LDAP篇之组和OU介绍