docker-mirror.front.sepia.ceph.com

This is a VM in RHEV that is a caching mirror for docker.io. We were being rate-limited by docker.io (https://tracker.ceph.com/issues/45343).

I was originally going to create an Ansible playbook to set this up but it was easy enough it wasn't worth the time.

https://docs.docker.com/registry/recipes/mirror/

## On reesi001
ceph auth add client.containers mds 'allow rw path=/containers' mon 'allow r' osd 'allow rw pool=data'
ceph auth get client.containers
# Copy the key output

## On docker-mirror.front.sepia.ceph.com
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
apt update
apt install ceph-common
mkdir /lrc
echo "172.21.2.201,172.21.2.202,172.21.2.203:/containers/mirror    /lrc/           ceph    name=containers,secretfile=/etc/ceph/secret,_netdev 0 2" >> /etc/fstab
echo "KEY_FROM_REESI001" > /etc/ceph/secret 
mount -a
apt install docker.io
docker run -it --rm --entrypoint cat registry:2 /etc/docker/registry/config.yml > /lrc/config.yml
# Used example from https://www.cloudkb.net/configure-docker-local-registry-proxy-cache/

# Then used parts of https://medium.com/@ifeanyiigili/how-to-setup-a-private-docker-registry-with-a-self-sign-certificate-43a7407a1613
mkdir /lrc/certs
openssl req  -newkey rsa:4096 -nodes -sha256 -keyout /lrc/certs/domain.key -x509 -days 3650 -addext "subjectAltName = DNS:docker-mirror.front.sepia.ceph.com" -out /lrc/certs/domain.crt

docker run -d --restart=always -p 5000:5000 --name registry-mirror -e REGISTRY_HTTP_TLS_CERTIFICATE=/var/lib/registry/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/var/lib/registry/certs/domain.key -v /lrc:/var/lib/registry registry:2 /var/lib/registry/config.yml

Super simple.

# Example using grafana
podman pull --tls-verify=false docker-mirror.front.sepia.ceph.com:5000/grafana/grafana

Or you can also now just run the container-host role which will configure podman and/or docker to use our mirror for docker.io.

The first time I created the cert, I accidentally left the date out so the cert was only good for a month. The second cert is good for 10 years.

1. root@docker-mirror:~# openssl req -newkey rsa:4096 -nodes -sha256 -keyout /lrc/certs/domain.key -x509 -days 3650 -addext "subjectAltName = DNS:docker-mirror.front.sepia.ceph.com" -out /lrc/certs/domain.crt
2. Copy the contents of /lrc/certs/domain.crt and update container_mirror_cert in https://github.com/ceph/ceph-sepia-secrets/blob/master/ansible/inventory/group_vars/all.yml
3. Run ansible-playbook container-host.yml against the appropriate hosts.
4. root@docker-mirror:~# docker stop registry-mirror; docker rm registry-mirror; docker run -d --restart=always -p 5000:5000 --name registry-mirror -e REGISTRY_HTTP_TLS_CERTIFICATE=/var/lib/registry/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/var/lib/registry/certs/domain.key -v /lrc:/var/lib/registry registry:2 /var/lib/registry/config.yml; docker logs -f registry-mirror

The mirror logs in to dockerhub using my personal API key. I just have a personal account but docker-mirror was getting rate-limited when anonymous.

root@docker-mirror:~# cat /lrc/config.yml 
version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
   storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
proxy:
  remoteurl: https://registry-1.docker.io
  username: XXXXX
  password: XXXXX