Sepia Lab Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== quay.ceph.io ======
===== Pulling from quay.ceph.io =====
In Quay's web UI, in the "Pull this container with the following Podman command:" fields, you'll see commands like this:

''podman pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph''

These will work just fine when connected to the Sepia VPN. A public endpoint is available at ''quay.ceph.io'', so you may pull images without the VPN like so:

''podman pull quay.ceph.io/ceph-ci/ceph''

===== Operations Summary =====
This is a VM in [[services:RHEV]] that was created because quay.io was having lots of issues the week of 25MAY2020 which broke the lab (https://tracker.ceph.com/issues/45343).

SSH only listens on the front interface at quay.front.sepia.ceph.com.

Due to the potentially large number of container images we'd be storing, I opted to use the [[services:longrunningcluster]] which is mounted at ''/lrc''.

I just chose quay because it was the same software/tool that was already in use.  Figured it'd be easy to just ''s/quay.io/quay.ceph.io'' wherever needed.

Passwords are in ''magna001.ceph.redhat.com:/root/secrets''.  I reused the same username (Dan's), password, and repo name so using the new registry would be plug-n-play with our CI.

I had some trouble getting the containers to communicate with one another. The Quay docs don't cover setting up the ''br_netfilter'' kernel module or firewall rules so I wrote https://access.redhat.com/solutions/5254621. 

===== Setup Commands =====
From dmick, 3Nov22: apparently this has changed a bit; it looks like the client.container auth doesn't exist anymore; rather, client.admin is used. Also, the cluster path is /containers/quay
<code>
## On reesi001
ceph auth add client.containers mds 'allow rw path=/containers' mon 'allow r' osd 'allow rw pool=data'
ceph auth get client.containers
# Copy the key output

## On quay.front.sepia.ceph.com
# run the ansible_managed and common roles
yum localinstall http://download.ceph.com/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm
yum install ceph-common
mkdir /lrc
echo "172.21.2.201,172.21.2.202,172.21.2.203:/containers/mirror    /lrc/           ceph    name=containers,secretfile=/etc/ceph/secret,_netdev 0 2" >> /etc/fstab
echo "KEY_FROM_REESI001" > /etc/ceph/secret 
mount -a

# Then I just followed https://access.redhat.com/documentation/en-us/red_hat_quay/3.3/html/deploy_red_hat_quay_-_basic/preparing_for_red_hat_quay_basic
</code>



===== Letsencrypt =====
Since the quay container listens on port 80 and 443, we have to temporarily stop it to renew the cert.  To avoid doing this too frequently, I have it done on the first Saturday of even-numbered months early in the morning when traffic should be minimal.

<code>
[root@quay ~]# crontab -l
# On the first Saturday of Feb,Apr,Jun,Aug,Oct,Dec, renew quay cert
0 4 * 2,4,6,8,10,12 6 [ $(date +\%d) -le 06 ] && /root/bin/quay-cert-renew.sh

[root@quay ~]# cat /root/bin/quay-cert-renew.sh 
#!/bin/bash
for container in $(docker ps | grep "quay\.io" | awk '{ print $1 }'); do docker stop $container; done
certbot renew
docker run --restart=always -p 443:8443 -p 80:8080 --sysctl net.core.somaxconn=4096 --privileged=true -v /etc/quay:/conf/stack:Z -v /lrc:/datastorage/registry:Z -d quay.io/redhat/quay:v3.3.0
</code>
Sepia Lab Wiki

User Tools

Site Tools

Page Tools
