User Tools

Site Tools


Sidebar

General Lab Info (Mainly for Devs)

Hardware

Lab Infrastructure Services

Misc Admin Tasks
These are infrequently completed tasks that don't fit under any specific service

Production Services

OVH = OVH
RHEV = Sepia RHE instance
Baremetal = Host in Sepia lab

The Attic/Legacy Info

services:quay.ceph.io

quay.ceph.io

Summary

This is a VM in RHEV that was created because quay.io was having lots of issues the week of 25MAY2020 which broke the lab (https://tracker.ceph.com/issues/45343).

SSH only listens on the front interface at quay.front.sepia.ceph.com.

Due to the potentially large number of container images we'd be storing, I opted to use the longrunningcluster which is mounted at /lrc.

I just chose quay because it was the same software/tool that was already in use. Figured it'd be easy to just s/quay.io/quay.ceph.io wherever needed.

Passwords are in magna001.ceph.redhat.com:/root/secrets. I reused the same username (Dan's), password, and repo name so using the new registry would be plug-n-play with our CI.

I had some trouble getting the containers to communicate with one another. The Quay docs don't cover setting up the br_netfilter kernel module or firewall rules so I wrote https://access.redhat.com/solutions/5254621.

Setup Commands

## On reesi001
ceph auth add client.containers mds 'allow rw path=/containers' mon 'allow r' osd 'allow rw pool=data'
ceph auth get client.containers
# Copy the key output

## On quay.front.sepia.ceph.com
# run the ansible_managed and common roles
yum localinstall http://download.ceph.com/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm
yum install ceph-common
mkdir /lrc
echo "172.21.2.201,172.21.2.202,172.21.2.203:/containers/mirror    /lrc/           ceph    name=containers,secretfile=/etc/ceph/secret,_netdev 0 2" >> /etc/fstab
echo "KEY_FROM_REESI001" > /etc/ceph/secret 
mount -a

# Then I just followed https://access.redhat.com/documentation/en-us/red_hat_quay/3.3/html/deploy_red_hat_quay_-_basic/preparing_for_red_hat_quay_basic

Letsencrypt

Since the quay container listens on port 80 and 443, we have to temporarily stop it to renew the cert. To avoid doing this too frequently, I have it done on the first Saturday of even-numbered months early in the morning when traffic should be minimal.

[root@quay ~]# crontab -l
# On the first Saturday of Feb,Apr,Jun,Aug,Oct,Dec, renew quay cert
0 4 * 2,4,6,8,10,12 6 [ $(date +\%d) -le 06 ] && /root/bin/quay-cert-renew.sh

[root@quay ~]# cat /root/bin/quay-cert-renew.sh 
#!/bin/bash
for container in $(docker ps | grep "quay\.io" | awk '{ print $1 }'); do docker stop $container; done
certbot renew
docker run --restart=always -p 443:8443 -p 80:8080 --sysctl net.core.somaxconn=4096 --privileged=true -v /etc/quay:/conf/stack:Z -v /lrc:/datastorage/registry:Z -d quay.io/redhat/quay:v3.3.0
services/quay.ceph.io.txt · Last modified: 2020/08/06 19:39 by djgalloway