Differences

This shows you the differences between two versions of the page.

--- services:quay.ceph.io [2024/03/04 21:29] – zmc
+++ services:quay.ceph.io [2026/10/02 18:58] (current) – [Superuser login] djgalloway
@@ Line 1: / Line 1: @@
 ====== quay.ceph.io ======
-===== Pulling from quay.ceph.io =====
+===== Summary =====
-In Quay's web UI, in the "Pull this container with the following Podman command:" fields, you'll see commands like this:
+quay.ceph.io migrated from AWS to the lab on 2026-09-09.  It's Quay (operator-managed) in the ''quay'' namespace in OpenShift.  Blobs are in an ODF RGW bucket, postgres on a pre-created RBD PVC.  See https://github.com/ceph/sepia-openshift/tree/main/quay - ''MIGRATION.md'' there is the full history.
-''podman pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph''
+Public path: NS1 A record -> 192.86.31.16 -> edge proxy -> the OpenShift route.  There is deliberately **no Anubis** in front of it - registry clients can't answer challenges.
-These will work just fine when connected to the Sepia VPN. A public endpoint is available at ''quay.ceph.io'', so you may pull images without the VPN like so:
+''quay-int'' is a separate namespace and must **never** be public.  It also hosts the proxy-cache orgs (''mirror'' -> quay.io, ''mirror-docker'' -> docker.io) that CI pulls through.
-''podman pull quay.ceph.io/ceph-ci/ceph''
+===== Admin Tasks =====
+==== Superuser login ====
+''quayadmin''.  Creds are in 1Password.
-===== Operations Summary =====
+==== Renewing the cert ====
-This is a VM in [[services:RHEV]] that was created because quay.io was having lots of issues the week of 25MAY2020 which broke the lab (https://tracker.ceph.com/issues/45343).
+The in-pod cert is manual acme.sh - see MIGRATION.md.  It does **not** auto-renew.
-SSH only listens on the front interface at quay.front.sepia.ceph.com.
+===== Troubleshooting =====
+==== Pulls fail with 401/unauthorized on podman hosts ====
-Due to the potentially large number of container images we'd be storing, I opted to use the [[services:longrunningcluster]] which is mounted at ''/lrc''.
+cephadm's stored registry login must use the **bare hostname**.  docker normalizes any URL to the hostname; podman matches path components verbatim, so a login stored as ''quay.ceph.io/ceph/prerelease'' doesn't cover ''ceph/prerelease-amd64'' and the pull goes out anonymous.
-I just chose quay because it was the same software/tool that was already in use.  Figured it'd be easy to just ''s/quay.io/quay.ceph.io'' wherever needed.
-Passwords are in ''magna001.ceph.redhat.com:/root/secrets''.  I reused the same username (Dan's), password, and repo name so using the new registry would be plug-n-play with our CI.
-I had some trouble getting the containers to communicate with one another. The Quay docs don't cover setting up the ''br_netfilter'' kernel module or firewall rules so I wrote https://access.redhat.com/solutions/5254621.
-===== Setup Commands =====
-From dmick, 3Nov22: apparently this has changed a bit; it looks like the client.container auth doesn't exist anymore; rather, client.admin is used. Also, the cluster path is /containers/quay
 <code>
-## On reesi001
+# diagnose without touching secrets
-ceph auth add client.containers mds 'allow rw path=/containers' mon 'allow r' osd 'allow rw pool=data'
+sudo podman login --get-login --authfile /etc/ceph/podman-auth.json quay.ceph.io
-ceph auth get client.containers
+# fix: re-run registry-login with url: quay.ceph.io (hostname only)
-# Copy the key output
+ceph cephadm registry-login -i login.json
-## On quay.front.sepia.ceph.com
-# run the ansible_managed and common roles
-yum localinstall http://download.ceph.com/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm
-yum install ceph-common
-mkdir /lrc
-echo "172.21.2.201,172.21.2.202,172.21.2.203:/containers/mirror    /lrc/           ceph    name=containers,secretfile=/etc/ceph/secret,_netdev 0 2" >> /etc/fstab
-echo "KEY_FROM_REESI001" > /etc/ceph/secret
-mount -a
-# Then I just followed https://access.redhat.com/documentation/en-us/red_hat_quay/3.3/html/deploy_red_hat_quay_-_basic/preparing_for_red_hat_quay_basic
 </code>
+On the Quay side, app pod nginx logs show ''GET /v2/auth?account=<user>'' - no ''account='' means the client sent no credentials at all.
+==== External pulls hang/fail on blob GETs ====
+Quay presign-redirects blob GETs to RGW.  Two requirements, both already set but worth knowing:
+  * ''FEATURE_PROXY_STORAGE: true'' - otherwise clients get redirected to the cluster-internal RGW service, dead from outside
+  * The ''DISTRIBUTED_STORAGE_CONFIG'' hostname must be the full ''.svc.cluster.local'' FQDN - the storage-proxy nginx resolver ignores search domains and 404s on short ''.svc'' names
+==== Anonymous CI pulls of mirror/* repos 401 ====
+Proxy-cache repos are born **private**, and only authenticated pulls create cache entries.  The daily reconcile CronJob (''quay-int/proxy-cache.yaml'') flips them public - check it ran, or flip the repo public by hand.
-===== Letsencrypt =====
+==== Config bundle changes get reverted ====
-Since the quay container listens on port 80 and 443, we have to temporarily stop it to renew the cert.  To avoid doing this too frequently, I have it done on the first Saturday of even-numbered months early in the morning when traffic should be minimal.
+The operator force-overrides some keys when the component is managed (e.g. ''REPO_MIRROR_TLS_VERIFY: true'').  If a bundle edit doesn't stick, that's why.
+==== Who changed something ====
+Console Logs view, **audit tenant**:
 <code>
-[root@quay ~]# crontab -l
+{log_type="audit"} |= "quay" | json | objectRef_namespace="quay"
-# On the first Saturday of Feb,Apr,Jun,Aug,Oct,Dec, renew quay cert
+</code>
-4 * 2,4,6,8,10,12 6 [ $(date +\%d) -le 06 ] && /root/bin/quay-cert-renew.sh
+See [[services:loki]].
-[root@quay ~]# cat /root/bin/quay-cert-renew.sh
-#!/bin/bash
-for container in $(docker ps | grep "quay\.io" | awk '{ print $1 }'); do docker stop $container; done
-certbot renew
-docker run --restart=always -p 443:8443 -p 80:8080 --sysctl net.core.somaxconn=4096 --privileged=true -v /etc/quay:/conf/stack:Z -v /lrc:/datastorage/registry:Z -d quay.io/redhat/quay:v3.3.0
-</code>