====== quay.ceph.io ======
===== Summary =====
quay.ceph.io migrated from AWS to the lab on 2026-09-09.  It's Quay (operator-managed) in the ''quay'' namespace in OpenShift.  Blobs are in an ODF RGW bucket, postgres on a pre-created RBD PVC.  See https://github.com/ceph/sepia-openshift/tree/main/quay - ''MIGRATION.md'' there is the full history.

Public path: NS1 A record -> 192.86.31.16 -> edge proxy -> the OpenShift route.  There is deliberately **no Anubis** in front of it - registry clients can't answer challenges.

''quay-int'' is a separate namespace and must **never** be public.  It also hosts the proxy-cache orgs (''mirror'' -> quay.io, ''mirror-docker'' -> docker.io) that CI pulls through.

===== Admin Tasks =====
==== Superuser login ====
''quayadmin''.  Creds are in 1Password.

==== Renewing the cert ====
The in-pod cert is manual acme.sh - see MIGRATION.md.  It does **not** auto-renew.

===== Troubleshooting =====
==== Pulls fail with 401/unauthorized on podman hosts ====
cephadm's stored registry login must use the **bare hostname**.  docker normalizes any URL to the hostname; podman matches path components verbatim, so a login stored as ''quay.ceph.io/ceph/prerelease'' doesn't cover ''ceph/prerelease-amd64'' and the pull goes out anonymous.
<code>
# diagnose without touching secrets
sudo podman login --get-login --authfile /etc/ceph/podman-auth.json quay.ceph.io
# fix: re-run registry-login with url: quay.ceph.io (hostname only)
ceph cephadm registry-login -i login.json
</code>
On the Quay side, app pod nginx logs show ''GET /v2/auth?account=<user>'' - no ''account='' means the client sent no credentials at all.

==== External pulls hang/fail on blob GETs ====
Quay presign-redirects blob GETs to RGW.  Two requirements, both already set but worth knowing:
  * ''FEATURE_PROXY_STORAGE: true'' - otherwise clients get redirected to the cluster-internal RGW service, dead from outside
  * The ''DISTRIBUTED_STORAGE_CONFIG'' hostname must be the full ''.svc.cluster.local'' FQDN - the storage-proxy nginx resolver ignores search domains and 404s on short ''.svc'' names

==== Anonymous CI pulls of mirror/* repos 401 ====
Proxy-cache repos are born **private**, and only authenticated pulls create cache entries.  The daily reconcile CronJob (''quay-int/proxy-cache.yaml'') flips them public - check it ran, or flip the repo public by hand.

==== Config bundle changes get reverted ====
The operator force-overrides some keys when the component is managed (e.g. ''REPO_MIRROR_TLS_VERIFY: true'').  If a bundle edit doesn't stick, that's why.

==== Who changed something ====
Console Logs view, **audit tenant**:
<code>
{log_type="audit"} |= "quay" | json | objectRef_namespace="quay"
</code>
See [[services:loki]].