pulpito (and pulpito-ng) are web dashboards for teuthology results. See https://github.com/ceph/pulpito and https://github.com/ceph/pulpito-ng

Both run in OpenShift. See https://github.com/ceph/sepia-openshift

Publicly they're served at pulpito.ceph.com and pulpito-ng.ceph.com through nginx on soko01, with Anubis challenging browsers in front of each vhost. The nginx + Anubis stack is managed by the public_facing role in ceph-cm-ansible.

/_paddles/(branches|suites|machine_types)/ on pulpito.ceph.com deliberately bypasses Anubis and is served from a 300s nginx cache instead. Don't “fix” that - the cache is the shock absorber; putting it behind Anubis took pulpito down within the hour when it was tried.

oc rollout restart deploy/pulpito
oc rollout restart deploy/pulpito-ng

nginx logs live on soko01; app logs in OpenShift.

# on soko01
tail -f /var/log/nginx/pulpito-ng-access.log
# or in Grafana (see services:loki)
{host="soko01.front.sepia.ceph.com"}

Scraper botnet. They crawl pulpito-ng's /runs/<run>/jobs/<id>/history pages, which fire the killer /jobs/?description= paddles query server-side. Check request rate in the nginx logs. The botnets rotate residential IPs and can solve the Anubis challenge, so the levers are:

• Restart the Anubis instance - signing keys are random per restart, so this invalidates every outstanding challenge cookie at once

  systemctl restart anubis@pulpito-ng

• Raise the challenge difficulty in /etc/anubis/botPolicies.yaml (shared by all instances)

nginx isn't passing X-Real-Ip. Check the vhost config.

The lab-and-monitoring allowlist in botPolicies.yaml covers 192.86.31.0/24, 172.16.0.0/12, and 10.20.192.0/20. Non-browser UAs (git, curl, apt) are never challenged.

sites-enabled includes everything in the directory, not just *.conf. Never leave a backup file in there - nginx -t fails on duplicate upstreams. Backups go in /root/. Hand-apply the change, then make the public_facing template match.