Sepia Lab Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== pulpito ======
===== Summary =====
pulpito (and pulpito-ng) are web dashboards for teuthology results.  See https://github.com/ceph/pulpito and https://github.com/ceph/pulpito-ng

Both run in OpenShift.  See https://github.com/ceph/sepia-openshift

Publicly they're served at pulpito.ceph.com and pulpito-ng.ceph.com through nginx on soko01, with [[https://github.com/TecharoHQ/anubis|Anubis]] challenging browsers in front of each vhost.  The nginx + Anubis stack is managed by the ''public_facing'' role in ceph-cm-ansible.

''%%/_paddles/(branches|suites|machine_types)/%%'' on pulpito.ceph.com deliberately **bypasses** Anubis and is served from a 300s nginx cache instead.  Don't "fix" that - the cache is the shock absorber; putting it behind Anubis took pulpito down within the hour when it was tried.

===== Admin Tasks =====
==== Restarting ====
<code>
oc rollout restart deploy/pulpito
oc rollout restart deploy/pulpito-ng
</code>

===== Troubleshooting =====
==== Check the logs ====
nginx logs live on soko01; app logs in OpenShift.
<code>
# on soko01
tail -f /var/log/nginx/pulpito-ng-access.log
# or in Grafana (see services:loki)
{host="soko01.front.sepia.ceph.com"}
</code>

==== Site slow/down, paddles pods crashlooping ====
Scraper botnet.  They crawl pulpito-ng's ''/runs/<run>/jobs/<id>/history'' pages, which fire the killer ''/jobs/?description='' paddles query server-side.  Check request rate in the nginx logs.  The botnets rotate residential IPs and **can solve the Anubis challenge**, so the levers are:
  * Restart the Anubis instance - signing keys are random per restart, so this invalidates every outstanding challenge cookie at once <code>
systemctl restart anubis@pulpito-ng
</code>
  * Raise the challenge difficulty in ''/etc/anubis/botPolicies.yaml'' (shared by all instances)

==== Anubis returns "Oh noes!" (500) ====
nginx isn't passing ''X-Real-Ip''.  Check the vhost config.

==== Lab/monitoring traffic getting challenged ====
The ''lab-and-monitoring'' allowlist in ''botPolicies.yaml'' covers 192.86.31.0/24, 172.16.0.0/12, and 10.20.192.0/20.  Non-browser UAs (git, curl, apt) are never challenged.

==== Editing nginx configs on soko01 ====
''sites-enabled'' includes **everything** in the directory, not just ''*.conf''.  Never leave a backup file in there - nginx -t fails on duplicate upstreams.  Backups go in ''/root/''.  Hand-apply the change, then make the ''public_facing'' template match.