Differences

This shows you the differences between two versions of the page.

--- services:pulpito [2020/08/04 17:13] – [Updating Pulpito] djgalloway
+++ services:pulpito [2026/10/02 18:55] (current) – djgalloway
@@ Line 1: / Line 1: @@
 ====== pulpito ======
 ===== Summary =====
-pulpito is a web interface/dashboard that displays teuthology job results.  See https://github.com/ceph/pulpito
+pulpito (and pulpito-ng) are web dashboards for teuthology results.  See https://github.com/ceph/pulpito and https://github.com/ceph/pulpito-ng
-The service runs on a baremetal host, [[hardware:infrastructure#pulpitofrontsepiacephcom|pulpito.front.sepia.ceph.com]] from ''/home/ubuntu''.
+Both run in OpenShift.  See https://github.com/ceph/sepia-openshift
-It's also viewable publicly at pulpito.ceph.com via an nginx reverse proxy on gw.sepia.ceph.com.
+Publicly they're served at pulpito.ceph.com and pulpito-ng.ceph.com through nginx on soko01, with [[https://github.com/TecharoHQ/anubis|Anubis]] challenging browsers in front of each vhost.  The nginx + Anubis stack is managed by the ''public_facing'' role in ceph-cm-ansible.
+''%%/_paddles/(branches|suites|machine_types)/%%'' on pulpito.ceph.com deliberately **bypasses** Anubis and is served from a 300s nginx cache instead.  Don't "fix" that - the cache is the shock absorber; putting it behind Anubis took pulpito down within the hour when it was tried.
 ===== Admin Tasks =====
-==== Starting/Restarting service ====
+==== Restarting ====
-''sudo supervisorctl start|stop|restart pulpito''
+<code>
+oc rollout restart deploy/pulpito
+oc rollout restart deploy/pulpito-ng
+</code>
-==== Updating Pulpito ====
+===== Troubleshooting =====
+==== Check the logs ====
+nginx logs live on soko01; app logs in OpenShift.
 <code>
-ssh ubuntu@pulpito.front.sepia.ceph.com
+# on soko01
-cd /home/ubuntu/pulpito
+tail -f /var/log/nginx/pulpito-ng-access.log
-git pull
+# or in Grafana (see services:loki)
-sudo supervisorctl restart pulpito
+{host="soko01.front.sepia.ceph.com"}
-sudo systemctl restart nginx
 </code>
+==== Site slow/down, paddles pods crashlooping ====
+Scraper botnet.  They crawl pulpito-ng's ''/runs/<run>/jobs/<id>/history'' pages, which fire the killer ''/jobs/?description='' paddles query server-side.  Check request rate in the nginx logs.  The botnets rotate residential IPs and **can solve the Anubis challenge**, so the levers are:
+  * Restart the Anubis instance - signing keys are random per restart, so this invalidates every outstanding challenge cookie at once <code>
+systemctl restart anubis@pulpito-ng
+</code>
+  * Raise the challenge difficulty in ''/etc/anubis/botPolicies.yaml'' (shared by all instances)
+==== Anubis returns "Oh noes!" (500) ====
+nginx isn't passing ''X-Real-Ip''.  Check the vhost config.
+==== Lab/monitoring traffic getting challenged ====
+The ''lab-and-monitoring'' allowlist in ''botPolicies.yaml'' covers 192.86.31.0/24, 172.16.0.0/12, and 10.20.192.0/20.  Non-browser UAs (git, curl, apt) are never challenged.
+==== Editing nginx configs on soko01 ====
+''sites-enabled'' includes **everything** in the directory, not just ''*.conf''.  Never leave a backup file in there - nginx -t fails on duplicate upstreams.  Backups go in ''/root/''.  Hand-apply the change, then make the ''public_facing'' template match.