paddles is a very simple JSON-based API used to report teuthology test results. See https://github.com/ceph/paddles
The service runs on a baremetal host, pulpito.front.sepia.ceph.com, deployed via the ceph-cm-ansible role: https://github.com/ceph/ceph-cm-ansible/tree/main/roles/paddles
It is operated by docker service: https://docs.docker.com/engine/reference/commandline/service/, utilizing replication, health checks, and automatic rollbacks.
The 'paddles' db is backed up daily by the backup.sh script on gitbuilder-archive.
Backups are located in gitbuilder-archive:/home/backup/paddles.front.sepia.ceph.com-psql/paddles
ssh paddles.front.sepia.ceph.com # Stopping sudo docker service scale paddles=0 # Starting sudo docker service scale paddles=10
For jobs that indicate they're running but aren't, expire_jobs can be used.
The following example would expire any queued jobs 14 days old or older and any running jobs that haven't been updated in 30 minutes.
ssh paddles.front.sepia.ceph.com
sudo docker exec -it $(sudo docker ps --filter "health=healthy" --filter "name=paddles" --format "{{.ID}}" | head -n1) sh -c "pecan expire_jobs config.py -q 14 -r 30"
From your workstation,
cd ~/src/teuthology source ./virtualenv/bin/activate # Edit docs/_static/create_nodes.py # (paddles_url, machine_type, lab_domain, and machine_index_range) # These can all be found in teuthology.yaml on a teuthology host python docs/_static/create_nodes.py
ssh pulpito.front.sepia.ceph.com sudo docker service update --image quay.io/ceph-infra/paddles:main paddles --force