Table of Contents

paddles

Summary

paddles is a very simple JSON-based API used to report teuthology test results. See https://github.com/ceph/paddles

The service runs on a baremetal host, pulpito.front.sepia.ceph.com, deployed via the ceph-cm-ansible role: https://github.com/ceph/ceph-cm-ansible/tree/main/roles/paddles

It is operated by docker service: https://docs.docker.com/engine/reference/commandline/service/, utilizing replication, health checks, and automatic rollbacks.

Backups

The 'paddles' db is backed up daily by the backup.sh script on gitbuilder-archive.

Backups are located in gitbuilder-archive:/home/backup/paddles.front.sepia.ceph.com-psql/paddles

Admin Tasks

Stopping/Starting the service

ssh paddles.front.sepia.ceph.com
# Stopping
sudo docker service scale paddles=0
# Starting
sudo docker service scale paddles=10

Updating/Fixing Zombie Jobs

For jobs that indicate they're running but aren't, expire_jobs can be used.

The following example would expire any queued jobs 14 days old or older and any running jobs that haven't been updated in 30 minutes.

ssh paddles.front.sepia.ceph.com
sudo docker exec -it $(sudo docker ps --filter "health=healthy" --filter "name=paddles" --format "{{.ID}}" | head -n1) sh -c "pecan expire_jobs config.py -q 14 -r 30"

Adding testnodes to the inventory/DB

From your workstation,

cd ~/src/teuthology
source ./virtualenv/bin/activate

# Edit docs/_static/create_nodes.py
# (paddles_url, machine_type, lab_domain, and machine_index_range)
# These can all be found in teuthology.yaml on a teuthology host

python docs/_static/create_nodes.py

Upgrade Paddles

ssh pulpito.front.sepia.ceph.com
sudo docker service update --image quay.io/ceph-infra/paddles:main paddles --force