This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
services:paddles [2016/12/20 17:54] dgalloway [Backups] |
services:paddles [2023/08/21 22:20] (current) zmc [Updating/Fixing Zombie Jobs] |
||
---|---|---|---|
Line 3: | Line 3: | ||
paddles is a very simple JSON-based API used to report teuthology test results. See https://github.com/ceph/paddles | paddles is a very simple JSON-based API used to report teuthology test results. See https://github.com/ceph/paddles | ||
- | The service runs on a baremetal host, [[hardware:infrastructure#pulpitofrontsepiacephcom|pulpito.front.sepia.ceph.com]] from ''/home/ubuntu''. | + | The service runs on a baremetal host, [[hardware:infrastructure#pulpitofrontsepiacephcom|pulpito.front.sepia.ceph.com]], deployed via the ceph-cm-ansible role: https://github.com/ceph/ceph-cm-ansible/tree/main/roles/paddles |
+ | It is operated by ''docker service'': https://docs.docker.com/engine/reference/commandline/service/, utilizing replication, health checks, and automatic rollbacks. | ||
===== Backups ===== | ===== Backups ===== | ||
The 'paddles' db is backed up daily by the [[services:backups#backupsh|backup.sh]] script on gitbuilder-archive. | The 'paddles' db is backed up daily by the [[services:backups#backupsh|backup.sh]] script on gitbuilder-archive. | ||
Line 10: | Line 11: | ||
Backups are located in ''gitbuilder-archive:/home/backup/paddles.front.sepia.ceph.com-psql/paddles'' | Backups are located in ''gitbuilder-archive:/home/backup/paddles.front.sepia.ceph.com-psql/paddles'' | ||
- | ===== TO-DO: Admin Tasks ===== | + | ===== Admin Tasks ===== |
- | ==== Starting/Restarting service ==== | + | ==== Stopping/Starting the service ==== |
<code> | <code> | ||
- | ssh ubuntu@paddles.front.sepia.ceph.com | + | ssh paddles.front.sepia.ceph.com |
- | sudo supervisorctl stop|stop|restart paddles | + | # Stopping |
+ | sudo docker service scale paddles=0 | ||
+ | # Starting | ||
+ | sudo docker service scale paddles=10 | ||
+ | </code> | ||
+ | |||
+ | ==== Updating/Fixing Zombie Jobs ==== | ||
+ | For jobs that indicate they're running but aren't, ''expire_jobs'' can be used. | ||
+ | |||
+ | The following example would expire any **queued** jobs 14 days old or older and any **running** jobs that haven't been updated in 30 minutes. | ||
+ | |||
+ | <code> | ||
+ | ssh paddles.front.sepia.ceph.com | ||
+ | sudo docker exec -it $(sudo docker ps --filter "health=healthy" --filter "name=paddles" --format "{{.ID}}" | head -n1) sh -c "pecan expire_jobs config.py -q 14 -r 30" | ||
+ | </code> | ||
+ | |||
+ | ==== Adding testnodes to the inventory/DB ==== | ||
+ | From your workstation, | ||
+ | |||
+ | <code> | ||
+ | cd ~/src/teuthology | ||
+ | source ./virtualenv/bin/activate | ||
+ | |||
+ | # Edit docs/_static/create_nodes.py | ||
+ | # (paddles_url, machine_type, lab_domain, and machine_index_range) | ||
+ | # These can all be found in teuthology.yaml on a teuthology host | ||
+ | |||
+ | python docs/_static/create_nodes.py | ||
+ | </code> | ||
+ | |||
+ | ==== Upgrade Paddles ==== | ||
+ | <code> | ||
+ | ssh pulpito.front.sepia.ceph.com | ||
+ | sudo docker service update --image quay.io/ceph-infra/paddles:main paddles --force | ||
</code> | </code> |