services:paddles
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
services:paddles [2016/12/20 17:54] dgalloway [Backups] |
services:paddles [2023/08/21 22:20] (current) zmc [Updating/Fixing Zombie Jobs] |
||
|---|---|---|---|
| Line 3: | Line 3: | ||
| paddles is a very simple JSON-based API used to report teuthology test results. See https://github.com/ceph/paddles | paddles is a very simple JSON-based API used to report teuthology test results. See https://github.com/ceph/paddles | ||
| - | The service runs on a baremetal host, [[hardware:infrastructure#pulpitofrontsepiacephcom|pulpito.front.sepia.ceph.com]] from ''/home/ubuntu''. | + | The service runs on a baremetal host, [[hardware:infrastructure#pulpitofrontsepiacephcom|pulpito.front.sepia.ceph.com]], deployed via the ceph-cm-ansible role: https://github.com/ceph/ceph-cm-ansible/tree/main/roles/paddles |
| + | It is operated by ''docker service'': https://docs.docker.com/engine/reference/commandline/service/, utilizing replication, health checks, and automatic rollbacks. | ||
| ===== Backups ===== | ===== Backups ===== | ||
| The 'paddles' db is backed up daily by the [[services:backups#backupsh|backup.sh]] script on gitbuilder-archive. | The 'paddles' db is backed up daily by the [[services:backups#backupsh|backup.sh]] script on gitbuilder-archive. | ||
| Line 10: | Line 11: | ||
| Backups are located in ''gitbuilder-archive:/home/backup/paddles.front.sepia.ceph.com-psql/paddles'' | Backups are located in ''gitbuilder-archive:/home/backup/paddles.front.sepia.ceph.com-psql/paddles'' | ||
| - | ===== TO-DO: Admin Tasks ===== | + | ===== Admin Tasks ===== |
| - | ==== Starting/Restarting service ==== | + | ==== Stopping/Starting the service ==== |
| <code> | <code> | ||
| - | ssh ubuntu@paddles.front.sepia.ceph.com | + | ssh paddles.front.sepia.ceph.com |
| - | sudo supervisorctl stop|stop|restart paddles | + | # Stopping |
| + | sudo docker service scale paddles=0 | ||
| + | # Starting | ||
| + | sudo docker service scale paddles=10 | ||
| + | </code> | ||
| + | |||
| + | ==== Updating/Fixing Zombie Jobs ==== | ||
| + | For jobs that indicate they're running but aren't, ''expire_jobs'' can be used. | ||
| + | |||
| + | The following example would expire any **queued** jobs 14 days old or older and any **running** jobs that haven't been updated in 30 minutes. | ||
| + | |||
| + | <code> | ||
| + | ssh paddles.front.sepia.ceph.com | ||
| + | sudo docker exec -it $(sudo docker ps --filter "health=healthy" --filter "name=paddles" --format "{{.ID}}" | head -n1) sh -c "pecan expire_jobs config.py -q 14 -r 30" | ||
| + | </code> | ||
| + | |||
| + | ==== Adding testnodes to the inventory/DB ==== | ||
| + | From your workstation, | ||
| + | |||
| + | <code> | ||
| + | cd ~/src/teuthology | ||
| + | source ./virtualenv/bin/activate | ||
| + | |||
| + | # Edit docs/_static/create_nodes.py | ||
| + | # (paddles_url, machine_type, lab_domain, and machine_index_range) | ||
| + | # These can all be found in teuthology.yaml on a teuthology host | ||
| + | |||
| + | python docs/_static/create_nodes.py | ||
| + | </code> | ||
| + | |||
| + | ==== Upgrade Paddles ==== | ||
| + | <code> | ||
| + | ssh pulpito.front.sepia.ceph.com | ||
| + | sudo docker service update --image quay.io/ceph-infra/paddles:main paddles --force | ||
| </code> | </code> | ||
services/paddles.1482256447.txt.gz · Last modified: 2016/12/20 17:54 by dgalloway
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
