Historically, infrastructure maintenance has been on an as-needed “if it ain't broke, don't fix it” basis. Now that most Infrastructure services are highly available and the lab network/services/suite have stabilized, we're able to afford occasional downtime to perform system updates, hardware swaps, OS upgrades, etc.

This document (like all good documentation) will be ever-evolving.

The PnT Labs team has scheduled monthly maintenance on the second Friday of every month. Since this maintenance only has the potential to disrupt operations in the downstream/Octo lab, it could be a good candidate. Maintenance on Fridays allows for troubleshooting and rollbacks, if necessary, either late Friday evening or on Saturdays during a time when the lab is usually only running scheduled test suites.

Maintenance should not disrupt testing cycles when an upstream release is imminent.

Maintenance should be announced on the Status Portal and sepia-users mailing list at least a week prior and all stakeholders, yet TBD, should give a green light before proceeding.

Updating the dev chacra nodes ({1..5}.chacra.ceph.com) has little chance to affect upstream teuthology testing except while the chacra service is redeployed or a host is rebooted. Because of thise, it's relatively safe to perform CI maintenance separate from Sepia lab maintenance. To be extra safe, you could pause the Sepia queue and wait ~30min to make sure no package manager processes get run against a chacra node.

1. Log into each Jenkins instance, Manage Jenkins → Prepare for Shutdown
2. Again in Jenkins, go to Manage Jenkins → Manage Plugins
3. Select All at the bottom and click Download now and install after restart
4. Wait for all jobs to finish and make sure plugins are downloaded
5. Once all jobs are completed, ssh to each Jenkins instance
   1. Updating the jenkins package or rebooting will restart the service so:
   2. systemctl stop jenkins
   3. systemctl disable jenkins
   4. apt update
   5. apt install linux-image-generic (or equivalent to just update the kernel)
   6. Reboot the host so you're running the latest kernel
6. If no service redeploy is needed for chacra, shaman, or mita, just ssh to each of those hosts and apt update && apt upgrade && reboot
7. If a redeploy is needed, see each service's individual wiki page
   1. Once all the other CI hosts are up to date, update each Jenkins instance: apt upgrade
   2. This should restart Jenkins but if it doesn't, systemctl start jenkins
   3. systemctl enable jenkins
8. Spot check a few jobs to make sure all plugins are working properly
   1. You can check this by commenting jenkins test make check in a PR
   2. Make sure the Github hooks are working (Was a job triggered? When the job finishes, does it update the status in the PR?)
   3. Make sure postbuild scripts are running when they're supposed to (Is the FAILURE script running when the build PASSED? That's a problem)

For the most part, these hosts' packages can be updated and hosts rebooted whenever. If you're feeling friendly, you could send a heads up to ceph-devel and/or ceph-users.

Check login to tracker.ceph.com and spot check a few pages on www.ceph.com afterwards.

As long as there isn't a job running, this host can be updated and rebooted whenever.

• Does http://docs.ceph.com load?
• Is the host reattached as a slave in Jenkins?

Rebooting this host is disruptive to upstream testing and should be part of a planned pre-announced outage.