This is an old revision of the document!
Table of Contents
Planned/Scheduled Maintenance Procedures
Historically, infrastructure maintenance has been on an as-needed “if it ain't broke, don't fix it” basis. Now that most Infrastructure services are highly available and the lab network/services/suite have stabilized, we're able to afford occasional downtime to perform system updates, hardware swaps, OS upgrades, etc.
This document (like all good documentation) will be ever-evolving.
Frequency/Timing
The PnT Labs team has scheduled monthly maintenance on the second Friday of every month. While this maintenance only has the potential to disrupt operations in the downstream/Octo lab, it could be a good candidate. Maintenance on Fridays allows for troubleshooting and rollbacks, if necessary, either late Friday evening or on Saturdays during a time when the lab is usually only running scheduled test suites.
Maintenance should not disrupt testing cycles when an upstream release is imminent.
Maintenance should be announced on the Status Portal| and sepia-users mailing list at least a week prior and all stakeholders, yet TBD, should give a green light before proceeding.
Maintenance Matrix
| Service | Disrupts Community | Disrupts Developers | Backups | Ansible | HA | Failover | Risk | Stakeholders | Applications | SME | Other Notes |
|---|---|---|---|---|---|---|---|---|---|---|---|
| www.ceph.com | Yes | No | Yes | Some | OVH | No | Medium | Wordpress | lvaz, Age of Peers | ||
| download.ceph.com | Yes | Yes | Yes | Most | OVH | No | Medium | All | Nginx | dgalloway | |
| tracker.ceph.com | Yes | Yes | Yes | No | OVH | No | High | All | Redmine | dgalloway,dmick | Redmine and its plugins are tricky. None of us are ruby experts. |
| docs.ceph.com | Yes | No | Yes | Some | OVH | No | Low | Nginx | dgalloway | ||
| chacra.ceph.com | No | Yes | Yes | Yes | OVH | No | Low | Release team, Core devs? | chacra, celery, postgres, nginx | dgalloway,alfredo | |
| chacra dev instances | No | Yes | No | Yes | OVH | Yes | Low | Devs | chacra, celery, postgres, nginx | dgalloway,alfredo | |
| shaman | No | Yes | No | Yes | OVH | Yes | Medium | Devs | shaman, ? | dgalloway,alfredo | |
| {apt-mirror,gitbuilder}.ceph.com | No | Yes | No | No | No | No | High | Devs | Apache | dgalloway,dmick | Still on single baremetal mira |
| jenkins{2}.ceph.com | Yes | Yes | Some | Yes | OVH | No | Medium | Devs | Jenkins, mita, celery, nginx | dgalloway,alfredo | |
| prado.ceph.com | Could | Could | No | Yes | OVH | No | Low | Devs | prado, nginx | dgalloway,alfredo | |
| git.ceph.com | Yes? | Yes | No | Yes | RHEV | No | Medium | Devs, others? | git, git-daemon, apache | dgalloway,dmick | |
| teuthology VM | No | Yes | Some | Most | RHEV | No | Low | Devs | Teuthology | dgalloway,zack | Relies on Paddles, git.ceph.com, apt-mirror, download.ceph.com, chacra, gitbuilder.ceph.com |
| pulpito.front | No | Not really | No | Yes | No | No | Medium | QE? | pulpito | zack | Relies on paddles. Still on baremetal mira |
| paddles.front | No | Yes | Yes | Yes | No | No | Medium | Devs | paddles | dgalloway,zack | Still on baremetal mira |
| Cobbler | No | No | No | Yes | RHEV | No | Low | dgalloway | Cobbler, apache | dgalloway | Really only needed for creating FOG images |
| conserver.front | No | Yes? | Some | No | RHEV | No | Low | Devs | conserver | dgalloway | |
| DHCP (store01) | No | Yes | Yes | No | No | No | Medium | Devs | dhcpd | dgalloway | |
| DNS | Could | Yes | N/A | Yes | RHEV/OVH | Not currently | Low Devs | named | dgalloway | ||
| FOG | No | Yes | No | Little | RHEV | No | Medium | Devs | fog | dgalloway | |
| LRC | No | Could | No | No | Yes | Ish | Medium | Devs | ceph | dgalloway,sage | |
| gw.sepia.ceph.com | Could | Yes | Yes | Yes | RHEV | No | Medium | All | openvpn, nginx | dgalloway | |
| RHEV | No | Could | Yes | No | Yes | Ish | Medium | All | RHEV, gluster | dgalloway | Packages are a mix between CentOS gluster and RHGS |
| Gluster | No | Could | No | No | Yes | Ish | Medium | All | Gluster | dgalloway | RHGS compatibility must remain aligned with RHEV version |