This is an old revision of the document!

Planned/Scheduled Maintenance Procedures

Historically, infrastructure maintenance has been on an as-needed “if it ain't broke, don't fix it” basis. Now that most Infrastructure services are highly available and the lab network/services/suite have stabilized, we're able to afford occasional downtime to perform system updates, hardware swaps, OS upgrades, etc.

This document (like all good documentation) will be ever-evolving.

Frequency/Timing

The PnT Labs team has scheduled monthly maintenance on the second Friday of every month. While this maintenance only has the potential to disrupt operations in the downstream/Octo lab, it could be a good candidate. Maintenance on Fridays allows for troubleshooting and rollbacks, if necessary, either late Friday evening or on Saturdays during a time when the lab is usually only running scheduled test suites.

Maintenance should not disrupt testing cycles when an upstream release is imminent.

Maintenance should be announced on the http://status.sepia.ceph.com and sepia-users mailing list at least a week prior and all stakeholders, yet TBD, should give a green light before proceeding.

Maintenance Matrix

Service	Disrupts Community	Disrupts Developers	Backups	Ansible	HA	Failover	Risk	Stakeholders	Applications	SME	Other Notes
www.ceph.com	Yes	No	Yes	Some	OVH	No	Medium	Wordpress	lvaz, Age of Peers
download.ceph.com	Yes	Yes	Yes	Most	OVH	No	Medium	All	Nginx	dgalloway
tracker.ceph.com	Yes	Yes	Yes	No	OVH	No	High	All	Redmine	dgalloway,dmick	Redmine and its plugins are tricky. None of us are ruby experts.
docs.ceph.com	Yes	No	Yes	Some	OVH	No	Low	Nginx	dgalloway
chacra.ceph.com	No	Yes	Yes	Yes	OVH	No	Low	Release team, Core devs?	chacra, celery, postgres, nginx	dgalloway,alfredo
chacra dev instances	No	Yes	No	Yes	OVH	Yes	Low	Devs	chacra, celery, postgres, nginx	dgalloway,alfredo
shaman	No	Yes	No	Yes	OVH	Yes	Medium	Devs	shaman, ?	dgalloway,alfredo
{apt-mirror,gitbuilder}.ceph.com	No	Yes	No	No	No	No	High	Devs	Apache	dgalloway,dmick	Still on single baremetal mira
jenkins{2}.ceph.com	Yes	Yes	Some	Yes	OVH	No	Medium	Devs	Jenkins, mita, celery, nginx	dgalloway,alfredo
prado.ceph.com	Could	Could	No	Yes	OVH	No	Low	Devs	prado, nginx	dgalloway,alfredo
git.ceph.com	Yes?	Yes	No	Yes	RHEV	No	Medium	Devs, others?	git, git-daemon, apache	dgalloway,dmick
teuthology VM	No	Yes	Some	Most	RHEV	No	Low	Devs	Teuthology	dgalloway,zack	Relies on Paddles, git.ceph.com, apt-mirror, download.ceph.com, chacra, gitbuilder.ceph.com
pulpito.front	No	Not really	No	Yes	No	No	Medium	QE?	pulpito	zack	Relies on paddles. Still on baremetal mira
paddles.front	No	Yes	Yes	Yes	No	No	Medium	Devs	paddles	dgalloway,zack	Still on baremetal mira
Cobbler	No	No	No	Yes	RHEV	No	Low	dgalloway	Cobbler, apache	dgalloway	Really only needed for creating FOG images
conserver.front	No	Yes?	Some	No	RHEV	No	Low	Devs	conserver	dgalloway
DHCP (store01)	No	Yes	Yes	No	No	No	Medium	Devs	dhcpd	dgalloway
DNS	Could	Yes	N/A	Yes	RHEV/OVH	Not currently	Low Devs	named	dgalloway
FOG	No	Yes	No	Little	RHEV	No	Medium	Devs	fog	dgalloway
LRC	No	Could	No	No	Yes	Ish	Medium	Devs	ceph	dgalloway,sage
gw.sepia.ceph.com	Could	Yes	Yes	Yes	RHEV	No	Medium	All	openvpn, nginx	dgalloway
RHEV	No	Could	Yes	No	Yes	Ish	Medium	All	RHEV, gluster	dgalloway	Packages are a mix between CentOS gluster and RHGS
Gluster	No	Could	No	No	Yes	Ish	Medium	All	Gluster	dgalloway	RHGS compatibility must remain aligned with RHEV version

Sepia Lab Wiki

Sidebar

Table of Contents

Planned/Scheduled Maintenance Procedures

Frequency/Timing

Maintenance Matrix

Sepia Lab Wiki

User Tools

Site Tools

Sidebar

Table of Contents

Planned/Scheduled Maintenance Procedures

Frequency/Timing

Maintenance Matrix

Page Tools