We have two RHCS clusters running on mira that serve as mirrors for download.ceph.com.
The goal of this project is to use/dogfood downstream RHCS software, docs, and ceph-ansible to uncover difficulties in installing and maintaining the product.
Setup notes: https://docs.google.com/document/d/197cRnSA8QF1vYcAejvIT6tCG9Vi1nhUWrX987HTg7P4/edit
There are two clusters. One uses RGW and one uses RBD as the client. Both have 3 MONs running on VMs in RHEV. Their public gateways are also VMs in RHEV.
There are affinity rules in place to ensure the RGW and RBD public gateways all run on separate baremetal hyperisors.
There are also affinity rules in place in RHEV to ensure the MONs all run on separate baremetal hyperisors.
The RBD cluster's OSDs run on mira046, mira052, and mira084.
The MONs run on dl-rbd-mon{01..03}.front.sepia.ceph.com.
Since the RBD cluster is just serving a single block device, only one public VM (dl-rbd-pub) is required.
The RGW cluster's OSDs run on mira102, mira103, and mira106.
According to Red Hat downstream docs, at least two gateway hosts (dl-rgw-pub{01,02}) and two HAProxy servers (dl-rgw-ha{01,02}) are required to serve a static website.
The clusters were set up to have all Ceph traffic go over the front VLAN.
The RBD cluster's public VM has a NIC on front and a NIC on the WAN VLAN.
The RGW cluster's HAProxy VMs have a NIC on front and a NIC on the WAN VLAN. The RGW clients themselves only have a NIC on the front VLAN.