Table of Contents

Networking

Networks

Name Purpose Size/CIDR VLAN ID Domain MTU
Sepia Front Copy Created to make lab migration easy 172.21.0.0/20 100 front.sepia.ceph.com 1500
Sepia IPMI Copy Created to make lab migration easy 172.21.32.0/20 102 ipmi.sepia.ceph.com 1500
New Front Server uplinks 10.20.192.0/20 1338 front.sepia.ceph.com 1500
New IPMI New server BMC 10.20.208.0/20 1339 ipmi.sepia.ceph.com 1500
VPN Clients VPN Clients 172.16.48.1/24 1340 vpn.sepia.ceph.com 1500
LRC Ceph Cluster Backend Storage Traffic 172.16.50.1/23 1341 lrc.sepia.ceph.com 9000
Openshift Cluster Openshift Node IPs (master, workers, VIPs) 172.16.53.1/25 1342 os.sepia.ceph.com 1500
Openshift Provision Openshift Provisioning/PXE 172.16.55.1/26 1343 pxe.os.sepia.ceph.com 1500
Openshift Clients Openshift VM Clients 172.16.56.1/23 1344 client.os.sepia.ceph.com 1500
ODF Public ODF Client-facing (Ceph Public) 172.16.59.1/25 1345 pub.odf.sepia.ceph.com 1500
ODF Storage ODF Cluster Traffic 172.16.60.1/25 1346 back.odf.sepia.ceph.com 9000
Public Internet 192.86.31.0/24 200
vlan104 Officinalis 172.21.64.0/24 104 vlan104.sepia.ceph.com
vlan105 Officinalis 172.21.65.0/24 105 vlan105.sepia.ceph.com
vlan106 Officinalis 172.21.66.0/24 106 vlan106.sepia.ceph.com
vlan107 Officinalis 172.21.67.0/24 107 vlan107.sepia.ceph.com

The first IP of each is the gateway and the first 5 IPs should not be used (.0 through .4)

Officinalis

In January 2020, 4 new small subnets were created for the officinalis nodes to each have an IP on different subnets. NIC1 is on VLAN104, NIC2 on VLAN105, etc. At present, the only switch configured to use these subnets is the QFX5200 purchased specifically for the Officinalis nodes but other switches could be configured if desired.

DNS and DHCP were configured to manage these VLANs in https://github.com/ceph/ceph-sepia-secrets/pull/463.

Hardware

All the switches in the Sepia lab are either Juniper EX4300 1Gb or QFX5100 10Gb (SFP+) except:

The Sepia Networking core is === OUT === of 40G ports. No more switches can be added without purchasing additional equipment.

Typical Switch config

In racks that only have 1Gb networking (e.g., Mira), the top-of-rack switch is most likely split

The racks full of Smithi have a 10Gb and 1Gb switch. The 10Gb switch is assigned to vlan100 (front) and the 1Gb switch to vlan102 (ipmi).

The infra rack (houses RHEV, teuthology, gitbuilder.ceph.com), AA-06, has a 10Gb switch with no standard port assignments (IOW, it's kinda random). I believe any port that's not already in use isn't configured.

Switch Port Config Changes

Red Hat IT manages all switches in the Sepia lab and switch port configuration changes should be submitted to servicedesk@redhat.com.

Providing the serial number of the switch helps IT determine which switch needs to be configured. Switch serial numbers can be found in PnT DevOps inventory tool, OpenDCIM. (Read the pop-up message for login credentials)

Example 

Subject: Community Cage switch port config
Body:
Hi,

Please configure the top-of-rack switch (Serial number XXXXXX) in rack AA-02 as follows:
Ports 0 - 23 assigned to vlan100 / 172.21.0.0/20
Ports 24 - 47 assigned to vlan102 / 172.21.31.254/20

Thanks

Public IPs

The Community Cage has a block of public IPs dedicated to it. Ceph's usable IPs are 8.43.84.129 - 8.43.84.186.

IP Use
8.43.84.129 OpenVPN gateway and reverse proxies
8.43.84.130 gitbuilder.ceph.com
8.43.84.131 Jump host for Chinese devs
8.43.84.132 git.ceph.com
8.43.84.133 Old OpenVPN server (store01)
8.43.84.134 drop.ceph.com
8.43.84.135 RDU download.ceph.com mirror (rgw s3 only)
8.43.84.136 lists.ceph.io
8.43.84.137 telemetry.ceph.com
8.43.84.138 Telemetry public-facing read-only VM
8.43.84.139 chacra.ceph.com
8.43.84.140 beta.ceph.io
8.43.84.141 quay.ceph.io

IPv6

IPv6 was enabled on 12/11/2018.

Our subnet is 2620:52:3:3:0:0:0:0/64
Our default gw is 2620:52:3:3:ffff:ffff:ffff:fffe/64