Differences

This shows you the differences between two versions of the page.

--- services:networking [2016/06/29 21:26]
dgalloway [Switch Port Config Changes]
+++ services:networking [2021/06/03 03:21] (current)
djgalloway [Hardware]
@@ Line 1: / Line 1: @@
 ====== Networking ======
-==== Networks ====
+===== Networks =====
-The Sepia internal LANs are divided up into four /20 networks described below.
+The Sepia internal LANs are divided up into four /20 networks (netmask 255.255.240.0) described below.
 |                     ^ front.sepia.ceph.com        ^ back.sepia.ceph.com          ^ ipmi.sepia.ceph.com            ^ Other                                                                                 ^
 ^ Available IP Range  | 172.21.0.1 - 172.21.15.250  | 172.21.16.1 - 172.21.31.250  | 172.21.32.1 - 172.21.47.250    | 172.21.48.1 - 172.21.63.250                                                           |
 ^ Gateway             | 172.21.15.254               | 172.21.31.254                | 172.21.47.254                  | 172.21.63.254                                                                         |
 ^ VLAN ID             | 100                         | 101                          | 102                            | 103                                                                                   |
-^ Use                 | Main Network interface      | Not used at the moment       | Used to access OOB management  | 172.21.48.0/24 is currently used for VPN client IPs.  The rest of the VLAN is unused  |
+^ Use                 | Main Network interface      | Not used at the moment       | Used to access OOB management  | 172.21.48.0/20 is currently used for VPN client IPs.  |
-==== Hardware ====
+In January 2020, 4 new small subnets were created for the [[hardware:officinalis]] nodes to each have an IP on different subnets.  NIC1 is on VLAN104, NIC2 on VLAN105, etc.  At present, the only switch configured to use these subnets is the QFX5200 purchased specifically for the Officinalis nodes but other switches could be configured if desired.
-All the switches in the Sepia lab are either Juniper EX4300 1Gb or QFX5100 10Gb (SFP+) with the exception of a [[https://opendcim.engineering.redhat.com/devices.php?DeviceID=7960|Mellanox SX1012]] private switch connected to the [[hardware:rhoda]] and [[hardware:incerta]] nodes.
-==== Switch Port Config Changes ====
+==== Officinalis ====
+[[services:DNS]] and [[services:DHCP]] were configured to manage these VLANs in https://github.com/ceph/ceph-sepia-secrets/pull/463.
+|                     ^ vlan104.sepia.ceph.com       ^ vlan105.sepia.ceph.com       ^ vlan106.sepia.ceph.com       ^ vlan107.sepia.ceph.com       ^
+^ Available IP Range  | 172.21.64.1 - 172.21.64.250  | 172.21.65.1 - 172.21.65.250  | 172.21.66.1 - 172.21.66.250  | 172.21.67.1 - 172.21.67.250  |
+^ Gateway             | 172.21.64.254                | 172.21.65.254                | 172.21.66.254                | 172.21.67.254                |
+^ VLAN ID             | 104                          | 105                          | 106                          | 107                          |
+===== Hardware =====
+All the switches in the Sepia lab are either Juniper EX4300 1Gb or QFX5100 10Gb (SFP+) except:
+  * A [[https://opendcim.engineering.redhat.com/devices.php?DeviceID=7960|Mellanox SX1012]] private switch connected to the [[hardware:rhoda]] and [[hardware:incerta]] nodes
+  * A QFX5200 in A09 for the Officinalis nodes
+  * An uplinked QFX5120 in B12 for the All Flash (name TBD) systems
+  * A NON-uplinked [[https://wiki.sepia.ceph.com/doku.php?id=hardware:robsoni#qfx-rgw|QFX5120]] in A15 for RGW Scale testing gear
+**The Sepia Networking core is === OUT === of 40G ports.  No more switches can be added without purchasing additional equipment.**
+===== Typical Switch config =====
+In racks that only have 1Gb networking (e.g., [[hardware:Mira]]), the top-of-rack switch is most likely split
+  * Ports 0-23 assigned to VLAN 100 (front)
+  * Ports 24-47 assigned to VLAN 102 (ipmi)
+The racks full of [[hardware:Smithi]] have a 10Gb and 1Gb switch.  The 10Gb switch is assigned to vlan100 (front) and the 1Gb switch to vlan102 (ipmi).
+The infra rack (houses RHEV, teuthology, gitbuilder.ceph.com), AA-06, has a 10Gb switch with no standard port assignments (IOW, it's kinda random).  I believe any port that's not already in use isn't configured.
+===== Switch Port Config Changes =====
 Red Hat IT manages all switches in the Sepia lab and switch port configuration changes should be submitted to [[servicedesk@redhat.com]].
@@ Line 30: / Line 57: @@
 Thanks
 </file>
+===== Public IPs =====
+The Community Cage has a block of public IPs dedicated to it.  Ceph's usable IPs are 8.43.84.129 - 8.43.84.186.
+^ IP           ^ Use                                         ^
+| 8.43.84.129  | OpenVPN gateway and reverse proxies         |
+| 8.43.84.130  | [[production:gitbuilder.ceph.com]]          |
+| 8.43.84.131  | Jump host for Chinese devs                  |
+| 8.43.84.132  | [[production:git.ceph.com]]                 |
+| 8.43.84.133  | Old OpenVPN server (store01)                |
+| 8.43.84.134  | [[production:drop.ceph.com]]                |
+| 8.43.84.135  | RDU download.ceph.com mirror (rgw s3 only)  |
+| 8.43.84.136  | [[production:lists.ceph.io]]                |
+| 8.43.84.137  | [[production:telemetry.ceph.com]]           |
+| 8.43.84.138  | Telemetry public-facing read-only VM        |
+| 8.43.84.139  | [[production:chacra.ceph.com]]              |
+| 8.43.84.140  | [[production:beta.ceph.io]]                 |
+| 8.43.84.141  | [[services:quay.ceph.io]]                   |
+===== IPv6 =====
+IPv6 was enabled on 12/11/2018.
+Our subnet is ''2620:52:3:3:0:0:0:0/64''\\
+Our default gw is ''2620:52:3:3:ffff:ffff:ffff:fffe/64''

Sepia Lab Wiki

User Tools

Site Tools

Differences

Page Tools