User Tools

Site Tools


services:networking

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
services:networking [2016/07/05 16:20]
dgalloway [Networks]
services:networking [2021/06/03 03:21] (current)
djgalloway [Hardware]
Line 1: Line 1:
 ====== Networking ====== ====== Networking ======
  
-==== Networks ====+===== Networks ​=====
  
-The Sepia internal LANs are divided up into four /20 networks described below.+The Sepia internal LANs are divided up into four /20 networks ​(netmask 255.255.240.0) ​described below.
 |                     ^ front.sepia.ceph.com ​       ^ back.sepia.ceph.com ​         ^ ipmi.sepia.ceph.com ​           ^ Other                                                                                 ^ |                     ^ front.sepia.ceph.com ​       ^ back.sepia.ceph.com ​         ^ ipmi.sepia.ceph.com ​           ^ Other                                                                                 ^
 ^ Available IP Range  | 172.21.0.1 - 172.21.15.250 ​ | 172.21.16.1 - 172.21.31.250 ​ | 172.21.32.1 - 172.21.47.250 ​   | 172.21.48.1 - 172.21.63.250 ​                                                          | ^ Available IP Range  | 172.21.0.1 - 172.21.15.250 ​ | 172.21.16.1 - 172.21.31.250 ​ | 172.21.32.1 - 172.21.47.250 ​   | 172.21.48.1 - 172.21.63.250 ​                                                          |
Line 10: Line 10:
 ^ Use                 | Main Network interface ​     | Not used at the moment ​      | Used to access OOB management ​ | 172.21.48.0/​20 is currently used for VPN client IPs.  | ^ Use                 | Main Network interface ​     | Not used at the moment ​      | Used to access OOB management ​ | 172.21.48.0/​20 is currently used for VPN client IPs.  |
  
-==== Hardware ==== +In January 2020, 4 new small subnets were created for the [[hardware:officinalis]] nodes to each have an IP on different subnets NIC1 is on VLAN104, NIC2 on VLAN105, etc At present, the only switch ​configured ​to use these subnets is the QFX5200 purchased specifically for the Officinalis ​nodes but other switches could be configured if desired.
-All the switches in the Sepia lab are either Juniper EX4300 1Gb or QFX5100 10Gb (SFP+) with the exception of a [[https://opendcim.engineering.redhat.com/​devices.php?​DeviceID=7960|Mellanox SX1012]] private ​switch ​connected ​to the [[hardware:​rhoda]] and [[hardware:​incerta]] ​nodes.+
  
-==== Typical Switch config ====+==== Officinalis ==== 
 + 
 +[[services:​DNS]] and [[services:​DHCP]] were configured to manage these VLANs in https://​github.com/​ceph/​ceph-sepia-secrets/​pull/​463. 
 + 
 +|                     ^ vlan104.sepia.ceph.com ​      ^ vlan105.sepia.ceph.com ​      ^ vlan106.sepia.ceph.com ​      ^ vlan107.sepia.ceph.com ​      ^ 
 +^ Available IP Range  | 172.21.64.1 - 172.21.64.250 ​ | 172.21.65.1 - 172.21.65.250 ​ | 172.21.66.1 - 172.21.66.250 ​ | 172.21.67.1 - 172.21.67.250 ​ | 
 +^ Gateway ​            | 172.21.64.254 ​               | 172.21.65.254 ​               | 172.21.66.254 ​               | 172.21.67.254 ​               | 
 +^ VLAN ID             | 104                          | 105                          | 106                          | 107                          | 
 + 
 +===== Hardware ===== 
 +All the switches in the Sepia lab are either Juniper EX4300 1Gb or QFX5100 10Gb (SFP+) except: 
 +  * A [[https://​opendcim.engineering.redhat.com/​devices.php?​DeviceID=7960|Mellanox SX1012]] private switch connected to the [[hardware:​rhoda]] and [[hardware:​incerta]] nodes 
 +  * A QFX5200 in A09 for the Officinalis nodes 
 +  * An uplinked QFX5120 in B12 for the All Flash (name TBD) systems 
 +  * A NON-uplinked [[https://​wiki.sepia.ceph.com/​doku.php?​id=hardware:​robsoni#​qfx-rgw|QFX5120]] in A15 for RGW Scale testing gear 
 + 
 +**The Sepia Networking core is === OUT === of 40G ports. ​ No more switches can be added without purchasing additional equipment.** 
 + 
 +===== Typical Switch config ​=====
 In racks that only have 1Gb networking (e.g., [[hardware:​Mira]]),​ the top-of-rack switch is most likely split In racks that only have 1Gb networking (e.g., [[hardware:​Mira]]),​ the top-of-rack switch is most likely split
   * Ports 0-23 assigned to VLAN 100 (front)   * Ports 0-23 assigned to VLAN 100 (front)
Line 20: Line 37:
 The racks full of [[hardware:​Smithi]] have a 10Gb and 1Gb switch. ​ The 10Gb switch is assigned to vlan100 (front) and the 1Gb switch to vlan102 (ipmi). The racks full of [[hardware:​Smithi]] have a 10Gb and 1Gb switch. ​ The 10Gb switch is assigned to vlan100 (front) and the 1Gb switch to vlan102 (ipmi).
  
-The infra rack (houses RHEV, teuthology, gitbuilder.ceph.com),​ AA-06, has a 10Gb switch with no standard port assignments. ​ I believe any port that's not already in use isn't configured.+The infra rack (houses RHEV, teuthology, gitbuilder.ceph.com),​ AA-06, has a 10Gb switch with no standard port assignments ​(IOW, it's kinda random).  I believe any port that's not already in use isn't configured.
  
-==== Switch Port Config Changes ====+===== Switch Port Config Changes ​=====
  
 Red Hat IT manages all switches in the Sepia lab and switch port configuration changes should be submitted to [[servicedesk@redhat.com]]. Red Hat IT manages all switches in the Sepia lab and switch port configuration changes should be submitted to [[servicedesk@redhat.com]].
Line 40: Line 57:
 Thanks Thanks
 </​file>​ </​file>​
 +
 +===== Public IPs =====
 +The Community Cage has a block of public IPs dedicated to it.  Ceph's usable IPs are 8.43.84.129 - 8.43.84.186.
 +
 +^ IP           ^ Use                                         ^
 +| 8.43.84.129 ​ | OpenVPN gateway and reverse proxies ​        |
 +| 8.43.84.130 ​ | [[production:​gitbuilder.ceph.com]] ​         |
 +| 8.43.84.131 ​ | Jump host for Chinese devs                  |
 +| 8.43.84.132 ​ | [[production:​git.ceph.com]] ​                |
 +| 8.43.84.133 ​ | Old OpenVPN server (store01) ​               |
 +| 8.43.84.134 ​ | [[production:​drop.ceph.com]] ​               |
 +| 8.43.84.135 ​ | RDU download.ceph.com mirror (rgw s3 only)  |
 +| 8.43.84.136 ​ | [[production:​lists.ceph.io]] ​               |
 +| 8.43.84.137 ​ | [[production:​telemetry.ceph.com]] ​          |
 +| 8.43.84.138 ​ | Telemetry public-facing read-only VM        |
 +| 8.43.84.139 ​ | [[production:​chacra.ceph.com]] ​             |
 +| 8.43.84.140 ​ | [[production:​beta.ceph.io]] ​                |
 +| 8.43.84.141 ​ | [[services:​quay.ceph.io]] ​                  |
 +
 +===== IPv6 =====
 +IPv6 was enabled on 12/11/2018.
 +
 +Our subnet is ''​2620:​52:​3:​3:​0:​0:​0:​0/​64''​\\
 +Our default gw is ''​2620:​52:​3:​3:​ffff:​ffff:​ffff:​fffe/​64''​
services/networking.1467735642.txt.gz · Last modified: 2016/07/05 16:20 by dgalloway