services:networking
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
services:networking [2020/01/15 19:34] djgalloway |
services:networking [2026/08/11 19:51] (current) djgalloway [Firewall Rules] |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== Networking ====== | ====== Networking ====== | ||
| - | ==== Networks ==== | + | ===== Networks ===== |
| - | The Sepia internal LANs are divided up into four /20 networks described below. | + | ^ Name ^ Purpose ^ Size/CIDR ^ VLAN ID ^ Domain ^ MTU ^ |
| - | | ^ front.sepia.ceph.com ^ back.sepia.ceph.com ^ ipmi.sepia.ceph.com ^ Other ^ | + | | Sepia Front Copy | Created to make lab migration easy | 172.21.0.0/20 | 100 | front.sepia.ceph.com | 1500 | |
| - | ^ Available IP Range | 172.21.0.1 - 172.21.15.250 | 172.21.16.1 - 172.21.31.250 | 172.21.32.1 - 172.21.47.250 | 172.21.48.1 - 172.21.63.250 | | + | | Sepia IPMI Copy | Created to make lab migration easy | 172.21.32.0/20 | 102 | ipmi.sepia.ceph.com | 1500 | |
| - | ^ Gateway | 172.21.15.254 | 172.21.31.254 | 172.21.47.254 | 172.21.63.254 | | + | | New Front | Server uplinks | 10.20.192.0/20 | 1338 | front.sepia.ceph.com | 1500 | |
| - | ^ VLAN ID | 100 | 101 | 102 | 103 | | + | | New IPMI | New server BMC | 10.20.208.0/20 | 1339 | ipmi.sepia.ceph.com | 1500 | |
| - | ^ Use | Main Network interface | Not used at the moment | Used to access OOB management | 172.21.48.0/20 is currently used for VPN client IPs. | | + | | VPN Clients | VPN Clients | 172.16.48.1/24 | 1340 | vpn.sepia.ceph.com | 1500 | |
| + | | LRC | Ceph Cluster Backend Storage Traffic | 172.16.50.1/23 | 1341 | lrc.sepia.ceph.com | 9000 | | ||
| + | | Openshift Cluster | Openshift Node IPs (master, workers, VIPs) | 172.16.53.1/25 | 1342 | os.sepia.ceph.com | 1500 | | ||
| + | | Openshift Provision | Openshift Provisioning/PXE | 172.16.55.1/26 | 1343 | pxe.os.sepia.ceph.com | 1500 | | ||
| + | | Openshift Clients | Openshift VM Clients | 172.16.56.1/23 | 1344 | client.os.sepia.ceph.com | 1500 | | ||
| + | | ODF Public | ODF Client-facing (Ceph Public) | 172.16.59.1/25 | 1345 | pub.odf.sepia.ceph.com | 1500 | | ||
| + | | ODF Storage | ODF Cluster Traffic | 172.16.60.1/25 | 1346 | back.odf.sepia.ceph.com | 9000 | | ||
| + | | Public | Internet | 192.86.31.0/24 | 200 | | | | ||
| + | | vlan104 | Officinalis | 172.21.64.0/24 | 104 | vlan104.sepia.ceph.com | | | ||
| + | | vlan105 | Officinalis | 172.21.65.0/24 | 105 | vlan105.sepia.ceph.com | | | ||
| + | | vlan106 | Officinalis | 172.21.66.0/24 | 106 | vlan106.sepia.ceph.com | | | ||
| + | | vlan107 | Officinalis | 172.21.67.0/24 | 107 | vlan107.sepia.ceph.com | | | ||
| - | In January 2020, 4 new small subnets were created for the [[hardware:officinalis]] nodes to each have an IP on different subnets. NIC1 is on VLAN104, NIC2 on VLAN105, etc. | + | The first IP of each is the gateway and the first 5 IPs should not be used (.0 through .4) |
| - | | ^ front.sepia.ceph.com ^ back.sepia.ceph.com ^ ipmi.sepia.ceph.com ^ Other ^ | + | ==== Officinalis ==== |
| - | ^ Available IP Range | 172.21.0.1 - 172.21.15.250 | 172.21.16.1 - 172.21.31.250 | 172.21.32.1 - 172.21.47.250 | 172.21.48.1 - 172.21.63.250 | | + | In January 2020, 4 new small subnets were created for the [[hardware:officinalis]] nodes to each have an IP on different subnets. NIC1 is on VLAN104, NIC2 on VLAN105, etc. At present, the only switch configured to use these subnets is the QFX5200 purchased specifically for the Officinalis nodes but other switches could be configured if desired. |
| - | ^ Gateway | 172.21.15.254 | 172.21.31.254 | 172.21.47.254 | 172.21.63.254 | | + | |
| - | ^ VLAN ID | 100 | 101 | 102 | 103 | | + | |
| - | ^ Use | Main Network interface | Not used at the moment | Used to access OOB management | 172.21.48.0/20 is currently used for VPN client IPs. | | + | |
| - | ==== Hardware ==== | + | [[services:DNS]] and [[services:DHCP]] were configured to manage these VLANs in https://github.com/ceph/ceph-sepia-secrets/pull/463. |
| + | |||
| + | ===== Firewall Rules ===== | ||
| + | |||
| + | IBM OneIT manages the Juniper SRX in front of the lab. | ||
| + | |||
| + | ^ Application Name ^ Inbound/Outbound ^ Destination IP or Subnet/CIDR ^ Port Number ^ Protocol ^ | ||
| + | | OpenVPN | Inbound | 192.86.31.5 | udp-1194 \\ udp-51820 | UDP | | ||
| + | | OpenVPN | Inbound | 192.86.31.12 | 1194 | UDP | | ||
| + | | nginx / file share (e.g., https://filedump.ceph.com/) | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | GitWeb | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | githelper (https://github.com/ceph/githelper) | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | Etherpad (pad.ceph.com) | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | Pulpito (https://github.com/ceph/pulpito) | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | qa-proxy.ceph.com | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | quay.ceph.io | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | sentry.ceph.com | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | teuthology-api.ceph.com | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | Dokuwiki (wiki.sepia.ceph.com) | Inbound | 192.86.31.5 | 443 | TCP | | ||
| + | | Postorius & Hyperkitty | Inbound | 192.86.31.12 | 443 | TCP | | ||
| + | | Postfix (Ceph project mailing lists) | Inbound | 192.86.31.12 | 25 | TCP | | ||
| + | | telemetry.ceph.com (apache2) | Inbound | 192.86.31.13 | 443 | TCP | | ||
| + | | telemetry-public.ceph.com (nginx serving grafana) | Inbound | 192.86.31.14 | 443 | | | ||
| + | | chacra.ceph.com (nginx) | Inbound | 192.86.31.15 | 443 | TCP | | ||
| + | | INBOUND OKAY TO BLOCK: 20,21,23,830,3389,58xx,59xx,53,69,111,389,NFS,Management,DB,ExplicitlyDenied | Inbound | 192.86.31.5-254 | | | | ||
| + | | Outbound web traffic | Outbound | 0.0.0.0/0 | 443 | | | ||
| + | | Jenkins agent SSH to Jenkins controller | Outbound | 158.69.68.47 | 2222 | | | ||
| + | | Jenkins agent SSH to Jenkins controller | Outbound | 158.69.65.239 | 2222 | | | ||
| + | | rsync unsigned packages from chacra.ceph.com | Outbound | 192.86.31.15 | 2222 | | | ||
| + | | rsync signed packages to download.ceph.com | Outbound | 158.69.68.124 | 22 | | | ||
| + | | OUTBOUND OKAY TO BLOCK: 20,21,23,830,3389,58xx,59xx,69,111,389,NFS,Management,DB,ExplicitlyDenied | Outbound | 0.0.0.0/0 | | | | ||
| + | | All non-blocklisted ports | Outbound | 0.0.0.0/0 | All | All | | ||
| + | | All, unrestricted | Outbound | 0.0.0.0/0 | All | All | | ||
| + | |||
| + | Firewall rules can be updated by following https://w3.ibm.com/w3publisher/oneit/nw-support/next-generation-datacenter/firewall-rules. | ||
| + | |||
| + | IES keeps [[https://ibm.box.com/s/in756ga4nfv79swvhj0btwv5bitxc95s|this spreadsheet]] up to date and should be considered the source of truth over this wiki article. | ||
| + | |||
| + | ===== Hardware ===== | ||
| All the switches in the Sepia lab are either Juniper EX4300 1Gb or QFX5100 10Gb (SFP+) except: | All the switches in the Sepia lab are either Juniper EX4300 1Gb or QFX5100 10Gb (SFP+) except: | ||
| * A [[https://opendcim.engineering.redhat.com/devices.php?DeviceID=7960|Mellanox SX1012]] private switch connected to the [[hardware:rhoda]] and [[hardware:incerta]] nodes | * A [[https://opendcim.engineering.redhat.com/devices.php?DeviceID=7960|Mellanox SX1012]] private switch connected to the [[hardware:rhoda]] and [[hardware:incerta]] nodes | ||
| * A QFX5200 in A09 for the Officinalis nodes | * A QFX5200 in A09 for the Officinalis nodes | ||
| + | * An uplinked QFX5120 in B12 for the All Flash (name TBD) systems | ||
| + | * A NON-uplinked [[https://wiki.sepia.ceph.com/doku.php?id=hardware:robsoni#qfx-rgw|QFX5120]] in A15 for RGW Scale testing gear | ||
| **The Sepia Networking core is === OUT === of 40G ports. No more switches can be added without purchasing additional equipment.** | **The Sepia Networking core is === OUT === of 40G ports. No more switches can be added without purchasing additional equipment.** | ||
| - | ==== Typical Switch config ==== | + | ===== Typical Switch config ===== |
| In racks that only have 1Gb networking (e.g., [[hardware:Mira]]), the top-of-rack switch is most likely split | In racks that only have 1Gb networking (e.g., [[hardware:Mira]]), the top-of-rack switch is most likely split | ||
| * Ports 0-23 assigned to VLAN 100 (front) | * Ports 0-23 assigned to VLAN 100 (front) | ||
| Line 34: | Line 82: | ||
| The infra rack (houses RHEV, teuthology, gitbuilder.ceph.com), AA-06, has a 10Gb switch with no standard port assignments (IOW, it's kinda random). I believe any port that's not already in use isn't configured. | The infra rack (houses RHEV, teuthology, gitbuilder.ceph.com), AA-06, has a 10Gb switch with no standard port assignments (IOW, it's kinda random). I believe any port that's not already in use isn't configured. | ||
| - | ==== Switch Port Config Changes ==== | + | ===== Switch Port Config Changes ===== |
| Red Hat IT manages all switches in the Sepia lab and switch port configuration changes should be submitted to [[servicedesk@redhat.com]]. | Red Hat IT manages all switches in the Sepia lab and switch port configuration changes should be submitted to [[servicedesk@redhat.com]]. | ||
| Line 53: | Line 101: | ||
| </file> | </file> | ||
| - | ==== Public IPs ==== | + | ===== Public IPs ===== |
| The Community Cage has a block of public IPs dedicated to it. Ceph's usable IPs are 8.43.84.129 - 8.43.84.186. | The Community Cage has a block of public IPs dedicated to it. Ceph's usable IPs are 8.43.84.129 - 8.43.84.186. | ||
| Line 64: | Line 112: | ||
| | 8.43.84.134 | [[production:drop.ceph.com]] | | | 8.43.84.134 | [[production:drop.ceph.com]] | | ||
| | 8.43.84.135 | RDU download.ceph.com mirror (rgw s3 only) | | | 8.43.84.135 | RDU download.ceph.com mirror (rgw s3 only) | | ||
| - | | 8.43.84.136 | lists.ceph.io | | + | | 8.43.84.136 | [[production:lists.ceph.io]] | |
| - | | | | | + | | 8.43.84.137 | [[production:telemetry.ceph.com]] | |
| + | | 8.43.84.138 | Telemetry public-facing read-only VM | | ||
| + | | 8.43.84.139 | [[production:chacra.ceph.com]] | | ||
| + | | 8.43.84.140 | [[production:beta.ceph.io]] | | ||
| + | | 8.43.84.141 | [[services:quay.ceph.io]] | | ||
| - | ==== IPv6 ==== | + | ===== IPv6 ===== |
| IPv6 was enabled on 12/11/2018. | IPv6 was enabled on 12/11/2018. | ||
| Our subnet is ''2620:52:3:3:0:0:0:0/64''\\ | Our subnet is ''2620:52:3:3:0:0:0:0/64''\\ | ||
| Our default gw is ''2620:52:3:3:ffff:ffff:ffff:fffe/64'' | Our default gw is ''2620:52:3:3:ffff:ffff:ffff:fffe/64'' | ||
services/networking.1579116852.txt.gz · Last modified: 2020/01/15 19:34 by djgalloway
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
