Differences

This shows you the differences between two versions of the page.

--- hardware:robsoni [2021/06/03 00:22] – [Table] djgalloway
+++ hardware:robsoni [2021/06/03 00:30] (current) – [Table] djgalloway
@@ Line 23: / Line 23: @@
 These PXE using Legacy/BIOS mode and can be provisioned via Cobbler normally.
-==== Network Config ====
+===== Network Config =====
-These nodes have a regular 1Gb uplink but then 2x 25Gb connections per server to a QFX5120 for backend storage traffic.
+==== qfx-rgw ====
+These nodes have a regular 1Gb uplink but then 2x 25Gb connections per server to a QFX5120-32C for backend storage traffic.  This switch is managed by Ceph Infra and not uplinked.
+I purchased qty 5 100Gb -> 4x 25Gb breakout cables with this.  See https://docs.google.com/spreadsheets/d/1_ETaz5ollV_66ODFGXiVWEibecdBZIvHg9X1zN1pF88/edit#gid=0
+I just created a ''/24'' vlan on the switch giving each server it's own Aggregated Ethernet interface.
+^ Server     ^ Private IP   ^
+| robsoni01  | 172.20.23.1  |
+| robsoni02  | 172.20.23.2  |
+| robsoni03  | 172.20.23.3  |
+| robsoni04  | 172.20.23.4  |
+| robsoni05  | 172.20.23.5  |
 The switch is named ''qfx-rgw.ipmi.sepia.ceph.com'' and can be accessed via SSH.  Here's the current switch config that seems to be working.
@@ Line 361: / Line 373: @@
     }
 }
+</code>
+==== Server bond config ====
+Here's what I ran to set up the bonds initially.  The bonding type may change.
+<code>
+nmcli connection add type bond con-name bond0 ifname bond0 bond.options "mode=802.3ad"
+nmcli connection add type ethernet slave-type bond con-name bond0-port1 ifname eno1np0 master bond0
+nmcli connection add type ethernet slave-type bond con-name bond0-port2 ifname eno2np1 master bond0
+### Change the IP in this line!
+nmcli connection modify bond0 ipv4.addresses '172.20.23.X/24'
+nmcli connection modify bond0 ipv4.gateway '172.20.23.254'
+nmcli connection modify bond0 ipv4.method manual
+nmcli connection modify bond0 connection.autoconnect-slaves 1
+nmcli connection up bond0
 </code>