This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
hardware:robsoni [2021/06/03 00:22] djgalloway [Table] |
hardware:robsoni [2021/06/03 00:30] (current) djgalloway [Table] |
||
---|---|---|---|
Line 23: | Line 23: | ||
These PXE using Legacy/BIOS mode and can be provisioned via Cobbler normally. | These PXE using Legacy/BIOS mode and can be provisioned via Cobbler normally. | ||
- | ==== Network Config ==== | + | ===== Network Config ===== |
- | These nodes have a regular 1Gb uplink but then 2x 25Gb connections per server to a QFX5120 for backend storage traffic. | + | ==== qfx-rgw ==== |
+ | These nodes have a regular 1Gb uplink but then 2x 25Gb connections per server to a QFX5120-32C for backend storage traffic. This switch is managed by Ceph Infra and not uplinked. | ||
+ | |||
+ | I purchased qty 5 100Gb -> 4x 25Gb breakout cables with this. See https://docs.google.com/spreadsheets/d/1_ETaz5ollV_66ODFGXiVWEibecdBZIvHg9X1zN1pF88/edit#gid=0 | ||
+ | |||
+ | I just created a ''/24'' vlan on the switch giving each server it's own Aggregated Ethernet interface. | ||
+ | |||
+ | ^ Server ^ Private IP ^ | ||
+ | | robsoni01 | 172.20.23.1 | | ||
+ | | robsoni02 | 172.20.23.2 | | ||
+ | | robsoni03 | 172.20.23.3 | | ||
+ | | robsoni04 | 172.20.23.4 | | ||
+ | | robsoni05 | 172.20.23.5 | | ||
The switch is named ''qfx-rgw.ipmi.sepia.ceph.com'' and can be accessed via SSH. Here's the current switch config that seems to be working. | The switch is named ''qfx-rgw.ipmi.sepia.ceph.com'' and can be accessed via SSH. Here's the current switch config that seems to be working. | ||
Line 361: | Line 373: | ||
} | } | ||
} | } | ||
+ | </code> | ||
+ | |||
+ | ==== Server bond config ==== | ||
+ | Here's what I ran to set up the bonds initially. The bonding type may change. | ||
+ | <code> | ||
+ | nmcli connection add type bond con-name bond0 ifname bond0 bond.options "mode=802.3ad" | ||
+ | nmcli connection add type ethernet slave-type bond con-name bond0-port1 ifname eno1np0 master bond0 | ||
+ | nmcli connection add type ethernet slave-type bond con-name bond0-port2 ifname eno2np1 master bond0 | ||
+ | |||
+ | ### Change the IP in this line! | ||
+ | nmcli connection modify bond0 ipv4.addresses '172.20.23.X/24' | ||
+ | |||
+ | nmcli connection modify bond0 ipv4.gateway '172.20.23.254' | ||
+ | nmcli connection modify bond0 ipv4.method manual | ||
+ | nmcli connection modify bond0 connection.autoconnect-slaves 1 | ||
+ | nmcli connection up bond0 | ||
</code> | </code> |