hardware:robsoni
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
|
hardware:robsoni [2021/06/03 00:14] djgalloway created |
hardware:robsoni [2021/06/03 00:30] (current) djgalloway [Table] |
||
|---|---|---|---|
| Line 8: | Line 8: | ||
| ===== Hardware Specs ===== | ===== Hardware Specs ===== | ||
| - | | ^ Count ^ Manufacturer ^ Model ^ Capacity ^ Notes ^ | + | | ^ Count ^ Manufacturer ^ Model ^ Capacity ^ Notes ^ |
| - | ^ Chassis | 1U | Dell | PowerEdge R6515 | N/A | | | + | ^ Chassis | 2U | Dell | PowerEdge R740 | N/A | | |
| - | ^ Mainboard | N/A | Dell | 0R4CNN | N/A | | | + | ^ Mainboard | N/A | Dell | 0WRPXK | N/A | | |
| - | ^ CPU | 1 | AMD | AMD EPYC 7742 | 128 cores | | | + | ^ CPU | 2 | INtel | Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz | 80 cores | [[https://ark.intel.com/content/www/us/en/ark/products/192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz.html|ARK]] | |
| - | ^ RAM | 8 DIMMs | Micron | 18ASF2G72PDZ-3G2E1 | 16GB | 128GB Total | | + | ^ RAM | 6 DIMMs | SK Hynix | HMA84GR7CJR4N-XN | 32GB | 192GB Total | |
| - | ^ SSD | 2 | Micron | MTFDDAV480TDS | 480GB | Behind hardware RAID1 for OS | | + | ^ SSD | 1 | Micron | MTFDDAV240TDU | 240GB | For OS | |
| - | ^ NVMe | 1 | Dell | P4510 | 1TB | For OSD journals? | | + | ^ HDD | 8 | Toshiba | MG04SCA40ENY | 4TB | For OSDs | |
| - | ^ NIC | 2 ports | Dell | | 1Gb | On-board. Unused. | | + | ^ NVMe | 1 | Dell | PM1725b | 1.6TB | For OSD journals? | |
| - | ^ NIC | 2 ports | Broadcom | 57416 BaseT | 1/10Gb | Oops. Won't be using this. | | + | ^ NIC | 4 ports | Intel | I350 | 1Gb | Uplink | |
| - | ^ NIC | 2 ports | Mellanox | ConnectX-6 | 100Gb | 1 port as uplink | | + | ^ NIC | 2 ports | Broadcom | | 25Gb | Both cabled per server | |
| - | ^ BMC | 1 | Quanta | N/A | N/A | Reachable at $host.ipmi.sepia.ceph.com using usual IPMI credentials. | | + | ^ BMC | 1 | Quanta | N/A | N/A | Reachable at $host.ipmi.sepia.ceph.com using usual IPMI credentials. | |
| ===== PXE/Reimaging ===== | ===== PXE/Reimaging ===== | ||
| These PXE using Legacy/BIOS mode and can be provisioned via Cobbler normally. | These PXE using Legacy/BIOS mode and can be provisioned via Cobbler normally. | ||
| - | ==== Network Config ==== | + | ===== Network Config ===== |
| - | These nodes have a regular 1Gb uplink but then 2x 25Gb connections per server to a QFX5120 for backend storage traffic. | + | ==== qfx-rgw ==== |
| + | These nodes have a regular 1Gb uplink but then 2x 25Gb connections per server to a QFX5120-32C for backend storage traffic. This switch is managed by Ceph Infra and not uplinked. | ||
| + | |||
| + | I purchased qty 5 100Gb -> 4x 25Gb breakout cables with this. See https://docs.google.com/spreadsheets/d/1_ETaz5ollV_66ODFGXiVWEibecdBZIvHg9X1zN1pF88/edit#gid=0 | ||
| + | |||
| + | I just created a ''/24'' vlan on the switch giving each server it's own Aggregated Ethernet interface. | ||
| + | |||
| + | ^ Server ^ Private IP ^ | ||
| + | | robsoni01 | 172.20.23.1 | | ||
| + | | robsoni02 | 172.20.23.2 | | ||
| + | | robsoni03 | 172.20.23.3 | | ||
| + | | robsoni04 | 172.20.23.4 | | ||
| + | | robsoni05 | 172.20.23.5 | | ||
| The switch is named ''qfx-rgw.ipmi.sepia.ceph.com'' and can be accessed via SSH. Here's the current switch config that seems to be working. | The switch is named ''qfx-rgw.ipmi.sepia.ceph.com'' and can be accessed via SSH. Here's the current switch config that seems to be working. | ||
| Line 361: | Line 373: | ||
| } | } | ||
| } | } | ||
| + | </code> | ||
| + | |||
| + | ==== Server bond config ==== | ||
| + | Here's what I ran to set up the bonds initially. The bonding type may change. | ||
| + | <code> | ||
| + | nmcli connection add type bond con-name bond0 ifname bond0 bond.options "mode=802.3ad" | ||
| + | nmcli connection add type ethernet slave-type bond con-name bond0-port1 ifname eno1np0 master bond0 | ||
| + | nmcli connection add type ethernet slave-type bond con-name bond0-port2 ifname eno2np1 master bond0 | ||
| + | |||
| + | ### Change the IP in this line! | ||
| + | nmcli connection modify bond0 ipv4.addresses '172.20.23.X/24' | ||
| + | |||
| + | nmcli connection modify bond0 ipv4.gateway '172.20.23.254' | ||
| + | nmcli connection modify bond0 ipv4.method manual | ||
| + | nmcli connection modify bond0 connection.autoconnect-slaves 1 | ||
| + | nmcli connection up bond0 | ||
| </code> | </code> | ||
hardware/robsoni.1622679263.txt.gz · Last modified: 2021/06/03 00:14 by djgalloway
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
