hardware:ivan
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| hardware:ivan [2022/05/10 13:37] – djgalloway | hardware:ivan [2022/06/08 15:17] (current) – djgalloway | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== ivan{01..07} ====== | ====== ivan{01..07} ====== | ||
| ===== Summary ===== | ===== Summary ===== | ||
| - | The Ceph Foundation purchased 7 more servers to join the [[service: | + | The Ceph Foundation purchased 7 more servers to join the [[services: |
| - | - Faster networking between hosts | + | - Faster networking |
| - Large NVMe devices as OSDs | - Large NVMe devices as OSDs | ||
| - 12TB HDDs (largest up until now was 4TB) | - 12TB HDDs (largest up until now was 4TB) | ||
| Line 11: | Line 11: | ||
| ===== Hardware Specs ===== | ===== Hardware Specs ===== | ||
| - | | ^ Count ^ Manufacturer | + | | ^ Count ^ Manufacturer |
| - | ^ Chassis | + | ^ Chassis |
| - | ^ Mainboard | + | ^ Mainboard |
| - | ^ CPU | 1 | Intel | Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz | N/A | [[https:// | + | ^ CPU | 2 | Intel | Intel(R) Xeon(R) |
| - | ^ RAM | 4 DIMMs | Samsung | + | ^ RAM | 4 DIMMs | SK Hynix |
| - | ^ SSD | 2 | Intel | + | ^ SSD | 2 | Intel |
| - | ^ HDD | 11 | Seagate | + | ^ HDD | 9 |
| - | ^ HDD | + | ^ NVMe | 2 |
| - | ^ NVMe | 1 | Micron | + | ^ NVMe | 1 | Intel | SSDPE21M375GA |
| - | ^ NIC | 2 ports | Intel | + | ^ NIC | 2 ports | Intel |
| - | ^ NIC | 2 ports | Intel | 82599ES | + | ^ NIC | 2 ports | Mellanox |
| - | ^ BMC | 1 | Supermicro | + | ^ BMC | 1 | Supermicro |
| ===== OSD/Block Device Information ===== | ===== OSD/Block Device Information ===== | ||
| - | The ivan have 9x 12TB HDD, 2x 1.5TB NVMe, and 1x 350GB NVMe. | + | I used the Orchestrator |
| - | + | ||
| - | The 12TB were added to so we can say we're testing | + | |
| - | + | ||
| - | The smaller NVMe device is split into eleven equal logical volumes. One for each OSD's journal. | + | |
| < | < | ||
| - | root@ivan04:~# lsblk | + | root@reesi001:~# cat ivan_osd_spec.yml |
| - | NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT | + | service_type: osd |
| - | sda 8:0 0 894.3G | + | service_id: osd_using_paths |
| - | `-sda1 | + | placement: |
| - | | + | |
| - | sdb 8:16 0 894.3G | + | - ivan01 |
| - | `-sdb1 | + | - ivan02 |
| - | `-md0 9:0 0 894.1G | + | - ivan03 |
| - | sdc 8:32 | + | - ivan04 |
| - | sdd 8:48 | + | - ivan05 |
| - | sde 8:64 | + | - ivan06 |
| - | sdf 8:80 | + | - ivan07 |
| - | sdg 8:96 | + | spec: |
| - | sdh 8:112 0 10.9T 0 disk | + | |
| - | sdi 8:128 0 10.9T 0 disk | + | |
| - | sdj 8:144 0 10.9T 0 disk | + | - /dev/sdc |
| - | sdk 8:160 0 10.9T 0 disk | + | |
| - | sr0 | + | |
| - | nvme0n1 | + | |
| - | `-nvme0n1p1 | + | |
| - | |-journals-sdc 253:0 0 31G 0 lvm | + | |
| - | |-journals-sdd 253:1 0 31G 0 lvm | + | - /dev/sdi |
| - | |-journals-sde 253:2 0 31G 0 lvm | + | - /dev/sdj |
| - | |-journals-sdf 253:3 0 31G 0 lvm | + | - /dev/sdk |
| - | |-journals-sdg 253:4 0 31G 0 lvm | + | - /dev/nvme1n1 |
| - | |-journals-sdh 253:5 0 31G 0 lvm | + | - /dev/nvme2n1 |
| - | | + | |
| - | |-journals-sdj | + | paths: |
| - | |-journals-sdk | + | |
| - | |-journals-nvme1n1 253:9 0 31G 0 lvm | + | |
| - | `-journals-nvme2n1 253: | + | |
| - | nvme1n1 | + | |
| - | nvme2n1 | + | |
| - | </code> | + | |
| - | + | ||
| - | ==== How to partition/re-partition the NVMe device ==== | + | |
| - | Here's my bash history that can be used to set up a reesi machine' | + | |
| - | + | ||
| - | < | + | |
| - | ansible -a "sudo parted | + | |
| - | ansible | + | |
| - | ansible | + | |
| - | ansible | + | |
| - | ansible | + | |
| - | for disk in sd{a..l}; do ansible -a "sudo lvcreate -L 31G -n $disk journals" | + | |
| </ | </ | ||
| Line 89: | Line 69: | ||
| ===== Installation Quirks/ | ===== Installation Quirks/ | ||
| + | ==== Networking ==== | ||
| + | Initially, I wanted to have the 1Gb interface cabled on VLAN100 and the 25Gb interfaces cabled to VLAN101 (back.sepia.ceph.com). Up until now I have never really used VLAN101. I was able to get both NICs up, IPs assigned, and the servers could reach each other. The LRC could also reach these servers on their 25Gb/'' | ||
| + | |||
| + | I added the hosts to the cluster using the '' | ||
| + | |||
| + | I reached out to Red Hat IT to have the 25Gb network ports switched over to VLAN100. After that, I struggled to get eno1 (the 1Gb interface) to **not** come up on boot since I didn't need it anymore. | ||
| + | |||
| + | Finally I figured out< | ||
| + | # cat / | ||
| + | [Match] | ||
| + | Name=eno1 | ||
| + | |||
| + | [Network] | ||
| + | DHCP=no | ||
| + | </ | ||
| + | |||
| ==== CentOS 8 ==== | ==== CentOS 8 ==== | ||
| I could not for the life of me get ivan05 to install using the Ubuntu preseed below. Its settings are identical to the rest of the machines. I remember someone (I think GregF?) suggest in a CLT call that we should have a mixture of OSes in the LRC so I decided to use CentOS8 instead. | I could not for the life of me get ivan05 to install using the Ubuntu preseed below. Its settings are identical to the rest of the machines. I remember someone (I think GregF?) suggest in a CLT call that we should have a mixture of OSes in the LRC so I decided to use CentOS8 instead. | ||
hardware/ivan.1652189829.txt.gz · Last modified: by djgalloway