The Ceph Foundation purchased 7 more servers to join the longrunningcluster. The three primary goals were:

1. Faster networking between hosts
2. Large NVMe devices as OSDs
3. 12TB HDDs (largest up until now was 4TB)

Racking ticket: https://redhat.service-now.com/surl.do?n=RITM0880714

The ivan have 9x 12TB HDD, 2x 1.5TB NVMe, and 1x 350GB NVMe.

The 12TB were added to so we can say we're testing on drives larger than 8TB.

The smaller NVMe device is split into eleven equal logical volumes. One for each OSD's journal.

root@ivan04:~# lsblk
NAME                 MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                    8:0    0 894.3G  0 disk  
`-sda1                 8:1    0 894.3G  0 part  
  `-md0                9:0    0 894.1G  0 raid1 /
sdb                    8:16   0 894.3G  0 disk  
`-sdb1                 8:17   0 894.3G  0 part  
  `-md0                9:0    0 894.1G  0 raid1 /
sdc                    8:32   0  10.9T  0 disk  
sdd                    8:48   0  10.9T  0 disk  
sde                    8:64   0  10.9T  0 disk  
sdf                    8:80   0  10.9T  0 disk  
sdg                    8:96   0  10.9T  0 disk  
sdh                    8:112  0  10.9T  0 disk  
sdi                    8:128  0  10.9T  0 disk  
sdj                    8:144  0  10.9T  0 disk  
sdk                    8:160  0  10.9T  0 disk  
sr0                   11:0    1   841M  0 rom   
nvme0n1              259:0    0 349.3G  0 disk  
`-nvme0n1p1          259:3    0 349.3G  0 part  
  |-journals-sdc     253:0    0    31G  0 lvm   
  |-journals-sdd     253:1    0    31G  0 lvm   
  |-journals-sde     253:2    0    31G  0 lvm   
  |-journals-sdf     253:3    0    31G  0 lvm   
  |-journals-sdg     253:4    0    31G  0 lvm   
  |-journals-sdh     253:5    0    31G  0 lvm   
  |-journals-sdi     253:6    0    31G  0 lvm   
  |-journals-sdj     253:7    0    31G  0 lvm   
  |-journals-sdk     253:8    0    31G  0 lvm   
  |-journals-nvme1n1 253:9    0    31G  0 lvm   
  `-journals-nvme2n1 253:10   0    31G  0 lvm   
nvme1n1              259:1    0   1.5T  0 disk  
nvme2n1              259:2    0   1.5T  0 disk

Here's my bash history that can be used to set up a reesi machine's NVMe card.

ansible -a "sudo parted -s /dev/nvme0n1 mktable gpt" reesi*
ansible -a "sudo parted /dev/nvme0n1 unit '%' mkpart foo 0 50" reesi*
ansible -a "sudo parted /dev/nvme0n1 unit '%' mkpart foo 51 100" reesi*
ansible -a "sudo pvcreate /dev/nvme0n1p1" reesi*
ansible -a "sudo vgcreate journals /dev/nvme0n1p1" reesi*
for disk in sd{a..l}; do ansible -a "sudo lvcreate -L 31G -n $disk journals" reesi*; done

nvme smart-log /dev/nvme0n1

TBD