hardware:reesi
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
| hardware:reesi [2018/02/16 21:42] – created djgalloway | hardware:reesi [2019/08/01 23:49] (current) – djgalloway | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== reesi{001..006} ====== | ====== reesi{001..006} ====== | ||
| ===== Summary ===== | ===== Summary ===== | ||
| - | The Sepia lab has 6 storage hosts purchased | + | The Sepia lab has 6 storage hosts purchased |
| ===== Purchasing details ===== | ===== Purchasing details ===== | ||
| Line 7: | Line 7: | ||
| https:// | https:// | ||
| - | Tickets: https:// | + | Racking ticket: https:// |
| ===== Hardware Specs ===== | ===== Hardware Specs ===== | ||
| - | | ^ Count ^ Manufacturer | + | | ^ Count ^ Manufacturer |
| - | ^ Chassis | + | ^ Chassis |
| - | ^ Mainboard | + | ^ Mainboard |
| - | ^ CPU | 1 | Intel | Xeon(R) CPU E5-1620 v3 @ 3.50GHz | N/A | [[http:// | + | ^ CPU | 1 | Intel |
| - | ^ RAM | 2 DIMMs | Hynix Semiconductor | + | ^ RAM | 4 DIMMs | Samsung |
| - | ^ HDD | + | ^ SSD |
| - | ^ NVMe | 1 | Intel | + | ^ HDD | 11 | Seagate |
| - | ^ NIC | 2 ports | Intel | I350 | + | ^ HDD | 1 | HGST | HUH721212AL5200 |
| - | ^ NIC | 2 ports | Intel | 82599ES | + | ^ NVMe | 1 | Micron |
| - | ^ BMC | 1 | Supermicro | + | ^ NIC | 2 ports | Intel |
| + | ^ NIC | 2 ports | Intel | ||
| + | ^ BMC | 1 | Supermicro | ||
| - | ===== NVMe card notes ===== | + | ===== OSD/Block Device Information |
| - | ==== nvme-cli ==== | + | The reesi have 11x 4TB HDD, 1x 12TB HDD, and 1x 800GB NVMe. |
| - | This is untested but claims to have packages available in Fedora 23+ and Ubuntu 16.04 and up. | + | |
| - | See https:// | + | The 12TB were added to so we can say we're testing on drives larger than 8TB. |
| + | |||
| + | The NVMe device is split into two equal partitions: | ||
| + | | ||
| + | | ||
| - | === Checking NVMe Card SMART Data === | ||
| < | < | ||
| - | cd /tmp | + | root@reesi001: |
| - | git clone https://github.com/linux-nvme/nvme-cli.git | + | NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT |
| - | cd nvme-cli/ | + | sda |
| - | make | + | └─ceph--28f7427e--5558--4ffd--ae1a--51ec3042759a-osd--block--63c0a64e--a0d2--4daf--87ec--c4b00b9f3ab9 |
| - | sudo ./nvme smart-log-add /dev/nvme0 | + | sdb |
| + | └─ceph--9ddb7c35--20a6--4099--9127--947141c5452e-osd--block--7233f174--6402--4094--a48b--9aaabf508cb2 | ||
| + | sdc | ||
| + | └─ceph--cbfd182d--02e9--4c5a--b06e--7497d6109c87-osd--block--58eecdfe--b984--46da--a37a--cd04867b4e3f | ||
| + | sdd | ||
| + | └─ceph--9632797e--c4d3--45df--a2bc--03e466c16224-osd--block--20171f57--5931--4402--bbe8--a7f703aa47db | ||
| + | sde | ||
| + | sdf | ||
| + | sdg | ||
| + | sdh | ||
| + | sdi | ||
| + | sdj | ||
| + | sdk | ||
| + | sdl | ||
| + | sdm | ||
| + | ├─sdm1 | ||
| + | │ └─md2 | ||
| + | ├─sdm2 | ||
| + | │ └─md1 | ||
| + | └─sdm3 | ||
| + | sdn | ||
| + | ├─sdn1 | ||
| + | │ └─md2 | ||
| + | └─sdn2 | ||
| + | └─md1 | ||
| + | nvme0n1 | ||
| + | ├─nvme0n1p1 | ||
| + | │ ├─journals-sda 252:0 0 31G 0 lvm | ||
| + | │ ├─journals-sdb 252:1 0 31G 0 lvm | ||
| + | │ ├─journals-sdc 252:2 0 31G 0 lvm | ||
| + | │ ├─journals-sdd | ||
| + | │ ├─journals-sde | ||
| + | │ ├─journals-sdf | ||
| + | │ ├─journals-sdg | ||
| + | │ ├─journals-sdh | ||
| + | │ ├─journals-sdi | ||
| + | │ ├─journals-sdj | ||
| + | │ ├─journals-sdk | ||
| + | │ └─journals-sdl | ||
| + | └─nvme0n1p2 | ||
| + | └─ceph--9f7b3261--4778--47f9--9291--55630a41c262-osd--block--90e64c51--3344--47ce--a390--7931be9f95f1 252: | ||
| </ | </ | ||
| - | ==== Flashing Firmware ==== | ||
| - | Intel provides an RPM ([[https:// | ||
| - | Download | + | ==== How to partition/ |
| + | Here's my bash history that can be used to set up a reesi machine's NVMe card. | ||
| - | **Show NVMe cards** | ||
| - | < | ||
| - | |||
| - | **Update Firmware Example** | ||
| < | < | ||
| - | [root@smithi001 ~]# isdct show -intelssd | + | ansible |
| + | ansible -a "sudo parted / | ||
| + | ansible -a "sudo parted / | ||
| + | ansible -a "sudo pvcreate / | ||
| + | ansible -a "sudo vgcreate journals / | ||
| + | for disk in sd{a..l}; do ansible -a "sudo lvcreate -L 31G -n $disk journals" | ||
| + | </ | ||
| - | - Intel SSD DC P3700 Series CVFT533000EN400BGN - | + | ===== Replacing Drives ===== |
| + | Like the [[hardware: | ||
| - | Bootloader : 8B1B012E | + | While watching the front of a system, send '' |
| - | DevicePath : /dev/nvme0n1 | + | |
| - | DeviceStatus : Healthy | + | |
| - | Firmware : 8DV10131 | + | |
| - | FirmwareUpdateAvailable : Firmware=8DV10171 Bootloader=8B1B0131 | + | |
| - | Index : 0 | + | |
| - | ModelNumber : INTEL SSDPEDMD400G4 | + | |
| - | ProductFamily : Intel SSD DC P3700 Series | + | |
| - | SerialNumber : CVFT533000EN400BGN | + | |
| - | ## NOTE: Use the Index number from above to specify the drive you want to update | + | ===== Set up a new OSD with journal on NVMe logical volume ===== |
| + | < | ||
| + | ceph-volume lvm create --bluestore --data /dev/sda --block.db journals/ | ||
| + | </ | ||
| - | [root@smithi001 | + | **Example of successfully deployed OSD** |
| - | WARNING! You have selected to update the drives firmware! | + | < |
| - | Proceed with the update? (Y|N): Y | + | root@reesi001:~# ceph-volume lvm list |
| - | Updating firmware... | + | |
| - | - Intel SSD DC P3700 Series CVFT533000EN400BGN - | ||
| - | Status : Firmware Updated Successfully. Please reboot the system. | + | ====== osd.94 ====== |
| - | </code> | + | [block] |
| - | ==== NVMe Failure Tracking ==== | + | type block |
| - | The NVMe cards have started failing at a faster rate. I'm keeping track of when and how often to see if there' | + | osd id 94 |
| + | cluster fsid 28f7427e-5558-4ffd-ae1a-51ec3042759a | ||
| + | cluster name ceph | ||
| + | osd fsid 63c0a64e-a0d2-4daf-87ec-c4b00b9f3ab9 | ||
| + | db device | ||
| + | encrypted | ||
| + | db uuid | ||
| + | cephx lockbox secret | ||
| + | block uuid Xvjsm3-95vU-KNmw-5DuK-i3cx-fmic-xfSK2w | ||
| + | block device | ||
| + | crush device class None | ||
| - | ^ System | + | |
| - | | smithi043 | + | |
| - | | smithi048 | + | |
| - | | smithi050 | + | |
| - | | smithi038 | + | |
| - | | smithi025 | + | |
| - | | smithi039 | + | |
| - | | smithi055 | + | |
| - | | smithi057 | + | |
| - | | smithi054 | + | |
| - | | smithi021 | + | |
| - | | smithi180 | + | |
| - | ===== Other notes ===== | + | type db |
| - | ==== Mounting remote ISOs ==== | + | osd id 94 |
| - | I (dgalloway) copy ISOs to '' | + | |
| - | | + | cluster name |
| - | - '' | + | osd fsid 63c0a64e-a0d2-4daf-87ec-c4b00b9f3ab9 |
| - | | + | db device |
| - | - **Share Host**: 172.21.0.11 | + | encrypted |
| - | - **Path**: '' | + | db uuid |
| - | | + | cephx lockbox secret |
| - | - Click **Mount** | + | block uuid Xvjsm3-95vU-KNmw-5DuK-i3cx-fmic-xfSK2w |
| - | - Reboot and enter the BIOS, when prompted, by hitting **<DEL>** | + | block device |
| - | - In the last BIOS tab, "Save & Exit", select **ATEN Virtual CDROM** under **Boot Override** | + | crush device class None |
| + | </code> | ||
| - | ===== Updating BIOS ===== | + | ===== Checking NVMe Card SMART Data ===== |
| - | As of this writing, the latest BIOS version is X10SRW7.410 (4/10/2017) | + | < |
| + | nvme smart-log | ||
| + | </code> | ||
| - | - Mount '' | + | ===== Updating |
| - | - Boot to the Virtual CD | + | TBD |
| - | - Once you get a DOS prompt, run '' | + | |
| - | - The first time, it'll put the system in " | + | |
| - | - Shut the system **OFF** then back on for the BIOS update to complete | + | |
| - | - Supermicro suggests restoring BIOS to default settings then setting back up although I've found some settings get reset automatically anyway (like boot order) | + | |
hardware/reesi.1518817378.txt.gz · Last modified: by djgalloway
