This is an old revision of the document!
Table of Contents
reesi{001..006}
Summary
The Sepia lab has 6 storage hosts purchased January 2018 to replace the miras used in the LONG_RUNNING_CLUSTER.
Purchasing details
https://docs.google.com/spreadsheets/d/1-CqEo-927duC0sClr78WcZgVIFdY55iE_K8oU2G7aSw
Racking ticket: https://redhat.service-now.com/surl.do?n=PNT0117459
Hardware Specs
| Count | Manufacturer | Model | Capacity | Notes | |
|---|---|---|---|---|---|
| Chassis | 1U | Supermicro | SYS-5018R-WR | N/A | |
| Mainboard | N/A | Supermicro | X10SRW-F | N/A | |
| CPU | 1 | Intel | Xeon(R) CPU E5-1620 v3 @ 3.50GHz | N/A | ARK |
| RAM | 2 DIMMs | Hynix Semiconductor | HMA42GR7MFR4N-TF | 16GB | 32GB Total |
| HDD | 1 | Seagate | ST1000NM0033 | 1TB | |
| NVMe | 1 | Intel | P3700 | 400GB | |
| NIC | 2 ports | Intel | I350 | 1Gb | Not used |
| NIC | 2 ports | Intel | 82599ES | 10Gb | 1 port cabled per system on front VLAN |
| BMC | 1 | Supermicro | N/A | N/A | Reachable at $host.ipmi.sepia.ceph.com |
NVMe card notes
nvme-cli
This is untested but claims to have packages available in Fedora 23+ and Ubuntu 16.04 and up.
Checking NVMe Card SMART Data
cd /tmp git clone https://github.com/linux-nvme/nvme-cli.git cd nvme-cli/ make sudo ./nvme smart-log-add /dev/nvme0
Flashing Firmware
Intel provides an RPM (Intel Datacenter Tool) to configure the NVMe cards, update firmware, etc. The firmwares are baked into the RPM so it's important to download the latest zip from Intel whenever possible.
Download the zip file, copy to testnode, unzip and yum localinstall the applicable isdct RPM.
Show NVMe cards
isdct show -intelssd
Update Firmware Example
[root@smithi001 ~]# isdct show -intelssd - Intel SSD DC P3700 Series CVFT533000EN400BGN - Bootloader : 8B1B012E DevicePath : /dev/nvme0n1 DeviceStatus : Healthy Firmware : 8DV10131 FirmwareUpdateAvailable : Firmware=8DV10171 Bootloader=8B1B0131 Index : 0 ModelNumber : INTEL SSDPEDMD400G4 ProductFamily : Intel SSD DC P3700 Series SerialNumber : CVFT533000EN400BGN ## NOTE: Use the Index number from above to specify the drive you want to update [root@smithi001 ~]# isdct load -intelssd 0 WARNING! You have selected to update the drives firmware! Proceed with the update? (Y|N): Y Updating firmware... - Intel SSD DC P3700 Series CVFT533000EN400BGN - Status : Firmware Updated Successfully. Please reboot the system.
NVMe Failure Tracking
The NVMe cards have started failing at a faster rate. I'm keeping track of when and how often to see if there's a pattern we can interrupt.
| System | Date Failed | Ticket | Notes |
|---|---|---|---|
| smithi043 | 1/3/2017 | RT 433092 | |
| smithi048 | 4/19/2017 | RT 444421 | |
| smithi050 | 4/19/2017 | RT 444421 | |
| smithi038 | 11/29/2017 | PNT0111325 | |
| smithi025 | 12/11/2017 | PNT0120775 | |
| smithi039 | 12/11/2017 | PNT0120775 | |
| smithi055 | 12/8/2017 | PNT0120775 | |
| smithi057 | 1/12/2018 | PNT0138316 | |
| smithi054 | 1/16/2018 | PNT0141432 | |
| smithi021 | 1/19/2018 | PNT0143431 | |
| smithi180 | 1/22/2018 | PNT0160022 | New batch of servers :( |
Other notes
Mounting remote ISOs
I (dgalloway) copy ISOs to cobbler.front.sepia.ceph.com:/samba/anonymous.
- Disable the firewall on cobbler.front.sepia.ceph.com
service iptables stop
- In the smithi IPMI web UI, under Virtual Media → CD-ROM Image, set the following parameters:
- Share Host: 172.21.0.11
- Path:
\Anonymous\foo.iso
- Click Save
- Click Mount
- Reboot and enter the BIOS, when prompted, by hitting <DEL>
- In the last BIOS tab, “Save & Exit”, select ATEN Virtual CDROM under Boot Override
Updating BIOS
As of this writing, the latest BIOS version is X10SRW7.410 (4/10/2017)
- Mount
smithibiosx10srw7410.isousing the instructions above - Boot to the Virtual CD
- Once you get a DOS prompt, run
flash.bat X10SRW7.410 - The first time, it'll put the system in “Flashing Mode” and you'll have to reboot and run this again for the actual BIOS update
- Shut the system OFF then back on for the BIOS update to complete
- Supermicro suggests restoring BIOS to default settings then setting back up although I've found some settings get reset automatically anyway (like boot order)
