hardware:braggi
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
|
hardware:braggi [2020/03/07 01:23] djgalloway created |
hardware:braggi [2022/08/25 21:08] (current) djgalloway [Setting up as Jenkins slaves] |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== braggi{01..22} ====== | ====== braggi{01..22} ====== | ||
| - | https://opendcim.engineering.redhat.com/cabnavigator.php?cabinetid=504 | + | https://opendcim.engineering.redhat.com/cabnavigator.php?cabinetid=501 |
| ===== Summary ===== | ===== Summary ===== | ||
| - | We have 8 irvingi systems. All are connected to [[services:RHEV]] and each one runs an Ubuntu and CentOS [[jenkins.ceph.com|Jenkins]] slave. | + | We have 22 braggi systems. The Ceph Foundation graciously purchased these late 2019/early 2020 to help reduce our reliance on ephemeral Jenkins slaves in OVH's Openstack public cloud. |
| ===== Hardware Specs ===== | ===== Hardware Specs ===== | ||
| - | | ^ Count ^ Manufacturer ^ Model ^ Capacity ^ Notes ^ | + | | ^ Count ^ Manufacturer ^ Model ^ Capacity ^ Notes ^ |
| - | ^ Chassis | N/A | Supermicro | SYS-1028R-TDW | N/A | | | + | ^ Chassis | N/A | Supermicro | SYS-1029P-MTR | N/A | | |
| - | ^ Mainboard | N/A | Supermicro | X10DDW-i | N/A | | | + | ^ Mainboard | N/A | Supermicro | X11DPL-i | N/A | | |
| - | ^ CPU | 2 | Intel | Xeon(R) CPU E5-2620 v3 @ 2.40GHz | N/A | [[http://ark.intel.com/products/83352/Intel-Xeon-Processor-E5-2620-v3-15M-Cache-2_40-GHz|ARK]] | | + | ^ CPU | 2 | Intel | Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz | 48 cores total | [[https://ark.intel.com/content/www/us/en/ark/products/193385/intel-xeon-silver-4214-processor-16-5m-cache-2-20-ghz.html|ARK]] | |
| - | ^ RAM | 4 DIMMs | Samsung | M393A2G40DB0-CPB | 16GB | 64GB total | | + | ^ RAM | 8 DIMMs | Micron Technology | 36ASF4G72PZ-2G6E1 | 32GB | 256GB total | |
| - | ^ HDD | 0 | | | | | | + | ^ SSD | 1 | Intel | MZ7KH480HAHQ-00005 | 480GB | | |
| - | ^ SSD | 2 | Intel | SSDSC2BB240G4 | 240GB | | | + | ^ SSD | 1 | Intel | MZ7LH960HAJR-00005 | 960GB | Mounted at /home/jenkins-build | |
| - | ^ SSD | 1 | Samsung | EVO 850 | 1TB | Mounted at /vms and configured as local storage in RHEV | | + | ^ NIC | 2 ports | Intel | ? | 1Gb | Unused | |
| - | ^ NIC | 2 ports | Intel | I350 | 1Gb | On-board | | + | ^ NIC | 2 ports | Intel | 82599ES | 10Gb | Port 1 cabled as uplink | |
| - | The irvingi have 8x 2.5" drive bays per chassis. | + | ===== Setting up as Jenkins slaves ===== |
| - | ===== BIOS Update ===== | + | ==== sda race condition === |
| - | **WARNING:** I had to hand modify/rename the ''/etc/sysconfig/network-scripts/ifcfg'' files after this BIOS update. udev changed the name from enp11s0f0 to enp8s0f0. I also lost access to the BMC Java KVM so make sure you can get SOL access to the machine/OS. | + | There is an incredibly annoying race condition when installing CentOS 9 Stream on these nodes where sda and sdb get flip flopped. dgalloway tried fixing this with https://github.com/ceph/ceph-cm-ansible/pull/703/commits/1058a3d0ca85f00a0f7a2df416c4a22a2e08883c which did help but not entirely. |
| - | The Supermicro X10 series BIOS can only be updated in a UEFI shell, Windows, or DOS prompt. I roughly followed [[https://siliconmechanics.zendesk.com/hc/en-us/articles/201466115-Creating-FreeDOS-bootable-ISO|this guide]] to create a bootable DOS image with the BIOS embedded. Here's a brief rundown. | + | You can check if anaconda picked the right disk by pulling up the Java KVM, switching to another terminal (Ctrl+Alt+F2) and checking if ''/mnt/sysroot'' is on the 800GB or 400GB drive. You want it to be on the 400GB drive. If it's not, just reboot and try again. |
| - | <code> | + | ==== How To ==== |
| - | # Download and unzip the BIOS zip file somewhere... then: | + | - Reimage using Cobbler |
| - | + | - Once the OS install is done, | |
| - | wget http://www.fdos.org/bootdisks/ISO/FDOEMCD.builder.zip | + | - Run the ceph-cm-ansible ''users'' role |
| - | unzip FDOEMCD.builder.zip | + | - Run ''%%ansible-playbook common.yml --tags vars,nagios,monitoring-scripts --limit braggi##%%'' |
| - | cp -r /path/to/BIOS/ /path/to/FDOEMCD/CDROOT/ | + | - Run https://github.com/djgalloway/sepia/blob/master/ansible-playbooks/jenkins-build-home.yml |
| - | cd /path/to/FDOEMCD/CDROOT/BIOS/DOS/ | + | - Run https://github.com/ceph/ceph-build/blob/master/ansible/examples/slave.yml |
| - | mv AFUDOSU.SMC AFUDOS.EXE | + | - ''%%ansible-playbook -M ./library/ -e permanent=True slave.yml --extra-vars '{"labels": "amd64 XXXXX huge sepia x86_64 braggi gigantic"}' --extra-vars '{"token": "XXXXX", "jenkins_credentials_uuid": "jenkins-build", "api_uri": "https://jenkins.ceph.com"}' --limit braggiXX*%%'' |
| - | mv CHOICE.SMC CHOICE.EXE | + | |
| - | mv FDT.smc FDT.EXE | + | |
| - | cd /path/to/FDOEMCD | + | |
| - | mkisofs -o irvingibiosupdate.iso -p "Silicon Mechanics Support" -publisher "www.siliconmechanics.com" -V "SiMech OEM CD" -b isolinux/isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table -N -J -r -c boot.catalog -hide boot.catalog -hide-joliet boot.catalog CDROOT | + | |
| - | </code> | + | |
| - | + | ||
| - | I then copied irvingibiosupdate.iso to cobbler.front.sepia.ceph.com:/samba/anonymous and mounted it via the irvingi BMC Web UI and booted to it. | + | |
| - | + | ||
| - | * **Share Host:** ''172.21.0.11'' | + | |
| - | * **Path:** ''\Anonymous\irvingibiosupdate.iso'' | + | |
| - | + | ||
| - | Run ''cd X10DDW46\DOS'', ''flash.bat $FW_BIN_FILE''. The system will check if a firmware update is needed and put the system board in [Flash Mode] if needed. The system will reboot. Boot back into FreeDOS and run the flash.bat script again. | + | |
| - | + | ||
| - | Basically, read the README and follow prompts. | + | |
| - | + | ||
| - | {{ :hardware:screenshot_at_2016-11-30_15-03-56.png?nolink |}} | + | |
hardware/braggi.1583544235.txt.gz · Last modified: 2020/03/07 01:23 by djgalloway
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
