User Tools

Site Tools


Sidebar

General Lab Info (Mainly for Devs)

Hardware

Lab Infrastructure Services

Misc Admin Tasks
These are infrequently completed tasks that don't fit under any specific service

Production Services

OVH = OVH
RHEV = Sepia RHE instance
Baremetal = Host in Sepia lab

The Attic/Legacy Info

testnodereimage

Reimaging Testnodes

Summary

mira, smithi, and gibba are automatically reimaged by FOG when locked using teuthology-lock.

Prior to 2018, we didn't have FOG and testnodes would need to be reimaged every so often using cobbler when the OS would get too mangled to re-use. The dhcp server is configured to point smithi and mira to the FOG PXE server. All other machine types can use Cobbler.

Cobbler is configured using ceph-cm-ansible and primarily to install on /dev/sda on a machine with a single NIC. If a machine has a more complex configuration than that, the Manual method below is recommended.

set-next-server

For machines that typically use FOG, you will need to tell the DHCP server to point whatever machine you're reimaging to Cobbler's PXE server instead of FOG's. You can do this with set-next-server-local.sh. e.g.,

dgalloway@p50 ~ ()$ set-next-server-local.sh smithi071 cobbler
+ ssh store01.front.sepia.ceph.com 'sudo /usr/local/sbin/set-next-server.sh smithi071 cobbler && sudo service dhcpd restart'
+ dhcpconfig=/etc/dhcp/dhcpd.front.conf
++ date +%s
+ timestamp=1656427483
+ cobblerip=172.21.0.11
+ cobblerfilename=/pxelinux.0
+ fogip=172.21.0.72
+ fogfilename=/undionly.kpxe
++ sed -n '/host smithi071-front/,/}/p' /etc/dhcp/dhcpd.front.conf
++ grep 'hardware ethernet'
++ awk '{ print $3 }'
++ tr -d ';'
+ macaddr=0c:c4:7a:8f:d7:65
++ sed -n '/host smithi071-front/,/}/p' /etc/dhcp/dhcpd.front.conf
++ grep fixed-address
++ awk '{ print $2 }'
++ tr -d ';'
+ ipaddr=172.21.15.71
++ grep -n smithi071 /etc/dhcp/dhcpd.front.conf
++ cut -d : -f1
+ linenum=2111
+ '[' -z 0c:c4:7a:8f:d7:65 ']'
+ '[' -z 172.21.15.71 ']'
+ '[' -z 2111 ']'
+ cp /etc/dhcp/dhcpd.front.conf /etc/dhcp/dhcpd.front.conf_1656427483.bak
+ sed -i '/host smithi071-front {/,/}/d' /etc/dhcp/dhcpd.front.conf
+ '[' cobbler == cobbler ']'
+ sed -i '2111 i \  host smithi071-front {\n\    hardware ethernet 0c:c4:7a:8f:d7:65;\n\    fixed-address 172.21.15.71;\n\    next-server 172.21.0.11;\n\    filename "/pxelinux.0";\n\  }' /etc/dhcp/dhcpd.front.conf
+ dhcpd -q -t -cf /etc/dhcp/dhcpd.front.conf
+ '[' 0 '!=' 0 ']'
+ rm /etc/dhcp/dhcpd.front.conf_1656427483.bak
Redirecting to /bin/systemctl restart  dhcpd.service

Be sure to set it back to FOG when you're done!

Cobbler Web UI

If you don't already have a Cobbler Web UI account, request one by filing a ticket.

Single Machine (Manual [NOT USING KICKSTART])

  1. Click Systems on the left sidebar
  2. Locate the system you want to reimage
    1. If desired, increase the Items/page dropdown in the upper right corner
  3. Click the system's name under the Name header
  4. Go to the Networking tab
  5. Make a note of the current MAC address
  6. Change the MAC address to DE:AD:BE:EF:AF:AF or something else you know is bogus (this makes sure the PXE server doesn't recognize the machine you're reimaging after it reboots so that you get the Cobbler Menu)
  7. Save
  8. Open a Java Web Console by logging into the target machine's IPMI Web UI (target.ipmi.sepia.ceph.com)
  9. Reboot the machine either using the Java Web Console menu or ipmitool
  10. Watch the Java Web Console. You should eventually be presented with the Cobbler Menu which should look something like this
  11. Arrow down and highlight the OS you intend to install. Then press Tab
  12. Erase everything except the initrd= and vmlinuz= parameters.
    1. If you're installing CentOS or RHEL, you must also add inst.repo=http://172.21.0.11/cobbler/ks_mirror/DISTRO (e.g., inst.repo=http://172.21.0.11/cobbler/ks_mirror/CentOS-7.5-x86_64)
  13. Press Enter
  14. You should end up with a graphical installation
  15. When you're done, go back in the Networking settings for the machine and set the MAC address back.

Single Machine (Automated [USING KICKSTART])

  1. Click Systems on the left sidebar
  2. Locate the system you want to reimage
    1. If desired, increase the Items/page dropdown in the upper right corner
  3. Click the system's name under the Name header
  4. Set the Profile dropdown to your desired distro/profile
  5. Check Netboot Enabled
  6. Click Save
  7. Power cycle the testnode

Multiple Machines (Automated)

  1. Click Systems on the left sidebar
  2. Check the boxes next to the systems you want to reimage
  3. If desired, select Change profile under the Batch Actions drop-down at the top
    1. Enter the desired profile name
    2. Click OK
  4. Check the boxes next to the systems you want to reimage (again if you changed the Profile)
  5. Select Netboot Enabled under the Batch Actions drop-down at the top
  6. Click OK
  7. Power cycle the testnodes you wish to reimage

Cobbler CLI

You will need to have sudo access on the cobbler host. This is typically reserved for Infra Admins.

ssh cobbler.front.sepia.ceph.com

# To see a list of available Profiles/Distros,
sudo cobbler profile list

# To list the systems
sudo cobbler system list

# After you've decided on a profile,
sudo cobbler system edit --name $SHORT_HOSTNAME --profile=$PROFILE_NAME --netboot-enabled=1

# Example:
sudo cobbler system edit --name smithi001 --profile=CentOS-7.2-x86_64 --netboot-enabled=1

# Example for reimaging a large number of testnodes at once:
for host in smithi{001..030}; do sudo cobbler system edit --name $host --profile=CentOS-7.2-x86_64 --netboot-enabled=1; ipmitool -I lanplus -U inktank -P XXXXX -H $host.ipmi.sepia.ceph.com chassis power cycle; done

Now power cycle the testnode.

testnodereimage.txt · Last modified: 2022/06/28 14:53 by djgalloway