User Tools

Site Tools


Sidebar

General Lab Info (Mainly for Devs)

Hardware

Lab Infrastructure Services

Misc Admin Tasks
These are infrequently completed tasks that don't fit under any specific service

Production Services

OVH = OVH
RHEV = Sepia RHE instance
Baremetal = Host in Sepia lab

The Attic/Legacy Info

services:vpshosts

VPSHOSTS

Summary

VPSHOSTs are Miras we took out of the testpool to use as hypervisors for testing VMs.

See the vps_hosts section of the ansible inventory to see which miras are used as VPSHOSTs. Or query the lock database teuthology-lock --brief -a | grep VPS.

VPSHOSTs are managed using the vmhost ansible role in ceph-cm-ansible.

Virtual Machines

Each VPSHOST is home to 8 4 virtual machines. Each VM has its own JBOD disk assigned to it with 4GB RAM and 1 vCPU except the first VM on each VPSHOST. It runs off the root drive.

In June 2016, we marked down all the even-numbered VPSes and made a change to teuthology which creates the VMs with 4GB RAM as 2GB per VM wasn't meeting the needs. See http://tracker.ceph.com/issues/15052

Common Tasks

Updating users so new users can downburst VMs

ansible-playbook common.yml --limit vps_hosts

Setting up a VPSHOST from scratch

NOTE: This has been adapted to be applicable for 4 VPSes and disks per host.

If setting up a host as a new or replacement VPSHOST, be sure to update your libvirt config. See http://docs.ceph.com/teuthology/docs/downburst_vms.html#vps-hosts.

If you install the machine using a -stock cobbler profile, you'll need to run the common role as well.

apt-get install xfsprogs vim

# Determine the first and last VPS number by reading the VPSHOST's description in the lock db
# In this example, the VPSes that live on the VPSHOST are vpm177 thru vpm184

for sys in vpm{049,051,053,055}; do mkdir -p /srv/libvirtpool/$sys; done
for disk in sd{b..d}; do mkfs -t xfs -f /dev/$disk; done

## Not really needed if next task and `mount -s` succeeds
# $num should be second VPM
num=51; for disk in sd{b..d}; do mount /dev/$disk /srv/libvirtpool/vpm0$num; let num=num+2; done
# OR if VPM$num is >= 100,
num=101; for disk in sd{b..d}; do mount /dev/$disk /srv/libvirtpool/vpm$num; let num=num+2; done

# $num should be second VPM
num=51; for disk in sd{b..d}; do echo -e "UUID=$(blkid -s UUID -o value /dev/$disk)\t/srv/libvirtpool/vpm0$num\txfs\tdefaults,noatime,nodiratime,nobarrier,inode64,logbufs=8,logbsize=256k,largeio\t0\t0"; let num=num+2; done >> /etc/fstab
# OR if VPM$num is >= 100,
num=101; for disk in sd{b..d}; do echo -e "UUID=$(blkid -s UUID -o value /dev/$disk)\t/srv/libvirtpool/vpm$num\txfs\tdefaults,noatime,nodiratime,nobarrier,inode64,logbufs=8,logbsize=256k,largeio\t0\t0"; let num=num+2; done >> /etc/fstab

# Verify fstab, then
mount -a

# On your workstation,
ansible-playbook vmhost.yml --limit="mira###.front.sepia.ceph.com"

# Run this again to fix libvirtpool permissions
ansible-playbook vmhost.yml --limit="mira###.front.sepia.ceph.com"

# Make sure the first VPM is down
tl --update --status down vpm049

# Lock the first VPM on the host to download disk images
tl --lock --os-type ubuntu --os-version 14.04 ubuntu@vpm049
tl --unlock ubuntu@vpm049
tl --lock --os-type ubuntu --os-version 16.04 ubuntu@vpm049
tl --unlock ubuntu@vpm049
tl --lock --os-type centos --os-version 7.3 ubuntu@vpm049
tl --unlock ubuntu@vpm049

# Copy the disk image to the other libvirtpools
for dir in $(ls /srv/libvirtpool/ | tail -n 3); do cp /srv/libvirtpool/$(ls /srv/libvirtpool/ | head -n 1)/{ubuntu*,centos*} /srv/libvirtpool/$dir/; done

for pool in $(ls /srv/libvirtpool/); do virsh pool-refresh $pool; done

# Lock then unlock all the VPSes to verify everything looks good
for sys in vpm{049,051,053,055}; do tl --lock ubuntu@$sys; done
for sys in vpm{049,051,053,055}; do tl --unlock ubuntu@$sys; done

Replace bad VPSHOST disk

# Mark the VM down
teuthology-lock --update --status down vpm###

# On the VPSHOST,
umount $bad_disk
# Comment out bad disk in /etc/fstab

# Physically replace the disk

# Create a new filesystem on the new disk
mkfs -t xfs /dev/$new_disk

# Mount the new disk partition to the VM's libvirtpool mount
mount /dev/$new_disk /srv/libvirtpool/vpm###

# Obtain the new UUID
lsblk -o name,uuid,mountpoint

# Replace the old UUID in /etc/fstab

# Mark the VM back up
teuthology-lock --update --status up vpm###
services/vpshosts.txt · Last modified: 2017/08/15 22:46 by djgalloway