User Tools

Site Tools


testnodeaccess

Testnode Access

Summary

We have about 250 baremetal testnodes that get automatically reserved (locked) and unlocked by teuthology “workers.” Workers are daemons on the teuthology.front.sepia.ceph.com VM that are fed jobs via a beanstalk queue. This page will cover setting up your workstation to lock and unlock testnodes as well as schedule teuthology suites.

Teuthology Config

Most developers schedule suites and lock/unlock machines from the teuthology.front.sepia.ceph.com VM.

However, if you wish to run teuthology commands from your workstation, see http://docs.ceph.com/teuthology/docs/INSTALL.html#installation-and-setup.

Once you've got teuthology added to your workstation path, make sure you copy the current /etc/teuthology.yaml from teuthology.front.sepia.ceph.com to your local ~/.teuthology.yaml.

SSH Config

Baremetal testnodes get reprovisioned with an already-configured OS image including a home dir for your user account. Your ssh public key should be in your user's and the /home/ubuntu/.ssh/authorized_keys file. You should ssh as your username unless the testnode is a VPS.

Example

ssh -i ~/.ssh/id_rsa YOURUSER@testnode123.front.sepia.ceph.com

You should replace ~/.ssh/id_rsa with whatever private key corresponds with your private key listed here

To avoid interfering with other contributors' tests, you should refrain from logging into hosts that aren't locked by you via teuthology-lock.

If you run into any issues with a testnode that appear to be OS, network, or environment relatated (in other words: not ceph/test related), please file a ticket.

Unless you have a separate public/private key pair on the teuthology machine, you'll want to use SSH agent forwarding when SSH'ing to teuthology.front.sepia.ceph.com. You can do this either by using ssh -A USER@teuthology.front.sepia.ceph.com or adding the following to your workstation's ~/.ssh/config:

host teuthology.front.sepia.ceph.com
        User $YOURUSER
        IdentityFile ~/.ssh/id_rsa # (This should be the private key matching the public key you provided in your user access ticket)
        ForwardAgent yes # <- This is the important part

This will allow you to SSH from your workstation → teuthology machine → all testnodes

VPSes

VPS (Virtual Private Servers) are ephemeral KVM virtual machines that are spun up on demand using, for example:
teuthology-lock --lock-many 1 --machine-type vps --os-type ubuntu --os-version 14.04

More Information

Their hostnames are vpm{001..200}.front.sepia.ceph.com. A few mira are set aside as hypervisors. See vpshosts.

You must create an RSA SSH keypair (named id_rsa/id_rsa.pub) to ssh to VPSes from the teuthology host. We also recommend you have us add the public key to your pubkey file in the keys repo.

Note: You still need a keypair even when using SSH agent forwarding from your workstation.

The following ~/.ssh/config is required for you to lock VPSes from the teuthology host.

Host *
  StrictHostKeyChecking no

Host vpm*
  User ubuntu

Out-Of-Band Management

Further reading

IPMI

Baremetal testnodes are accessible via out-of-band (OOB) management controllers, or BMCs. If you're unable to reach a host via ssh on its front.sepia.ceph.com address, you can try accessing it using conserver and power cycle via ipmitool.

Power Cycle Example

ipmitool -I lanplus -U inktank -P XXXXX -H testnode123.ipmi.sepia.ceph.com chassis power cycle

Note the 'ipmi' vs 'front' in the FQDN

Other common IPMI power commands include:

chassis power on
chassis power off
chassis power status

Java Console

Some OOB controllers will also have a Java console you can access by opening the IPMI address in a browser. On most SuperMicro machines, this can be accessed at Remote Control > Console Redirection

Example

testnodeaccess.txt · Last modified: 2018/05/22 13:39 by djgalloway