Differences

This shows you the differences between two versions of the page.

--- testnodeaccess [2019/06/10 17:20] – djgalloway
+++ testnodeaccess [2026/09/14 14:25] (current) – djgalloway
@@ Line 1: / Line 1: @@
 ====== Testnode Access ======
 ===== Summary =====
-We have about 250 baremetal testnodes that get automatically reserved (locked) and unlocked by teuthology "workers."  Workers are daemons on the ''teuthology.front.sepia.ceph.com'' VM that are fed jobs via a beanstalk queue.  This page will cover setting up your workstation to lock and unlock testnodes as well as schedule teuthology suites.
+We have about 400 baremetal testnodes that get automatically reserved (locked) and unlocked by teuthology "workers" or "dispatchers."  Workers are daemons on the ''teuthology.front.sepia.ceph.com'' host that are fed jobs via a beanstalk queue.  This page will cover setting up your workstation to lock and unlock testnodes as well as schedule teuthology suites.
+Baremetal testnodes get OSes automatically "installed" using [[services:FOG]] when you lock them either via ''teuthology-lock'' or ''teuthology-suite''.
 ===== Teuthology Config =====
-Most developers schedule suites and lock/unlock machines from the ''teuthology.front.sepia.ceph.com'' VM.
+Most developers schedule suites from the ''teuthology.front.sepia.ceph.com'' host which automatically locks/unlocks machines.
-However, if you wish to run ''teuthology'' commands from your workstation, see http://docs.ceph.com/teuthology/docs/INSTALL.html#installation-and-setup.
+However, if you wish to run ''teuthology'' commands from your workstation, see https://docs.ceph.com/projects/teuthology/en/latest/INSTALL.html#installation-and-setup.
-Once you've got ''teuthology'' added to your workstation path, make sure you copy the current ''/etc/teuthology.yaml'' from ''teuthology.front.sepia.ceph.com'' to your **local** ''~/.teuthology.yaml''.
+Once you've got ''teuthology'' added to your workstation path, make sure you copy the current ''/etc/teuthology.yaml'' from ''teuthology.front.sepia.ceph.com'' to your **local** workstation's ''~/.teuthology.yaml''.
 ===== SSH Config =====
-Baremetal testnodes get reprovisioned with an already-configured OS image including a home dir for your user account.  Your ssh public key should be in your user's **and** the ''/home/ubuntu/.ssh/authorized_keys'' file.  You should ssh as your username unless the testnode is a VPS.
+Baremetal testnodes get reprovisioned with an already-configured OS image including a home dir for your user account.  Your SSH public key should be in your user's **and** the ''/home/ubuntu/.ssh/authorized_keys'' file on each testnode.  You should always SSH as your username.
 **Example**
@@ Line 24: / Line 26: @@
 <code>
-host teuthology.front.sepia.ceph.com
+host teuthology.front.sepia.ceph.com soko04.front.sepia.ceph.com
         User $YOURUSER
         IdentityFile ~/.ssh/id_rsa # (This should be the private key matching the public key you provided in your user access ticket)
         ForwardAgent yes # <- This is the important part
+host smithi* gibba* trial*
+        StrictHostKeyChecking no
+        UserKnownHostsFile=/dev/null
 </code>
 This will allow you to SSH from your workstation -> teuthology machine -> all testnodes
-===== VPSes =====
+Your SSH config on ''teuthology.front.sepia.ceph.com'' should have this:
-VPS (Virtual Private Servers) are ephemeral KVM virtual machines that are spun up on demand using, for example:
-\\ ''%%teuthology-lock --lock-many 1 --machine-type vps --os-type ubuntu --os-version 14.04%%''
-**More Information**
-  * http://docs.ceph.com/teuthology/docs/downburst_vms.html
-  * https://github.com/ceph/downburst
-Their hostnames are ''vpm{001..200}.front.sepia.ceph.com''.  A few [[hardware:mira]] are set aside as hypervisors.  See [[services:vpshosts]].
-You must create an RSA SSH keypair (named id_rsa/id_rsa.pub) to ssh to VPSes from the teuthology host.  We also recommend you have us add the public key to your pubkey file in the [[https://github.com/ceph/keys|keys repo]].
-//Note: You still need a keypair even when using SSH agent forwarding from your workstation.//
-The following ''~/.ssh/config'' is required for you to lock VPSes from the teuthology host.
 <code>
 Host *
   StrictHostKeyChecking no
+  UserKnownHostsFile=/dev/null
+</code>
-Host vpm*
+===== Troubleshooting =====
-  User ubuntu
+==== SSHException: Error reading SSH protocol banner ====
+<code>
+teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds
 </code>
+  - If you're using a static key file (as in you have a ''~/.ssh/id_rsa'' file) on ''teuthology.front.sepia.ceph.com'', make sure its permissions are ''0600''
+  - The SSH key can NOT have a passphrase (unless you're doing SSH Agent Forwarding?)
+  - The SSH key can NOT have been generated using OpenSSH version >= 7.8p1-1 (''[dpkg -l|rpm -qa] | grep openssh'' to find out)
+    - Either generate your SSH key from ''teuthology.front.sepia.ceph.com'' or try ''ssh-keygen -t rsa -m PEM''
+  - ''rm -f ~/.ssh/known_hosts'' and add ''UserKnownHostsFile /dev/null'' to your SSH config.
+  - Ask Adam Kraitman or Dan Mick to capture new FOG images that include your public SSH key.
+  - You **must** have ''ForwardAgent yes'' set for ''teuthology.front.sepia.ceph.com'' in your workstation's ''~/.ssh/config'' file.
+The newest version of paramiko doesn't support SSH keys that have ''BEGIN OPENSSH PRIVATE KEY'' in them.  See https://github.com/paramiko/paramiko/issues/1015.
 -----
@@ Line 66: / Line 72: @@
 Baremetal testnodes are accessible via out-of-band (OOB) management controllers, or BMCs.  If you're unable to reach a host via ssh on its front.sepia.ceph.com address, you can try accessing it using [[services:conserver]] and power cycle via ''ipmitool''.
-**Power Cycle Example**
+==== Power Cycle Example ====
 <code>
 ipmitool -I lanplus -U inktank -P XXXXX -H testnode123.ipmi.sepia.ceph.com chassis power cycle