User Tools

Site Tools


testnodeaccess

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
testnodeaccess [2018/05/22 13:39]
djgalloway
testnodeaccess [2024/03/01 19:41] (current)
zmc
Line 2: Line 2:
 ===== Summary ===== ===== Summary =====
 We have about 250 baremetal testnodes that get automatically reserved (locked) and unlocked by teuthology "​workers." ​ Workers are daemons on the ''​teuthology.front.sepia.ceph.com''​ VM that are fed jobs via a beanstalk queue. ​ This page will cover setting up your workstation to lock and unlock testnodes as well as schedule teuthology suites. We have about 250 baremetal testnodes that get automatically reserved (locked) and unlocked by teuthology "​workers." ​ Workers are daemons on the ''​teuthology.front.sepia.ceph.com''​ VM that are fed jobs via a beanstalk queue. ​ This page will cover setting up your workstation to lock and unlock testnodes as well as schedule teuthology suites.
 +
 +Baremetal testnodes get OSes automatically "​installed"​ using [[services:​FOG]] when you lock them either via ''​teuthology-lock''​ or ''​teuthology-suite''​.
  
 ===== Teuthology Config ===== ===== Teuthology Config =====
-Most developers schedule suites ​and lock/unlock machines ​from the ''​teuthology.front.sepia.ceph.com''​ VM.+Most developers schedule suites from the ''​teuthology.front.sepia.ceph.com''​ VM which automatically locks/​unlocks machines.
  
 However, if you wish to run ''​teuthology''​ commands from your workstation,​ see http://​docs.ceph.com/​teuthology/​docs/​INSTALL.html#​installation-and-setup. However, if you wish to run ''​teuthology''​ commands from your workstation,​ see http://​docs.ceph.com/​teuthology/​docs/​INSTALL.html#​installation-and-setup.
  
-Once you've got ''​teuthology''​ added to your workstation path, make sure you copy the current ''/​etc/​teuthology.yaml''​ from ''​teuthology.front.sepia.ceph.com''​ to your **local** ''​~/​.teuthology.yaml''​.+Once you've got ''​teuthology''​ added to your workstation path, make sure you copy the current ''/​etc/​teuthology.yaml''​ from ''​teuthology.front.sepia.ceph.com''​ to your **local** ​workstation'​s ​''​~/​.teuthology.yaml''​.
  
 ===== SSH Config ===== ===== SSH Config =====
-Baremetal testnodes get reprovisioned with an already-configured OS image including a home dir for your user account. ​ Your ssh public key should be in your user's **and** the ''/​home/​ubuntu/​.ssh/​authorized_keys''​ file.  You should ​ssh as your username ​unless the testnode is a VPS.+Baremetal testnodes get reprovisioned with an already-configured OS image including a home dir for your user account. ​ Your ssh public key should be in your user's **and** the ''/​home/​ubuntu/​.ssh/​authorized_keys''​ file on each testnode.  You should ​always SSH as your username.
  
 **Example** **Example**
Line 17: Line 19:
 //You should replace// ''​~/​.ssh/​id_rsa''​ //with whatever private key corresponds with your private key listed [[https://​github.com/​ceph/​keys/​tree/​master/​ssh|here]]//​ //You should replace// ''​~/​.ssh/​id_rsa''​ //with whatever private key corresponds with your private key listed [[https://​github.com/​ceph/​keys/​tree/​master/​ssh|here]]//​
  
-To avoid interfering with other contributors'​ tests, you should refrain from logging into hosts that aren't locked by you via teuthology-lock.+To avoid interfering with other contributors'​ tests, you should refrain from logging into hosts that aren't locked by you via ''​teuthology-lock''​.
  
 If you run into any issues with a testnode that appear to be OS, network, or environment relatated (in other words: not ceph/test related), please [[http://​tracker.ceph.com/​projects/​teuthology/​issues/​new|file a ticket]]. If you run into any issues with a testnode that appear to be OS, network, or environment relatated (in other words: not ceph/test related), please [[http://​tracker.ceph.com/​projects/​teuthology/​issues/​new|file a ticket]].
Line 28: Line 30:
         IdentityFile ~/​.ssh/​id_rsa # (This should be the private key matching the public key you provided in your user access ticket)         IdentityFile ~/​.ssh/​id_rsa # (This should be the private key matching the public key you provided in your user access ticket)
         ForwardAgent yes # <- This is the important part         ForwardAgent yes # <- This is the important part
 +
 +host smithi* mira* gibba*
 +        StrictHostKeyChecking no
 +        UserKnownHostsFile=/​dev/​null
 </​code>​ </​code>​
  
 This will allow you to SSH from your workstation -> teuthology machine -> all testnodes This will allow you to SSH from your workstation -> teuthology machine -> all testnodes
 +
 +Your SSH config on ''​teuthology.front.sepia.ceph.com''​ should have this:
 +
 +<​code>​
 +Host *
 +  StrictHostKeyChecking no
 +  UserKnownHostsFile=/​dev/​null
 +</​code>​
  
 ===== VPSes ===== ===== VPSes =====
Line 51: Line 65:
 Host * Host *
   StrictHostKeyChecking no   StrictHostKeyChecking no
 +  UserKnownHostsFile /dev/null
  
 Host vpm* Host vpm*
   User ubuntu   User ubuntu
 </​code>​ </​code>​
 +
 +===== Troubleshooting =====
 +==== SSHException:​ Error reading SSH protocol banner ====
 +<​code>​
 +teuthology.exceptions.MaxWhileTries:​ reached maximum tries (100) after waiting for 600 seconds
 +</​code>​
 +
 +  - If you're using a static key file (as in you have a ''​~/​.ssh/​id_rsa''​ file) on ''​teuthology.front.sepia.ceph.com'',​ make sure its permissions are ''​0600''​
 +  - The SSH key can NOT have a passphrase (unless you're doing SSH Agent Forwarding?​)
 +  - The SSH key can NOT have been generated using OpenSSH version >= 7.8p1-1 (''​[dpkg -l|rpm -qa] | grep openssh''​ to find out)
 +    - Either generate your SSH key from ''​teuthology.front.sepia.ceph.com''​ or try ''​ssh-keygen -t rsa -m PEM''​
 +  - ''​rm -f ~/​.ssh/​known_hosts''​ and add ''​UserKnownHostsFile /​dev/​null''​ to your SSH config.
 +  - Ask Adam Kraitman or Dan Mick to capture new FOG images that include your public SSH key.
 +  - You **must** have ''​ForwardAgent yes''​ set for ''​teuthology.front.sepia.ceph.com''​ in your workstation'​s ''​~/​.ssh/​config''​ file.
 +
 +The newest version of paramiko doesn'​t support SSH keys that have ''​BEGIN OPENSSH PRIVATE KEY''​ in them.  See https://​github.com/​paramiko/​paramiko/​issues/​1015.
  
 ----- -----
Line 66: Line 97:
 Baremetal testnodes are accessible via out-of-band (OOB) management controllers,​ or BMCs.  If you're unable to reach a host via ssh on its front.sepia.ceph.com address, you can try accessing it using [[services:​conserver]] and power cycle via ''​ipmitool''​. Baremetal testnodes are accessible via out-of-band (OOB) management controllers,​ or BMCs.  If you're unable to reach a host via ssh on its front.sepia.ceph.com address, you can try accessing it using [[services:​conserver]] and power cycle via ''​ipmitool''​.
  
-**Power Cycle Example**+==== Power Cycle Example ​====
 <​code>​ <​code>​
 ipmitool -I lanplus -U inktank -P XXXXX -H testnode123.ipmi.sepia.ceph.com chassis power cycle ipmitool -I lanplus -U inktank -P XXXXX -H testnode123.ipmi.sepia.ceph.com chassis power cycle
testnodeaccess.1526996396.txt.gz ยท Last modified: 2018/05/22 13:39 by djgalloway