User Tools

Site Tools


Sidebar

General Lab Info (Mainly for Devs)

Hardware

Lab Infrastructure Services

Misc Admin Tasks
These are infrequently completed tasks that don't fit under any specific service

Production Services

OVH = OVH
RHEV = Sepia RHE instance
Baremetal = Host in Sepia lab

The Attic/Legacy Info

services:satellite

Satellite

Summary

We have a Red Hat Satellite server running on satellite.front.sepia.ceph.com which is a VM in RHEV.

It is configured to sync repos daily. This allows us to subscribe RHEL testnodes to the Satellite to do upstream Ceph testing on RHEL.

Web Login: https://satellite.front.sepia.ceph.com
Username: admin
Password: Standard root password

How-Tos

Adding new releases

Because we're using our Employee SKU with the Satellite, we have access to every Red Hat repo. So when you go to the Repositories page to enable repos, it can take an hour to load just the first 20 Available repos.

I found a way to do this on the CLI. It still takes a long time to compile the list of available products but it works!

Taken from: https://mariopang.blogspot.com/2014/12/satellite-60-subscribe-and-enable-repos.html

Here's how I added RHEL 7.8

# The foreman.yml has a parameter to prevent hammer timeouts (https://access.redhat.com/solutions/1274723)
hammer -c /etc/hammer/cli.modules.d/foreman.yml shell

# First get the list of available products
hammer> product list --organization-id 5

----|----------------------------------------------------------------------------------|-------------|--------------|--------------|------------------
ID  | NAME                                                                             | DESCRIPTION | ORGANIZATION | REPOSITORIES | SYNC STATE       
----|----------------------------------------------------------------------------------|-------------|--------------|--------------|------------------
712 | Red Hat Enterprise Linux Server                                                  |             | Ceph         | 19           | Syncing Complete.
746 | Red Hat Enterprise Linux High Availability for RHEL Server                       |             | Ceph         | 3            | Syncing Complete.


# Next get all the repositories available for that product (This will take a LONG time to return)
hammer> repository-set list --organization-id 5 --product-id 712
-----|-----------|---------------------------------------------------------------------------------
ID   | TYPE      | NAME                                                                            
-----|-----------|---------------------------------------------------------------------------------
2456 | yum       | Red Hat Enterprise Linux 7 Server (RPMs)                                        
2463 | yum       | Red Hat Enterprise Linux 7 Server - Optional (RPMs)                             
2476 | yum       | Red Hat Enterprise Linux 7 Server - Supplementary (RPMs)                        

hammer> repository-set list --organization-id 5 --product-id 746
2762 | yum  | Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs)           


# Next, enable the new version of RHEL for those repos
hammer> repository-set enable --id 2463 --product-id 712 --organization-id 5 --releasever 7.8 --basearch x86_64
Repository enabled.
hammer> repository-set enable --id 2456 --product-id 712 --organization-id 5 --releasever 7.8 --basearch x86_64
Repository enabled.
hammer> repository-set enable --id 2476 --product-id 712 --organization-id 5 --releasever 7.8 --basearch x86_64
Repository enabled.
hammer> repository-set enable --id 2762 --product-id 746 --organization-id 5 --releasever 7.8 --basearch x86_64
Repository enabled.

SHORT VERSION FOR RHEL7.9

RELEASEVER=7.9; for id in 2463 2456 2476; do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --id $id --product-id 712 --organization-id 5 --releasever $RELEASEVER --basearch x86_64; done
hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --id 2762 --product-id 746 --organization-id 5 --releasever $RELEASEVER --basearch x86_64

RHEL 8.2 Example:

hammer> repository-set list --organization-id 5 --product-id 608
-----|-----------|--------------------------------------------------------------------
ID   | TYPE      | NAME                                                               
-----|-----------|--------------------------------------------------------------------
7442 | yum       | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Debug RPMs)     
7446 | kickstart | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Kickstart)      
7441 | yum       | Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)           
7443 | yum       | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Source RPMs)    
7417 | yum       | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Debug RPMs)        
7419 | file      | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (ISOs)              
7421 | kickstart | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Kickstart)         
7416 | yum       | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)              
7420 | file      | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Source ISOs)       
7418 | yum       | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Source RPMs)       
7933 | yum       | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (Debug RPMs) 
7935 | file      | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (ISOs)       
7932 | yum       | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (RPMs)       
7934 | yum       | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (Source RPMs)
8694 | yum       | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (Debug RPMs)         
8693 | yum       | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (RPMs)               
8695 | yum       | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (Source RPMs)        
9000 | yum       | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (Debug RPMs)      
9001 | yum       | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (RPMs)            
9002 | yum       | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (Source RPMs)     
-----|-----------|--------------------------------------------------------------------

hammer> repository-set enable --organization-id 5 --id 7416 --product-id 608 --releasever 8.2 --basearch x86_64
Repository enabled.
hammer> repository-set enable --organization-id 5 --id 7441 --product-id 608 --releasever 8.2 --basearch x86_64
Repository enabled.

hammer> repository-set list --organization-id 5 --product-id 633
-----|------|----------------------------------------------------------------
ID   | TYPE | NAME                                                           
-----|------|----------------------------------------------------------------
7955 | yum  | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (Debug RPMs) 
7954 | yum  | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)       
7956 | yum  | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (Source RPMs)
-----|------|----------------------------------------------------------------
hammer> repository-set enable --organization-id 5 --id 7954 --product-id 633 --releasever 8.2 --basearch x86_64
Repository enabled.

SHORT VERSION FOR RHEL8.6

RELEASEVER=8.6; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 5 --id 7416 --product-id 608 --releasever $RELEASEVER; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 5 --id 7441 --product-id 608 --releasever $RELEASEVER; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 5 --id 7954 --product-id 633 --releasever $RELEASEVER

Now you can go in the Web UI and manually sync those repos.

Testnodes won't register

This was observed after a bad reboot of the satellite VM once.

Symptoms:

  • ceph-cm-ansible fails in jobs with:
    TASK [common : Register with subscription-manager.] ****************************
    
    2020-01-02T23:51:00.955 INFO:teuthology.task.ansible.out:fatal: [smithi197.front.sepia.ceph.com]: FAILED! => {"attempts": 5, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}
  • Attempting to register a testnode manually results in: HTTP error (500 - Internal Server Error): Unable to register system, not all services available

OR

  • HTTP error (500 - Internal Server Error): Required lock is already taken by other running tasks.
    • Saw this when ruby got killed by OOM. Check dmesg -T.

Resolution: ssh satellite.front.sepia.ceph.com “sudo katello-service restart”

Errors and Fixes

Error: Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed

If RHEL clients are failing to perform yum/dnf transactions, check dmesg -T output on the Satellite server for OOM.

[root@smithi077 ~]# dnf -v update
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
DNF version: 4.7.0
cachedir: /var/cache/dnf
User-Agent: constructed: 'libdnf (Red Hat Enterprise Linux 8.6; generic; Linux.x86_64)'
repo: using cache for: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh
copr:copr.fedorainfracloud.org:ceph:python3-asyncssh: using metadata from Tue 27 Jul 2021 07:25:39 PM UTC.
repo: using cache for: epel
epel: using metadata from Tue 09 Aug 2022 02:13:36 AM UTC.
repo: using cache for: lab-extras
lab-extras: using metadata from Thu 28 Jan 2021 11:16:50 PM UTC.
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                                                                                                                        157 kB/s | 2.8 kB     00:00    
reviving: 'rhel-8-for-x86_64-appstream-rpms' can be revived - repomd matches.
rhel-8-for-x86_64-appstream-rpms: using metadata from Mon 08 Aug 2022 05:07:41 AM UTC.
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                                                                                                                            89 kB/s | 1.5 kB     00:00    
reviving: 'rhel-8-for-x86_64-baseos-rpms' can be revived - repomd matches.
loading repo 'rhel-8-for-x86_64-baseos-rpms' failure: loading of MD_TYPE_PRIMARY has failed.
Error: Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed

After you resolve the OOM issue, you will need to resync the repository. If BaseOS is the one that was failing:

for id in $(hammer -c /etc/hammer/cli.modules.d/foreman.yml repository list | grep -i "baseos" | awk '{ print $1 }'); do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository synchronize --async --validate-contents true --id $id; done

Unable to verify server's identity: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:618)

mv -f /etc/rhsm/rhsm.conf.kat-backup /etc/rhsm/rhsm.conf
subscription-manager unregister
subscription-manager refresh
yum reinstall http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-latest.noarch.rpm

The DMI UUID of this host (00000000-0000-0000-0000-0CC47AD93748) matches other registered hosts

Satellite 6.4 and before didn't care if you reused a hostname and the UUID was never checked. This was introduced in 6.5.

An option to ignore this was added in 6.7.

[root@satellite ~]# grep uuid /etc/candlepin/candlepin.conf 
candlepin.use_system_uuid_for_matching=false

Turns out this doesn't work. So instead, I modified a file to throw a warning instead of an error if Candlepin detects a duplicate UUID.

# diff /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.0.32/app/services/katello/registration_manager.rb.orig /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.0.32/app/services/katello/registration_manager.rb
92c92,93
<         registration_error("The DMI UUID of this host (%{uuid}) matches other registered hosts: %{existing}", uuid: host_uuid, existing: joined_hostnames(hosts))
---
> #        registration_error("The DMI UUID of this host (%{uuid}) matches other registered hosts: %{existing}", uuid: host_uuid, existing: joined_hostnames(hosts))
>         Rails.logger.warn("The DMI UUID of this host (%s) matches another host." % host_name)

None of this actually ended up working so…

Solved On SATELLITE 6.12

Despite this, in Jan 2023 we had to install a new Satellite instance, and it's version 6.12; this was chosen for allowing continued support, but initially suffered from the same issue. A support case, https://access.redhat.com/support/cases/#/case/03399123, was opened to deal with it and the solution was this - https://github.com/ceph/ceph-cm-ansible/pull/717 https://github.com/ceph/ceph-cm-ansible/pull/718

services/satellite.txt · Last modified: 2023/11/30 19:57 by dmick