Sepia Lab Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Satellite ======
===== Summary =====
We have a Red Hat Satellite server running on ''satellite.front.sepia.ceph.com'' which is a VM in [[services:RHEV]].

It is configured to sync repos daily.  This allows us to subscribe RHEL testnodes to the Satellite to do upstream Ceph testing on RHEL.

**Web Login**: https://satellite.front.sepia.ceph.com\\
**Username**: admin\\
**Password**: Standard root password

===== How-Tos =====
==== Adding new releases ====
Because we're using our Employee SKU with the Satellite, we have access to every Red Hat repo.  So when you go to the Repositories page to enable repos, it can take an hour to load just the first 20 Available repos.

I found a way to do this on the CLI.  It still takes a long time to compile the list of available products but it works!

Taken from: https://mariopang.blogspot.com/2014/12/satellite-60-subscribe-and-enable-repos.html

Here's how I added RHEL 7.8

<code>
# The foreman.yml has a parameter to prevent hammer timeouts (https://access.redhat.com/solutions/1274723)
hammer -c /etc/hammer/cli.modules.d/foreman.yml shell

# First get the list of available products
hammer> product list --organization-id 5

----|----------------------------------------------------------------------------------|-------------|--------------|--------------|------------------
ID  | NAME                                                                             | DESCRIPTION | ORGANIZATION | REPOSITORIES | SYNC STATE       
----|----------------------------------------------------------------------------------|-------------|--------------|--------------|------------------
712 | Red Hat Enterprise Linux Server                                                  |             | Ceph         | 19           | Syncing Complete.
746 | Red Hat Enterprise Linux High Availability for RHEL Server                       |             | Ceph         | 3            | Syncing Complete.


# Next get all the repositories available for that product (This will take a LONG time to return)
hammer> repository-set list --organization-id 5 --product-id 712
-----|-----------|---------------------------------------------------------------------------------
ID   | TYPE      | NAME                                                                            
-----|-----------|---------------------------------------------------------------------------------
2456 | yum       | Red Hat Enterprise Linux 7 Server (RPMs)                                        
2463 | yum       | Red Hat Enterprise Linux 7 Server - Optional (RPMs)                             
2476 | yum       | Red Hat Enterprise Linux 7 Server - Supplementary (RPMs)                        

hammer> repository-set list --organization-id 5 --product-id 746
2762 | yum  | Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs)           


# Next, enable the new version of RHEL for those repos
hammer> repository-set enable --id 2463 --product-id 712 --organization-id 5 --releasever 7.8 --basearch x86_64
Repository enabled.
hammer> repository-set enable --id 2456 --product-id 712 --organization-id 5 --releasever 7.8 --basearch x86_64
Repository enabled.
hammer> repository-set enable --id 2476 --product-id 712 --organization-id 5 --releasever 7.8 --basearch x86_64
Repository enabled.
hammer> repository-set enable --id 2762 --product-id 746 --organization-id 5 --releasever 7.8 --basearch x86_64
Repository enabled.
</code>

**SHORT VERSION FOR RHEL7.9**
<code>
RELEASEVER=7.9; for id in 2463 2456 2476; do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --id $id --product-id 712 --organization-id 5 --releasever $RELEASEVER --basearch x86_64; done
hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --id 2762 --product-id 746 --organization-id 5 --releasever $RELEASEVER --basearch x86_64
</code>

RHEL 8.2 Example:

<code>
hammer> repository-set list --organization-id 5 --product-id 608
-----|-----------|--------------------------------------------------------------------
ID   | TYPE      | NAME                                                               
-----|-----------|--------------------------------------------------------------------
7442 | yum       | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Debug RPMs)     
7446 | kickstart | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Kickstart)      
7441 | yum       | Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)           
7443 | yum       | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Source RPMs)    
7417 | yum       | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Debug RPMs)        
7419 | file      | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (ISOs)              
7421 | kickstart | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Kickstart)         
7416 | yum       | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)              
7420 | file      | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Source ISOs)       
7418 | yum       | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Source RPMs)       
7933 | yum       | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (Debug RPMs) 
7935 | file      | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (ISOs)       
7932 | yum       | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (RPMs)       
7934 | yum       | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (Source RPMs)
8694 | yum       | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (Debug RPMs)         
8693 | yum       | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (RPMs)               
8695 | yum       | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (Source RPMs)        
9000 | yum       | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (Debug RPMs)      
9001 | yum       | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (RPMs)            
9002 | yum       | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (Source RPMs)     
-----|-----------|--------------------------------------------------------------------

hammer> repository-set enable --organization-id 5 --id 7416 --product-id 608 --releasever 8.2 --basearch x86_64
Repository enabled.
hammer> repository-set enable --organization-id 5 --id 7441 --product-id 608 --releasever 8.2 --basearch x86_64
Repository enabled.

hammer> repository-set list --organization-id 5 --product-id 633
-----|------|----------------------------------------------------------------
ID   | TYPE | NAME                                                           
-----|------|----------------------------------------------------------------
7955 | yum  | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (Debug RPMs) 
7954 | yum  | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)       
7956 | yum  | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (Source RPMs)
-----|------|----------------------------------------------------------------
hammer> repository-set enable --organization-id 5 --id 7954 --product-id 633 --releasever 8.2 --basearch x86_64
Repository enabled.

</code>

**SHORT VERSION FOR RHEL8.6**
<code>
RELEASEVER=8.6; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 5 --id 7416 --product-id 608 --releasever $RELEASEVER; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 5 --id 7441 --product-id 608 --releasever $RELEASEVER; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 5 --id 7954 --product-id 633 --releasever $RELEASEVER
</code>

Now you can go in the Web UI and manually sync those repos.

==== Testnodes won't register ====
This was observed after a bad reboot of the satellite VM once.

**Symptoms:**
  * ceph-cm-ansible fails in jobs with:<code>
TASK [common : Register with subscription-manager.] ****************************

2020-01-02T23:51:00.955 INFO:teuthology.task.ansible.out:fatal: [smithi197.front.sepia.ceph.com]: FAILED! => {"attempts": 5, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}</code>

  * Attempting to register a testnode manually results in: ''HTTP error (500 - Internal Server Error): Unable to register system, not all services available''
OR
  * ''HTTP error (500 - Internal Server Error): Required lock is already taken by other running tasks.''
    * Saw this when ''ruby'' got killed by OOM.  Check ''dmesg -T''.

**Resolution:**
''ssh satellite.front.sepia.ceph.com "sudo katello-service restart"''

===== Errors and Fixes =====
==== Error: Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed ====
If RHEL clients are failing to perform yum/dnf transactions, check ''dmesg -T'' output on the Satellite server for OOM.

<code>
[root@smithi077 ~]# dnf -v update
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
DNF version: 4.7.0
cachedir: /var/cache/dnf
User-Agent: constructed: 'libdnf (Red Hat Enterprise Linux 8.6; generic; Linux.x86_64)'
repo: using cache for: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh
copr:copr.fedorainfracloud.org:ceph:python3-asyncssh: using metadata from Tue 27 Jul 2021 07:25:39 PM UTC.
repo: using cache for: epel
epel: using metadata from Tue 09 Aug 2022 02:13:36 AM UTC.
repo: using cache for: lab-extras
lab-extras: using metadata from Thu 28 Jan 2021 11:16:50 PM UTC.
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                                                                                                                        157 kB/s | 2.8 kB     00:00    
reviving: 'rhel-8-for-x86_64-appstream-rpms' can be revived - repomd matches.
rhel-8-for-x86_64-appstream-rpms: using metadata from Mon 08 Aug 2022 05:07:41 AM UTC.
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                                                                                                                            89 kB/s | 1.5 kB     00:00    
reviving: 'rhel-8-for-x86_64-baseos-rpms' can be revived - repomd matches.
loading repo 'rhel-8-for-x86_64-baseos-rpms' failure: loading of MD_TYPE_PRIMARY has failed.
Error: Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed
</code>

After you resolve the OOM issue, you will need to resync the repository.  If BaseOS is the one that was failing:
<code>
for id in $(hammer -c /etc/hammer/cli.modules.d/foreman.yml repository list | grep -i "baseos" | awk '{ print $1 }'); do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository synchronize --async --validate-contents true --id $id; done
</code>

==== Unable to verify server's identity: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:618) ====
<code>
mv -f /etc/rhsm/rhsm.conf.kat-backup /etc/rhsm/rhsm.conf
subscription-manager unregister
subscription-manager refresh
yum reinstall http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-latest.noarch.rpm
</code>

==== The DMI UUID of this host (00000000-0000-0000-0000-0CC47AD93748) matches other registered hosts ====
Satellite 6.4 and before didn't care if you reused a hostname and the UUID was never checked.  This was [[https://access.redhat.com/solutions/4396951|introduced in 6.5]].

An option to ignore this was [[https://bugzilla.redhat.com/show_bug.cgi?id=1772199|added in 6.7]].

<code>
[root@satellite ~]# grep uuid /etc/candlepin/candlepin.conf 
candlepin.use_system_uuid_for_matching=false
</code>

Turns out this doesn't work.  So instead, I modified a file to throw a warning instead of an error if Candlepin detects a duplicate UUID.
<code>
# diff /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.0.32/app/services/katello/registration_manager.rb.orig /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.0.32/app/services/katello/registration_manager.rb
92c92,93
<         registration_error("The DMI UUID of this host (%{uuid}) matches other registered hosts: %{existing}", uuid: host_uuid, existing: joined_hostnames(hosts))
---
> #        registration_error("The DMI UUID of this host (%{uuid}) matches other registered hosts: %{existing}", uuid: host_uuid, existing: joined_hostnames(hosts))
>         Rails.logger.warn("The DMI UUID of this host (%s) matches another host." % host_name)
</code>

None of this actually ended up working so...

====== Solved On SATELLITE 6.12 ======

Despite this, in Jan 2023 we had to install a new Satellite instance, and it's version 6.12; this was chosen for allowing continued support, but initially suffered from the same issue.  A support case, https://access.redhat.com/support/cases/#/case/03399123, was opened to deal with it and the solution was this - https://github.com/ceph/ceph-cm-ansible/pull/717
https://github.com/ceph/ceph-cm-ansible/pull/718
Sepia Lab Wiki

User Tools

Site Tools

Page Tools
