We have a Red Hat Satellite server running on satellite.front.sepia.ceph.com which is a VM in RHEV.

It is configured to sync repos daily. This allows us to subscribe RHEL testnodes to the Satellite to do upstream Ceph testing on RHEL.

Web Login: https://satellite.front.sepia.ceph.com
Username: root
Password: Standard root password

Because we're using our Employee SKU with the Satellite, we have access to every Red Hat repo. So when you go to the Repositories page to enable repos, it can take an hour to load just the first 20 Available repos.

I found a way to do this on the CLI. It still takes a long time to compile the list of available products but it works!

Taken from: https://mariopang.blogspot.com/2014/12/satellite-60-subscribe-and-enable-repos.html

Here's how I added RHEL 7.8

# The foreman.yml has a parameter to prevent hammer timeouts (https://access.redhat.com/solutions/1274723)
hammer -c /etc/hammer/cli.modules.d/foreman.yml shell

# First get the list of available products
hammer> product list --organization-id 1

----|----------------------------------------------------------------------------------|-------------|--------------|--------------|------------------
ID  | NAME                                                                             | DESCRIPTION | ORGANIZATION | REPOSITORIES | SYNC STATE       
----|----------------------------------------------------------------------------------|-------------|--------------|--------------|------------------
234 | Red Hat Enterprise Linux Server                                                  |             | Ceph         | 19           | Syncing Complete.
58  | Red Hat Enterprise Linux High Availability for RHEL Server                       |             | Ceph         | 3            | Syncing Complete.


# Next get all the repositories available for that product (This will take a LONG time to return)
hammer> repository-set list --organization-id 1 --product-id 234
-----|-----------|---------------------------------------------------------------------------------
ID   | TYPE      | NAME                                                                            
-----|-----------|---------------------------------------------------------------------------------
2456 | yum       | Red Hat Enterprise Linux 7 Server (RPMs)                                        
2463 | yum       | Red Hat Enterprise Linux 7 Server - Optional (RPMs)                             
2476 | yum       | Red Hat Enterprise Linux 7 Server - Supplementary (RPMs)                        

hammer> repository-set list --organization-id 1 --product-id 58
2762 | yum  | Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs)           


# Next, enable the new version of RHEL for those repos
hammer> repository-set enable --id 2463 --product-id 234 --organization-id 1 --releasever 7.8 --basearch x86_64
Repository enabled.
hammer> repository-set enable --id 2456 --product-id 234 --organization-id 1 --releasever 7.8 --basearch x86_64
Repository enabled.
hammer> repository-set enable --id 2476 --product-id 234 --organization-id 1 --releasever 7.8 --basearch x86_64
Repository enabled.
hammer> repository-set enable --id 2762 --product-id 58 --organization-id 1 --releasever 7.8 --basearch x86_64
Repository enabled.

SHORT VERSION FOR RHEL7.9

for id in 2463 2456 2476; do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --id $id --product-id 234 --organization-id 1 --releasever 7.9 --basearch x86_64; done
hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --id 2762 --product-id 58 --organization-id 1 --releasever 7.9 --basearch x86_64

RHEL 8.2 Example:

hammer> repository-set list --organization-id 1 --product-id 249
-----|-----------|--------------------------------------------------------------------
ID   | TYPE      | NAME                                                               
-----|-----------|--------------------------------------------------------------------
7442 | yum       | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Debug RPMs)     
7446 | kickstart | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Kickstart)      
7441 | yum       | Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)           
7443 | yum       | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Source RPMs)    
7417 | yum       | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Debug RPMs)        
7419 | file      | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (ISOs)              
7421 | kickstart | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Kickstart)         
7416 | yum       | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)              
7420 | file      | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Source ISOs)       
7418 | yum       | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Source RPMs)       
7933 | yum       | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (Debug RPMs) 
7935 | file      | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (ISOs)       
7932 | yum       | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (RPMs)       
7934 | yum       | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (Source RPMs)
8694 | yum       | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (Debug RPMs)         
8693 | yum       | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (RPMs)               
8695 | yum       | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (Source RPMs)        
9000 | yum       | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (Debug RPMs)      
9001 | yum       | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (RPMs)            
9002 | yum       | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (Source RPMs)     
-----|-----------|--------------------------------------------------------------------

hammer> repository-set enable --organization-id 1 --id 7416 --product-id 249 --releasever 8.2 --basearch x86_64
Repository enabled.
hammer> repository-set enable --organization-id 1 --id 7441 --product-id 249 --releasever 8.2 --basearch x86_64
Repository enabled.

hammer> repository-set list --organization-id 1 --product-id 239
-----|------|----------------------------------------------------------------
ID   | TYPE | NAME                                                           
-----|------|----------------------------------------------------------------
7955 | yum  | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (Debug RPMs) 
7954 | yum  | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)       
7956 | yum  | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (Source RPMs)
-----|------|----------------------------------------------------------------
hammer> repository-set enable --organization-id 1 --id 7954 --product-id 239 --releasever 8.2 --basearch x86_64
Repository enabled.

SHORT VERSION FOR RHEL8.6

RELEASEVER=8.6; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 1 --id 7416 --product-id 249 --releasever $RELEASEVER --basearch x86_64; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 1 --id 7441 --product-id 249 --releasever $RELEASEVER --basearch x86_64; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 1 --id 7954 --product-id 239 --releasever $RELEASEVER --basearch x86_64

Now you can go in the Web UI and manually sync those repos.

This was observed after a bad reboot of the satellite VM once.

Symptoms:

• ceph-cm-ansible fails in jobs with:

  TASK [common : Register with subscription-manager.] ****************************
  
  2020-01-02T23:51:00.955 INFO:teuthology.task.ansible.out:fatal: [smithi197.front.sepia.ceph.com]: FAILED! => {"attempts": 5, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}

• Attempting to register a testnode manually results in: HTTP error (500 - Internal Server Error): Unable to register system, not all services available

OR

• HTTP error (500 - Internal Server Error): Required lock is already taken by other running tasks.
  • Saw this when ruby got killed by OOM. Check dmesg -T.

Resolution: ssh satellite.front.sepia.ceph.com “sudo katello-service restart”

mv -f /etc/rhsm/rhsm.conf.kat-backup /etc/rhsm/rhsm.conf
subscription-manager unregister
subscription-manager refresh
yum reinstall http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-latest.noarch.rpm

Satellite 6.4 and before didn't care if you reused a hostname and the UUID was never checked. This was introduced in 6.5.

An option to ignore this was added in 6.7.

[root@satellite ~]# grep uuid /etc/candlepin/candlepin.conf 
candlepin.use_system_uuid_for_matching=false

Turns out this doesn't work. So instead, I modified a file to throw a warning instead of an error if Candlepin detects a duplicate UUID.

# diff /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.0.32/app/services/katello/registration_manager.rb.orig /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.0.32/app/services/katello/registration_manager.rb
92c92,93
<         registration_error("The DMI UUID of this host (%{uuid}) matches other registered hosts: %{existing}", uuid: host_uuid, existing: joined_hostnames(hosts))
---
> #        registration_error("The DMI UUID of this host (%{uuid}) matches other registered hosts: %{existing}", uuid: host_uuid, existing: joined_hostnames(hosts))
>         Rails.logger.warn("The DMI UUID of this host (%s) matches another host." % host_name)

None of this actually ended up working so…