====== Satellite ====== ===== Summary ===== We have a Red Hat Satellite server running on ''satellite.front.sepia.ceph.com'' which is a VM in [[services:RHEV]]. It is configured to sync repos daily. This allows us to subscribe RHEL testnodes to the Satellite to do upstream Ceph testing on RHEL. **Web Login**: https://satellite.front.sepia.ceph.com\\ **Username**: admin\\ **Password**: Standard root password ===== How-Tos ===== ==== Adding new releases ==== Because we're using our Employee SKU with the Satellite, we have access to every Red Hat repo. So when you go to the Repositories page to enable repos, it can take an hour to load just the first 20 Available repos. I found a way to do this on the CLI. It still takes a long time to compile the list of available products but it works! Taken from: https://mariopang.blogspot.com/2014/12/satellite-60-subscribe-and-enable-repos.html Here's how I added RHEL 7.8 # The foreman.yml has a parameter to prevent hammer timeouts (https://access.redhat.com/solutions/1274723) hammer -c /etc/hammer/cli.modules.d/foreman.yml shell # First get the list of available products hammer> product list --organization-id 5 ----|----------------------------------------------------------------------------------|-------------|--------------|--------------|------------------ ID | NAME | DESCRIPTION | ORGANIZATION | REPOSITORIES | SYNC STATE ----|----------------------------------------------------------------------------------|-------------|--------------|--------------|------------------ 712 | Red Hat Enterprise Linux Server | | Ceph | 19 | Syncing Complete. 746 | Red Hat Enterprise Linux High Availability for RHEL Server | | Ceph | 3 | Syncing Complete. # Next get all the repositories available for that product (This will take a LONG time to return) hammer> repository-set list --organization-id 5 --product-id 712 -----|-----------|--------------------------------------------------------------------------------- ID | TYPE | NAME -----|-----------|--------------------------------------------------------------------------------- 2456 | yum | Red Hat Enterprise Linux 7 Server (RPMs) 2463 | yum | Red Hat Enterprise Linux 7 Server - Optional (RPMs) 2476 | yum | Red Hat Enterprise Linux 7 Server - Supplementary (RPMs) hammer> repository-set list --organization-id 5 --product-id 746 2762 | yum | Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs) # Next, enable the new version of RHEL for those repos hammer> repository-set enable --id 2463 --product-id 712 --organization-id 5 --releasever 7.8 --basearch x86_64 Repository enabled. hammer> repository-set enable --id 2456 --product-id 712 --organization-id 5 --releasever 7.8 --basearch x86_64 Repository enabled. hammer> repository-set enable --id 2476 --product-id 712 --organization-id 5 --releasever 7.8 --basearch x86_64 Repository enabled. hammer> repository-set enable --id 2762 --product-id 746 --organization-id 5 --releasever 7.8 --basearch x86_64 Repository enabled. **SHORT VERSION FOR RHEL7.9** RELEASEVER=7.9; for id in 2463 2456 2476; do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --id $id --product-id 712 --organization-id 5 --releasever $RELEASEVER --basearch x86_64; done hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --id 2762 --product-id 746 --organization-id 5 --releasever $RELEASEVER --basearch x86_64 RHEL 8.2 Example: hammer> repository-set list --organization-id 5 --product-id 608 -----|-----------|-------------------------------------------------------------------- ID | TYPE | NAME -----|-----------|-------------------------------------------------------------------- 7442 | yum | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Debug RPMs) 7446 | kickstart | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Kickstart) 7441 | yum | Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 7443 | yum | Red Hat Enterprise Linux 8 for x86_64 - AppStream (Source RPMs) 7417 | yum | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Debug RPMs) 7419 | file | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (ISOs) 7421 | kickstart | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Kickstart) 7416 | yum | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 7420 | file | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Source ISOs) 7418 | yum | Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Source RPMs) 7933 | yum | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (Debug RPMs) 7935 | file | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (ISOs) 7932 | yum | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (RPMs) 7934 | yum | Red Hat Enterprise Linux 8 for x86_64 - Supplementary (Source RPMs) 8694 | yum | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (Debug RPMs) 8693 | yum | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (RPMs) 8695 | yum | Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (Source RPMs) 9000 | yum | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (Debug RPMs) 9001 | yum | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (RPMs) 9002 | yum | Red Hat Satellite Tools 6 Beta for RHEL 8 x86_64 (Source RPMs) -----|-----------|-------------------------------------------------------------------- hammer> repository-set enable --organization-id 5 --id 7416 --product-id 608 --releasever 8.2 --basearch x86_64 Repository enabled. hammer> repository-set enable --organization-id 5 --id 7441 --product-id 608 --releasever 8.2 --basearch x86_64 Repository enabled. hammer> repository-set list --organization-id 5 --product-id 633 -----|------|---------------------------------------------------------------- ID | TYPE | NAME -----|------|---------------------------------------------------------------- 7955 | yum | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (Debug RPMs) 7954 | yum | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs) 7956 | yum | Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (Source RPMs) -----|------|---------------------------------------------------------------- hammer> repository-set enable --organization-id 5 --id 7954 --product-id 633 --releasever 8.2 --basearch x86_64 Repository enabled. **SHORT VERSION FOR RHEL8.6** RELEASEVER=8.6; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 5 --id 7416 --product-id 608 --releasever $RELEASEVER; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 5 --id 7441 --product-id 608 --releasever $RELEASEVER; hammer -c /etc/hammer/cli.modules.d/foreman.yml repository-set enable --organization-id 5 --id 7954 --product-id 633 --releasever $RELEASEVER Now you can go in the Web UI and manually sync those repos. ==== Testnodes won't register ==== This was observed after a bad reboot of the satellite VM once. **Symptoms:** * ceph-cm-ansible fails in jobs with: TASK [common : Register with subscription-manager.] **************************** 2020-01-02T23:51:00.955 INFO:teuthology.task.ansible.out:fatal: [smithi197.front.sepia.ceph.com]: FAILED! => {"attempts": 5, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} * Attempting to register a testnode manually results in: ''HTTP error (500 - Internal Server Error): Unable to register system, not all services available'' OR * ''HTTP error (500 - Internal Server Error): Required lock is already taken by other running tasks.'' * Saw this when ''ruby'' got killed by OOM. Check ''dmesg -T''. **Resolution:** ''ssh satellite.front.sepia.ceph.com "sudo katello-service restart"'' ===== Errors and Fixes ===== ==== Error: Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed ==== If RHEL clients are failing to perform yum/dnf transactions, check ''dmesg -T'' output on the Satellite server for OOM. [root@smithi077 ~]# dnf -v update Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile Updating Subscription Management repositories. DNF version: 4.7.0 cachedir: /var/cache/dnf User-Agent: constructed: 'libdnf (Red Hat Enterprise Linux 8.6; generic; Linux.x86_64)' repo: using cache for: copr:copr.fedorainfracloud.org:ceph:python3-asyncssh copr:copr.fedorainfracloud.org:ceph:python3-asyncssh: using metadata from Tue 27 Jul 2021 07:25:39 PM UTC. repo: using cache for: epel epel: using metadata from Tue 09 Aug 2022 02:13:36 AM UTC. repo: using cache for: lab-extras lab-extras: using metadata from Thu 28 Jan 2021 11:16:50 PM UTC. Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 157 kB/s | 2.8 kB 00:00 reviving: 'rhel-8-for-x86_64-appstream-rpms' can be revived - repomd matches. rhel-8-for-x86_64-appstream-rpms: using metadata from Mon 08 Aug 2022 05:07:41 AM UTC. Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 89 kB/s | 1.5 kB 00:00 reviving: 'rhel-8-for-x86_64-baseos-rpms' can be revived - repomd matches. loading repo 'rhel-8-for-x86_64-baseos-rpms' failure: loading of MD_TYPE_PRIMARY has failed. Error: Loading repository 'rhel-8-for-x86_64-baseos-rpms' has failed After you resolve the OOM issue, you will need to resync the repository. If BaseOS is the one that was failing: for id in $(hammer -c /etc/hammer/cli.modules.d/foreman.yml repository list | grep -i "baseos" | awk '{ print $1 }'); do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository synchronize --async --validate-contents true --id $id; done ==== Unable to verify server's identity: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:618) ==== mv -f /etc/rhsm/rhsm.conf.kat-backup /etc/rhsm/rhsm.conf subscription-manager unregister subscription-manager refresh yum reinstall http://satellite.front.sepia.ceph.com/pub/katello-ca-consumer-latest.noarch.rpm ==== The DMI UUID of this host (00000000-0000-0000-0000-0CC47AD93748) matches other registered hosts ==== Satellite 6.4 and before didn't care if you reused a hostname and the UUID was never checked. This was [[https://access.redhat.com/solutions/4396951|introduced in 6.5]]. An option to ignore this was [[https://bugzilla.redhat.com/show_bug.cgi?id=1772199|added in 6.7]]. [root@satellite ~]# grep uuid /etc/candlepin/candlepin.conf candlepin.use_system_uuid_for_matching=false Turns out this doesn't work. So instead, I modified a file to throw a warning instead of an error if Candlepin detects a duplicate UUID. # diff /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.0.32/app/services/katello/registration_manager.rb.orig /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.0.32/app/services/katello/registration_manager.rb 92c92,93 < registration_error("The DMI UUID of this host (%{uuid}) matches other registered hosts: %{existing}", uuid: host_uuid, existing: joined_hostnames(hosts)) --- > # registration_error("The DMI UUID of this host (%{uuid}) matches other registered hosts: %{existing}", uuid: host_uuid, existing: joined_hostnames(hosts)) > Rails.logger.warn("The DMI UUID of this host (%s) matches another host." % host_name) None of this actually ended up working so... ====== Solved On SATELLITE 6.12 ====== Despite this, in Jan 2023 we had to install a new Satellite instance, and it's version 6.12; this was chosen for allowing continued support, but initially suffered from the same issue. A support case, https://access.redhat.com/support/cases/#/case/03399123, was opened to deal with it and the solution was this - https://github.com/ceph/ceph-cm-ansible/pull/717 https://github.com/ceph/ceph-cm-ansible/pull/718