services:longrunningcluster
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
services:longrunningcluster [2022/08/26 21:37] djgalloway |
services:longrunningcluster [2023/02/19 15:18] (current) akraitman [Summary] |
||
|---|---|---|---|
| Line 13: | Line 13: | ||
| * [[production:signer.front.sepia.ceph.com]] | * [[production:signer.front.sepia.ceph.com]] | ||
| + | |||
| + | Cluster dashboard | ||
| + | |||
| + | https://reesi004.front.sepia.ceph.com:8443/ | ||
| ===== Topology ===== | ===== Topology ===== | ||
| <code> | <code> | ||
| Line 37: | Line 41: | ||
| ===== Upgrading the Cluster ===== | ===== Upgrading the Cluster ===== | ||
| - | As of this writing, the luminous branch is the repo defined in ''/etc/apt/sources.list.d/ceph.list'' on the LRC nodes. The Ceph docs can be followed for this procedure but, basically, update and reboot each host at a time starting with MONs, MGRs, MDSs, then OSD hosts. | + | The LRC is a testbed we use to test a release candidate before announcing. |
| + | For example: | ||
| + | <code> | ||
| + | ceph orch upgrade start quay.ceph.io/ceph-ci/ceph:da36d2c9a106ed5231aa923e6c04a2485c89ef4b | ||
| + | |||
| + | watch "ceph -s; ceph orch upgrade status; ceph versions" | ||
| + | </code> | ||
| ===== MONs run out of disk space ===== | ===== MONs run out of disk space ===== | ||
| I sadly got too small of disks for the reesi when we purchased them so they occasionally run out of space in ''/var/log/ceph'' before logrotate gets a chance to run (even though it runs 4x a day. The process below will get you back up and running again but will wipe out all logs. | I sadly got too small of disks for the reesi when we purchased them so they occasionally run out of space in ''/var/log/ceph'' before logrotate gets a chance to run (even though it runs 4x a day. The process below will get you back up and running again but will wipe out all logs. | ||
| Line 47: | Line 57: | ||
| </code> | </code> | ||
| - | ===== Replace LRC Host's root drive ===== | + | ===== One-liners ====== |
| - | ==== On non-mon hosts ==== | + | Most of the stuff above is no longer valuable since Ceph has evolved over time. Here's some one-liners that were useful at the time I posted them. |
| - | - ''ceph osd set noout'' on admin host | + | |
| - | - ''ceph osd set noscrub; ceph osd set nodeep-scrub'' to avoid unnecessary I/O | + | |
| - | - Stop ceph services on OSD host | + | |
| - | - ''stop ceph-osd-all'' on Ubuntu | + | |
| - | - ''service ceph stop osd.#'' on RHEL | + | |
| - | - Back up ''/etc/ceph'' | + | |
| - | - ''scp root@mira###.front.sepia.ceph.com:/etc/ceph/ceph.conf .'' | + | |
| - | - ''umount /var/lib/ceph/osd/*'' | + | |
| - | - Back up ''/var/lib/ceph/osd'' | + | |
| - | - ''scp -r root@mira###.front.sepia.ceph.com:/var/lib/ceph/osd/ .'' | + | |
| - | - Reimage the machine | + | |
| - | - Install ceph packages | + | |
| - | - If needed. set up repo file | + | |
| - | - Also if needed, import repo GPG key ''wget -qO - %%http://download.ceph.com/keys/release.asc%% | sudo apt-key add -'' | + | |
| - | - ''apt-get install ceph ceph-base ceph-common ceph-osd ceph-test libcephfs1 python-cephfs ceph-deploy'' | + | |
| - | - Make sure ntpd is configured and enabled | + | |
| - | - Manually run ''ntpdate $ntpserver'' for one-time sync | + | |
| - | - Configure or disable firewall | + | |
| - | - Replace ''/etc/ceph'' and ''/var/lib/ceph/osd'' structures | + | |
| - | - ''scp ceph.conf root@mira###.front.sepia.ceph.com:/etc/ceph/'' | + | |
| - | - ''scp -r osd/* root@mira###.front.sepia.ceph.com:/var/lib/ceph/osd/'' | + | |
| - | - Set permissions | + | |
| - | - ''chown -R ceph:ceph /var/lib/ceph/osd/'' | + | |
| - | - ''chown ceph:ceph /etc/ceph/ceph.conf'' | + | |
| - | - Create an ssh key, copy the pubkey to ''/root/.ssh/authorized_keys'' on a monhost and run ''ceph-deploy gatherkeys $mon'' where ''$mon'' is a mon host | + | |
| - | - Copy keys to their appropriate places | + | |
| - | - For the bootstrap key, | + | |
| - | - ''mv ceph.bootstrap-osd.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring'' | + | |
| - | - ''mv ceph.client.admin.keyring /etc/ceph/'' | + | |
| - | - ''chown ceph:ceph /var/lib/ceph/bootstrap-osd/ceph.keyring'' | + | |
| - | - ''reboot'' | + | |
| - | - Unset flags from step 1 | + | |
| - | See [[http://docs.ceph.com/docs/jewel/rados/troubleshooting/troubleshooting-osd/#stopping-w-out-rebalancing|Ceph Docs - Stopping without rebalancing]] | + | === Restart mon service === |
| - | ===== Add blank disk as OSD ===== | + | |
| <code> | <code> | ||
| - | disk=sdX | + | systemctl restart ceph-28f7427e-5558-4ffd-ae1a-51ec3042759a@mon.$(hostname -s).service |
| - | ceph-disk zap /dev/$disk | + | |
| - | ceph-disk prepare /dev/$disk | + | |
| - | ceph-disk activate /dev/${disk}1 | + | |
| </code> | </code> | ||
| - | ===== Replace Failing OSD disk ===== | ||
| - | ==== Evacuating OSD data ==== | ||
| - | If the disk is still relatively healthy and you think it can survive a while longer, you should evacuate the data off it slowly. | ||
| - | - On a mon node, ''ceph osd reweight $osdnum 0.75'' or -0.25 the current weight | + | === Watch logs for a mon === |
| - | - Wait until recovery I/O is done and keep doing this until the OSD is reweighted to 0 | + | <code> |
| + | podman logs -f $(podman ps | grep "\-mon" | awk '{ print $1 }') | ||
| + | </code> | ||
| - | ==== Taking the OSD out of the cluster ==== | + | === LRC iscsi volume for the RHEV cluster=== |
| - | - On a mon node, ''ceph osd out $id''. This makes sure there are 3 replicas of each PG evacuated. | + | On Nov 2022 we started seeing data corruption on our main gluster volume where we have all our critical VM's so we connected an iscsi volume from the LRC, those are the steps to connect an iscsi volume to a rev cluster according to this doc |
| - | - If any recovery I/O occurs, wait for it to finish | + | https://docs.google.com/document/d/1GYwv5y4T5vy-1oeAzw-zoLgQs0I3y5v_xD1wXscAA7M/edit |
| - | - On the OSD host, ''stop ceph-osd id=$id'' | + | |
| - | - Some recovery I/O will occur. This is just the cluster rebalancing. It's fine. | + | First, make sure you configured the iscsi clients(the RHEV hypervisor hosts in our case) according to this doc and copy the iscsi initiator located under(we will need it for step 11 when we create the hosts on the lrc) /etc/iscsi/initiatorname.iscsi on each host |
| - | - Back on the mon host, <code> | + | https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html-single/block_device_guide/index#configuring-the-iscsi-initiator-for-rhel_block |
| - | ceph osd crush remove osd.$id | + | also configure CHAP on each rhev host by adding this in /etc/iscsi/iscsid.conf |
| - | ceph osd down osd.$id # may not be needed as long as osd service is stopped | + | |
| - | ceph osd rm osd.$id | + | node.session.auth.authmethod = CHAP |
| - | ceph auth del osd.$id | + | node.session.auth.username = <username> |
| + | node.session.auth.password = <password> | ||
| + | |||
| + | |||
| + | ssh to one of the reesi hosts(I configured it from reesi005) and follow the next steps to configure iscsi and create a volume on the LRC | ||
| + | |||
| + | 1. Create an rbd pool | ||
| + | <code> | ||
| + | ceph osd pool create <poolname> | ||
| + | ceph osd pool application enable <poolname> rbd | ||
| </code> | </code> | ||
| - | - Unmount the disk from the OSD host | + | |
| - | - ''umount /var/lib/ceph/osd/ceph-$id'' | + | 2. Deploy iscsi on at least four hosts - create a yaml file |
| - | - ''rm -rf /var/lib/ceph/osd/ceph-$id'' | + | <code> |
| - | - Replace the disk | + | service_type: iscsi |
| - | - On the OSD host, <code> | + | service_id: iscsi |
| - | disk=sdX | + | placement: |
| - | ceph-disk zap /dev/$disk | + | hosts: |
| - | ceph-disk prepare /dev/$disk | + | - reesi002 |
| - | mkdir /mnt/tmp | + | - reesi003 |
| - | mount /dev/${disk}1 /mnt/tmp | + | - reesi004 |
| - | mkdir /var/lib/ceph/osd/ceph-$(cat /mnt/tmp/whoami) | + | - reesi005 |
| - | chown ceph:ceph /var/lib/ceph/osd/ceph-$(cat /mnt/tmp/whoami) | + | spec: |
| - | umount /mnt/tmp | + | pool: lrc |
| - | ceph-disk activate /dev/${disk}1 | + | api_secure: false |
| </code> | </code> | ||
| - | ===== One-liners ====== | + | 3. Connect to the iscsi container on one of the deployed hosts, to find the exact container id run "podman ps" and look for the iscsi container with the word "tcmu" in the end. |
| - | Most of the stuff above is no longer valuable since Ceph has evolved over time. Here's some one-liners that were useful at the time I posted them. | + | <code> |
| + | Podman exec -it <iscsi container id> /bin/bash | ||
| + | </code> | ||
| - | === Restart mon service === | + | for example: |
| <code> | <code> | ||
| - | systemctl restart ceph-28f7427e-5558-4ffd-ae1a-51ec3042759a@mon.$(hostname -s).service | + | podman exec -it ceph-28f7427e-5558-4ffd-ae1a-51ec3042759a-iscsi-iscsi-reesi005-luegfv-tcmu /bin/bash |
| </code> | </code> | ||
| + | 4. Enter the gwcli | ||
| + | <code> | ||
| + | gwcli | ||
| + | </code> | ||
| - | === Watch logs for a mon === | + | 5. Go to the iscsi-targets |
| <code> | <code> | ||
| - | podman logs -f $(podman ps | grep "\-mon" | awk '{ print $1 }') | + | cd iscsi-targets/ |
| </code> | </code> | ||
| + | |||
| + | 6. Go to the storage iqn | ||
| + | <code> | ||
| + | cd iqn.2003-01.com.redhat.iscsi-gw:lrc-iscsi1/ | ||
| + | </code> | ||
| + | |||
| + | 7. Go to gateways | ||
| + | <code> | ||
| + | cd gateways | ||
| + | </code> | ||
| + | |||
| + | 8. Create all four gateway's as you specified in the yaml file on step 2 | ||
| + | <code> | ||
| + | create reesi002.front.sepia.ceph.com 172.21.2.202 | ||
| + | create reesi003.front.sepia.ceph.com 172.21.2.203 | ||
| + | create reesi004.front.sepia.ceph.com 172.21.2.204 | ||
| + | create reesi005.front.sepia.ceph.com 172.21.2.205 | ||
| + | </code> | ||
| + | |||
| + | 9. Go to disks | ||
| + | <code> | ||
| + | cd .. | ||
| + | cd disks/ | ||
| + | </code> | ||
| + | |||
| + | 9. Create RBD image with the name "vol1" in the "lrc" pool | ||
| + | <code> | ||
| + | create pool=lrc image=vol1 size=20T image=rbdimage size=50g | ||
| + | </code> | ||
| + | |||
| + | 10. Go to hosts | ||
| + | <code> | ||
| + | cd .. | ||
| + | cd hosts/ | ||
| + | </code> | ||
| + | |||
| + | 11. Create the hosts(RHEV hosts, if you have four rhev hosts you will need to run this four times one for each iqn ) | ||
| + | <code> | ||
| + | create client_iqn=<iqn from the rhev host> | ||
| + | </code> | ||
| + | |||
| + | 12. cd to each iqn you created in step 11 and enable chap | ||
| + | <code> | ||
| + | auth username=<username> password=<password> | ||
| + | </code> | ||
| + | |||
| + | 13. cd to each iqn you added in step 11 and add the RBD image created in step 9 | ||
| + | <code> | ||
| + | disk add <pool_name>/<RBD image name> | ||
| + | </code> | ||
| + | |||
| + | 14. Set discovery auth to CHAP on the iscsi-targets | ||
| + | <code> | ||
| + | cd ../../ | ||
| + | discovery_auth username=<username> password=<password> | ||
| + | </code> | ||
| + | |||
| + | The final step is to mount this RBD_image/lun in RHEV-M Dashboard | ||
| + | |||
| + | go to https://mgr01.front.sepia.ceph.com/ovirt-engine/webadmin/?locale=en_US#storage | ||
| + | Create a new Storage domain and choose the iscsi storage type and fill out the discovery targets section with an IP on one of the iscsi gateway ip's you configured in the yaml in step 2 and fill out the auth with the CHAP username & password you configured in step 14 | ||
| + | |||
| + | |||
services/longrunningcluster.1661549839.txt.gz · Last modified: 2022/08/26 21:37 by djgalloway
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
