services:cephfs
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
services:cephfs [2023/06/12 20:25] pdonnell |
services:cephfs [2025/12/17 14:53] (current) pdonnell [Mounting all Sepia CephFS file systems] |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== CephFS ====== | ====== CephFS ====== | ||
| - | There are a few CephFS file systems available in the sepia lab. These reside on the [[services:longrunningcluster|Long Running Cluster]]. Access to this Ceph cluster is available for any machine on the [[:vpnaccess|sepia VPN]]. You can and **should** mount these file systems on your laptop or development machine. This reduces load on some shared machines, like [[services:teuthology|teuthology]], and usually provides faster access depending on your client or network. | + | There are a few CephFS file systems available in the sepia lab. These reside on the [[services:longrunningcluster|Long Running Cluster]]. |
| - | Below we ssh to the ''reesi001.front.sepia.ceph.com'' machine. When given access to the [[:vpnaccess#requesting_access|sepia VPN]], the ssh key you shared should allow accessing this machine. | + | Access to this Ceph cluster is available for any machine on the [[:vpnaccess|sepia VPN]]. You can and **should** mount these file systems on your laptop or development machine. This reduces load on some shared machines, like [[services:teuthology|teuthology]], and usually provides faster access depending on your client or network. However, for everyday use, you may prefer to access CephFS from a machine co-located with the other lab infrastructure, such as a [[:devplayground|Developer Playground]] machine, where OSD latency/bandwidth will be optimal. |
| + | |||
| + | In the scripts/commands below, we will ssh to the ''doli01.front.sepia.ceph.com'' machine. When [[:vpnaccess#requesting_access|given access to the sepia VPN]], the ssh key you shared should allow accessing this machine. | ||
| Line 11: | Line 13: | ||
| <code bash> | <code bash> | ||
| + | |||
| sudo mkdir -p -m 755 /etc/ceph | sudo mkdir -p -m 755 /etc/ceph | ||
| - | ssh reesi001.front.sepia.ceph.com 'env CEPH_KEYRING=/etc/ceph/client.sepian.keyring ceph --id sepian config generate-minimal-conf' | sudo tee /etc/ceph/ceph.conf | + | ssh doli01.front.sepia.ceph.com 'env CEPH_KEYRING=/etc/ceph/client.sepian.keyring ceph --id sepian config generate-minimal-conf' | sudo tee /etc/ceph/ceph.conf |
| sudo chmod 644 /etc/ceph/ceph.conf | sudo chmod 644 /etc/ceph/ceph.conf | ||
| - | ssh reesi001.front.sepia.ceph.com 'cat /etc/ceph/client.sepian.keyring' | sudo tee /etc/ceph/client.sepian.keyring | + | ssh doli01.front.sepia.ceph.com 'cat /etc/ceph/client.sepian.keyring' | sudo tee /etc/ceph/client.sepian.keyring | sudo tee -a /etc/ceph/keyring |
| sudo chmod 644 /etc/ceph/client.sepian.keyring | sudo chmod 644 /etc/ceph/client.sepian.keyring | ||
| - | ssh reesi001.front.sepia.ceph.com 'ceph-authtool /etc/ceph/client.sepian.keyring -n client.sepian -p' | sudo tee /etc/ceph/client.sepian.secret | + | ssh doli01.front.sepia.ceph.com 'ceph-authtool /etc/ceph/client.sepian.keyring -n client.sepian -p' | sudo tee /etc/ceph/client.sepian.secret |
| sudo chmod 600 /etc/ceph/client.sepian.secret | sudo chmod 600 /etc/ceph/client.sepian.secret | ||
| </code> | </code> | ||
| Line 23: | Line 26: | ||
| ==== Mounting all Sepia CephFS file systems ==== | ==== Mounting all Sepia CephFS file systems ==== | ||
| + | |||
| + | **Note**: you need the ''mount.ceph'' program installed to use this mount syntax. It is usually part of the ''ceph-common'' package. | ||
| Generate mounts for your ''/etc/fstab'' using the script below. Copy it locally, mark it executable, and run: | Generate mounts for your ''/etc/fstab'' using the script below. Copy it locally, mark it executable, and run: | ||
| Line 28: | Line 33: | ||
| <code bash> | <code bash> | ||
| #!/bin/bash | #!/bin/bash | ||
| + | |||
| + | set -ex | ||
| function genmount { | function genmount { | ||
| + | local path="$1" | ||
| + | local mntpnt="$2" | ||
| + | local fsname="$3" | ||
| + | |||
| local secret=$(sudo cat /etc/ceph/client.sepian.secret) | local secret=$(sudo cat /etc/ceph/client.sepian.secret) | ||
| # create mountpoint | # create mountpoint | ||
| - | sudo mkdir -p -- "$2" | + | sudo mkdir -p -- "$mntpnt" || true |
| # make the mountpoint directory (shadowed) unwriteable to prevent accidental modification | # make the mountpoint directory (shadowed) unwriteable to prevent accidental modification | ||
| - | sudo chmod 000 -- "$2" | + | sudo chmod 000 -- "$mntpnt" || true |
| # set it immutable to enforce that even for root | # set it immutable to enforce that even for root | ||
| - | sudo chattr +i -- "$2" | + | sudo chattr +i -- "$mntpnt" || true |
| - | printf '172.21.2.201,172.21.2.202,172.21.2.203:%s\t%s\tceph\tname=sepian,secret=%s,mds_namespace=%s,_netdev\t0\t2\n' "$1" "$2" "$secret" "$3" | + | printf 'sepian@b6f4aaad-d45d-11f0-b949-905a08286547.%s=%s\t%s\tceph\t_netdev\t0\t0\n' "$fsname" "$path" "$mntpnt" |
| } | } | ||
| + | |||
| genmount /teuthology-archive /teuthology teuthology | sudo tee -a /etc/fstab | genmount /teuthology-archive /teuthology teuthology | sudo tee -a /etc/fstab | ||
| genmount / /scratch scratch | sudo tee -a /etc/fstab | genmount / /scratch scratch | sudo tee -a /etc/fstab | ||
| - | genmount / /postfile postfile | sudo tee -a /etc/fstab | + | </code> |
| + | |||
| + | The fstab changes will cause these file systems to mount on boot. After adding these entries for the first time, you need to manually mount them: | ||
| + | |||
| + | <code bash> | ||
| + | sudo mount /teuthology | ||
| + | sudo mount /scratch | ||
| + | sudo mount /postfile | ||
| </code> | </code> | ||
| Line 62: | Line 80: | ||
| <code bash> | <code bash> | ||
| - | ln -s /teuthology /a | + | sudo ln -s /teuthology /a |
| </code> | </code> | ||
| Line 73: | Line 91: | ||
| The [[https://docs.ceph.com/en/latest/man/8/ceph-post-file/|ceph-post-file]] utility dumps results in this file system. Users are encouraged to use this utility to share artifacts with Ceph developers. | The [[https://docs.ceph.com/en/latest/man/8/ceph-post-file/|ceph-post-file]] utility dumps results in this file system. Users are encouraged to use this utility to share artifacts with Ceph developers. | ||
| + | |||
| + | ==== The home FS ==== | ||
| + | |||
| + | There is a "home" file system which hosts the home directories of users of teuthology and potentially other development nodes. Its access is restricted to administrators. | ||
services/cephfs.1686601514.txt.gz · Last modified: 2023/06/12 20:25 by pdonnell
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
