services:cephfs
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
services:cephfs [2023/06/12 20:11] pdonnell |
services:cephfs [2025/12/17 14:53] (current) pdonnell [Mounting all Sepia CephFS file systems] |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== CephFS ====== | ====== CephFS ====== | ||
| - | There are a few CephFS file systems available in the sepia lab. These reside on the [[services:longrunningcluster|Long Running Cluster]]. Access to this Ceph cluster is available for any machine on the VPN. You can and should (!!) mount these file systems on your laptop or development machine. This reduces load on some shared machines, like [[services:teuthology|teuthology]], and usually provides faster access depending on your client or network. | + | There are a few CephFS file systems available in the sepia lab. These reside on the [[services:longrunningcluster|Long Running Cluster]]. |
| - | Below we ssh to the ''reesi001.front.sepia.ceph.com'' machine. When given access to the [[:vpnaccess|sepia VPN]], the ssh key you shared should allow accessing this machine. | + | Access to this Ceph cluster is available for any machine on the [[:vpnaccess|sepia VPN]]. You can and **should** mount these file systems on your laptop or development machine. This reduces load on some shared machines, like [[services:teuthology|teuthology]], and usually provides faster access depending on your client or network. However, for everyday use, you may prefer to access CephFS from a machine co-located with the other lab infrastructure, such as a [[:devplayground|Developer Playground]] machine, where OSD latency/bandwidth will be optimal. |
| + | |||
| + | In the scripts/commands below, we will ssh to the ''doli01.front.sepia.ceph.com'' machine. When [[:vpnaccess#requesting_access|given access to the sepia VPN]], the ssh key you shared should allow accessing this machine. | ||
| Line 11: | Line 13: | ||
| <code bash> | <code bash> | ||
| + | |||
| sudo mkdir -p -m 755 /etc/ceph | sudo mkdir -p -m 755 /etc/ceph | ||
| - | ssh reesi001.front.sepia.ceph.com 'env CEPH_KEYRING=/etc/ceph/client.sepian.keyring ceph --id sepian config generate-minimal-conf' | sudo tee /etc/ceph/ceph.conf | + | ssh doli01.front.sepia.ceph.com 'env CEPH_KEYRING=/etc/ceph/client.sepian.keyring ceph --id sepian config generate-minimal-conf' | sudo tee /etc/ceph/ceph.conf |
| sudo chmod 644 /etc/ceph/ceph.conf | sudo chmod 644 /etc/ceph/ceph.conf | ||
| - | ssh reesi001.front.sepia.ceph.com 'cat /etc/ceph/client.sepian.keyring' | sudo tee /etc/ceph/client.sepian.keyring | + | ssh doli01.front.sepia.ceph.com 'cat /etc/ceph/client.sepian.keyring' | sudo tee /etc/ceph/client.sepian.keyring | sudo tee -a /etc/ceph/keyring |
| sudo chmod 644 /etc/ceph/client.sepian.keyring | sudo chmod 644 /etc/ceph/client.sepian.keyring | ||
| - | ssh reesi001.front.sepia.ceph.com 'ceph-authtool /etc/ceph/client.sepian.keyring -n client.sepian -p' | sudo tee /etc/ceph/client.sepian.secret | + | ssh doli01.front.sepia.ceph.com 'ceph-authtool /etc/ceph/client.sepian.keyring -n client.sepian -p' | sudo tee /etc/ceph/client.sepian.secret |
| sudo chmod 600 /etc/ceph/client.sepian.secret | sudo chmod 600 /etc/ceph/client.sepian.secret | ||
| </code> | </code> | ||
| Line 24: | Line 27: | ||
| ==== Mounting all Sepia CephFS file systems ==== | ==== Mounting all Sepia CephFS file systems ==== | ||
| - | Generate mounts for your ''/etc/fstab'' using the script below: | + | **Note**: you need the ''mount.ceph'' program installed to use this mount syntax. It is usually part of the ''ceph-common'' package. |
| + | |||
| + | Generate mounts for your ''/etc/fstab'' using the script below. Copy it locally, mark it executable, and run: | ||
| <code bash> | <code bash> | ||
| #!/bin/bash | #!/bin/bash | ||
| + | |||
| + | set -ex | ||
| function genmount { | function genmount { | ||
| + | local path="$1" | ||
| + | local mntpnt="$2" | ||
| + | local fsname="$3" | ||
| + | |||
| local secret=$(sudo cat /etc/ceph/client.sepian.secret) | local secret=$(sudo cat /etc/ceph/client.sepian.secret) | ||
| # create mountpoint | # create mountpoint | ||
| - | sudo mkdir -p -- "$2" | + | sudo mkdir -p -- "$mntpnt" || true |
| # make the mountpoint directory (shadowed) unwriteable to prevent accidental modification | # make the mountpoint directory (shadowed) unwriteable to prevent accidental modification | ||
| - | sudo chmod 000 -- "$2" | + | sudo chmod 000 -- "$mntpnt" || true |
| # set it immutable to enforce that even for root | # set it immutable to enforce that even for root | ||
| - | sudo chattr +i -- "$2" | + | sudo chattr +i -- "$mntpnt" || true |
| - | printf '172.21.2.201,172.21.2.202,172.21.2.203:%s\t%s\tceph\tname=sepian,secret=%s,mds_namespace=%s,_netdev\t0\t2\n' "$1" "$2" "$secret" "$3" | + | printf 'sepian@b6f4aaad-d45d-11f0-b949-905a08286547.%s=%s\t%s\tceph\t_netdev\t0\t0\n' "$fsname" "$path" "$mntpnt" |
| } | } | ||
| + | |||
| genmount /teuthology-archive /teuthology teuthology | sudo tee -a /etc/fstab | genmount /teuthology-archive /teuthology teuthology | sudo tee -a /etc/fstab | ||
| genmount / /scratch scratch | sudo tee -a /etc/fstab | genmount / /scratch scratch | sudo tee -a /etc/fstab | ||
| - | genmount / /postfile postfile | sudo tee -a /etc/fstab | + | </code> |
| + | |||
| + | The fstab changes will cause these file systems to mount on boot. After adding these entries for the first time, you need to manually mount them: | ||
| + | |||
| + | <code bash> | ||
| + | sudo mount /teuthology | ||
| + | sudo mount /scratch | ||
| + | sudo mount /postfile | ||
| </code> | </code> | ||
| Line 62: | Line 80: | ||
| <code bash> | <code bash> | ||
| - | ln -s /teuthology /a | + | sudo ln -s /teuthology /a |
| </code> | </code> | ||
| Line 73: | Line 91: | ||
| The [[https://docs.ceph.com/en/latest/man/8/ceph-post-file/|ceph-post-file]] utility dumps results in this file system. Users are encouraged to use this utility to share artifacts with Ceph developers. | The [[https://docs.ceph.com/en/latest/man/8/ceph-post-file/|ceph-post-file]] utility dumps results in this file system. Users are encouraged to use this utility to share artifacts with Ceph developers. | ||
| + | |||
| + | ==== The home FS ==== | ||
| + | |||
| + | There is a "home" file system which hosts the home directories of users of teuthology and potentially other development nodes. Its access is restricted to administrators. | ||
services/cephfs.1686600701.txt.gz · Last modified: 2023/06/12 20:11 by pdonnell
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
