Sepia Lab Wiki

User Tools

Site Tools

Trace:
services:shortrunningcluster

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
services:shortrunningcluster [2026/08/26 17:56] – pdonnellservices:shortrunningcluster [2026/10/29 17:33] (current) – djgalloway
Line 1: Line 1:
 ====== SHORT_RUNNING_CLUSTER ====== ====== SHORT_RUNNING_CLUSTER ======
 ===== Summary ===== ===== Summary =====
-6 [[hardware:doli]] systems comprise this Ceph cluster.  It was deployed using the Dashboard at https://10.20.192.104:8443/.+6 [[hardware:doli]] systems comprise this Ceph cluster (fsid ''b6f4aaad-d45d-11f0-b949-905a08286547'', 54 x 14 TiB NVMe OSDs, 755 TiB raw).  It was deployed using cephadm and the Dashboard at https://10.20.192.104:8443/.
  
 +Day-to-day administration is done from ''doli01'' (the ''_admin'' host):
 +<code>
 +dgalloway@doli01:~$ sudo cephadm shell -- ceph status
 +</code>
  
 Here's a rundown of what this cluster stores Here's a rundown of what this cluster stores
-  * teuthology run logs after the 2025 lab migration +  * teuthology run logs after the 2025 lab migration (the ''teuthology'' CephFS) 
-  * chacra.ceph.com packages +  * home directories and shared data for teuthology/dev nodes (the ''tank'' CephFS) 
 +  * persistent volumes for the sepia OpenShift cluster (the ''openshift'' CephFS and ''openshift-rbd'' pool) 
 +  * chacra.ceph.com packages (RGW) 
 +  * scratch, signing, NFS and performance-testing file systems (see below)
  
 ===== Services ===== ===== Services =====
  
-Primarily this cluster serves [[services:cephfs]] for the lab. Notably this includes teuthology artifacts (logs, etc.).+Primarily this cluster serves [[services:cephfs]] for the lab. Notably this includes teuthology artifacts (logs, etc.).  It also runs RGW (7 daemons, ''standard'' and ''internal'' realms) and provides RBD for OpenShift.
  
 +===== File systems =====
 +
 +All CephFS file systems reside on this cluster.  Every file system has its own MDS service (''mds.<fsname>'') and its own pools (''cephfs.<fsname>.meta'' / ''cephfs.<fsname>.data'').  Mons: ''10.20.192.101,10.20.192.103,10.20.192.104,10.20.192.105,10.20.192.106'' (port 6789).
 +
 +^ File system ^ Use ^ Data pool ^ MDS ^ Notes ^
 +| ''teuthology'' | teuthology run archive (''/teuthology-archive'', mounted as ''/teuthology'' aka ''/a'' on teuthology hosts) | ''cephfs.teuthology.ec-data'' (EC 4+2) | 3 active + 2 standby | Largest consumer of the cluster; ''remote/'' logs are pruned by teuthology-prune-logs on soko04 |
 +| ''tank'' | home directories for teuthology/dev nodes, mounted at ''/tank'' (replaced the old ''home'' FS) | ''cephfs.tank.data'' | 3 active + 2 standby | Access restricted to administrators |
 +| ''scratch'' | general-purpose scratch space; anything here may be deleted at any time.  Create a personal top-level directory | ''cephfs.scratch.data'' | 1 active + 1 standby | |
 +| ''openshift'' | PVs for the sepia OpenShift cluster via ceph-csi (subvolumegroup ''csi'') | ''cephfs.openshift.data'' | 1 active | Managed by ceph-csi, do not touch by hand |
 +| ''signer'' | storage for the package signing host | ''cephfs.signer.data'' | 1 active | |
 +| ''cephfs-nfs'' | backing FS for NFS exports (''nfs.cephfs-nfs'' pool) | ''cephfs.cephfs-nfs.data'' | 1 active + 1 standby | No NFS cluster is currently defined (''ceph nfs cluster ls'' is empty) |
 +| ''performance'' | performance testing; created 2026-10-29 | ''cephfs.performance.data'' | 1 active + 1 standby | Not for long-term storage |
 +
 +===== Credentials =====
 +
 +Keys are **not** listed here.  Retrieve one on ''doli01'' with
 +<code>
 +sudo cephadm shell -- ceph auth get client.<name>
 +</code>
 +
 +^ Client ^ File system(s) ^ Access ^ Used by / purpose ^
 +| ''client.teuthologist'' | teuthology | rw (+p) | teuthology dispatcher/workers (rw archive mount on soko04) |
 +| ''client.teuthologist-ro'' | teuthology | ro | read-only archive mounts, e.g. ''/www-data/teuthology'' for the log web server |
 +| ''client.sepian'' | teuthology, tank, scratch | ro teuthology, ro tank, rw scratch | general lab users / ceph-devplayground mounts |
 +| ''client.tank'' | tank | rw (+p) | hosts mounting ''/tank'' home directories |
 +| ''client.scratch'' | scratch | rw (+p) | scratch mounts |
 +| ''client.signer'' | signer | rw | package signing host |
 +| ''client.cephfs-nfs'' | cephfs-nfs (+ ''nfs.cephfs-nfs'' pool) | rw (+ps) | NFS Ganesha |
 +| ''client.csi-cephfs-openshift'' | openshift | rw (plus broad mds/osd caps for provisioning) | ceph-csi CephFS driver in OpenShift |
 +| ''client.csi-rbd-openshift'', ''client.openshift-rbd'' | n/a (''openshift-rbd'' pool) | rw | ceph-csi RBD driver in OpenShift |
 +| ''client.performance'' | performance | rw (+p) | performance testing clients |
 +| ''client.performance-ro'' | performance | ro | read-only access to performance test results |
 +| ''client.pdonnell'' | all | rw (''allow *'' on mds) | CephFS team admin access |
 +| ''client.vshankar-rollback'' | n/a (''vshankar-rollback-*'' pools) | rw | EC rollback testing pools |
 +| ''client.admin'' | all | admin | cephadm / administrators only |
 +
 +The ''rwp'' MDS cap lets a client set layouts and quotas; ''s'' additionally allows snapshots.
 +
 +==== Mounting a file system ====
 +
 +Kernel client example for the ''performance'' FS with the read-write credential:
 +<code>
 +sudo mkdir -p /etc/ceph /mnt/performance
 +# put the output of "ceph auth get client.performance" (from doli01) in /etc/ceph/ceph.client.performance.keyring
 +sudo mount -t ceph performance@b6f4aaad-d45d-11f0-b949-905a08286547.performance=/ /mnt/performance \
 +  -o mon_addr=10.20.192.101:6789/10.20.192.103:6789/10.20.192.104:6789/10.20.192.105:6789/10.20.192.106:6789
 +</code>
 +Substitute the client name and fsname for the other file systems.  Any machine on the [[:vpnaccess|sepia VPN]] can mount these; see [[services:cephfs]] and the ceph-devplayground playbook.
 +
 +==== Adding a file system and credentials ====
 +<code>
 +sudo cephadm shell
 +ceph fs volume create <fsname> --placement=2
 +ceph fs authorize <fsname> client.<fsname> / rwp        # read-write
 +ceph fs authorize <fsname> client.<fsname>-ro / r       # read-only
 +</code>
  
 ===== Topology ===== ===== Topology =====
 <code> <code>
 root@doli01:~# cephadm shell -- ceph status root@doli01:~# cephadm shell -- ceph status
-Inferring fsid b6f4aaad-d45d-11f0-b949-905a08286547 
-Inferring config /var/lib/ceph/b6f4aaad-d45d-11f0-b949-905a08286547/mon.doli01/config 
   cluster:   cluster:
     id:     b6f4aaad-d45d-11f0-b949-905a08286547     id:     b6f4aaad-d45d-11f0-b949-905a08286547
Line 25: Line 85:
    
   services:   services:
-    mon: 5 daemons, quorum doli01,doli03,doli06,doli05,doli04 (age 4d) [leader: doli01] +    mon: 5 daemons, quorum doli01,doli03,doli06,doli05,doli04 (age 9w) [leader: doli01] 
-    mgr: doli04.uzocyw(active, since 4d), standbys: doli01.tabmpz +    mgr: doli04.uzocyw(active, since 9w), standbys: doli01.tabmpz 
-    mds: 12/12 daemons up, 4 standby +    mds: 13/13 daemons up, 5 standby 
-    osd: 54 osds: 54 up (since 4d), 54 in (since 6w) +    osd: 54 osds: 54 up (since 9w), 54 in (since 3M) 
-    rgw: 6 daemons active (6 hosts, 1 zones)+    rgw: 7 daemons active (6 hosts, 1 zones)
    
   data:   data:
-    volumes: 6/6 healthy +    volumes: 7/7 healthy 
-    pools:   27 pools, 3009 pgs +    pools:   29 pools, 3011 pgs 
-    objects: 290.52M objects, 310 TiB +    objects: 280.08M objects, 204 TiB 
-    usage:   474 TiB used, 280 TiB / 755 TiB avail +    usage:   376 TiB used, 379 TiB / 755 TiB avail 
-    pgs:     2994 active+clean +    pgs:     3009 active+clean 
-             10   active+clean+scrubbing+deep +             2    active+clean+scrubbing+deep
-             5    active+clean+scrubbing +
-  +
-  io: +
-    client:   169 B/s rd, 14 MiB/s wr, 0 op/s rd, 28 op/s wr +
-  +
-root@doli01:~# cephadm shell -- ceph osd tree +
-Inferring fsid b6f4aaad-d45d-11f0-b949-905a08286547 +
-Inferring config /var/lib/ceph/b6f4aaad-d45d-11f0-b949-905a08286547/mon.doli01/config +
-ID   CLASS  WEIGHT     TYPE NAME        STATUS  REWEIGHT  PRI-AFF +
- -1         754.52014  root default                               +
- -3         125.75336      host doli01                            +
-  0    ssd   13.97260          osd.0        up   1.00000  1.00000 +
-  1    ssd   13.97260          osd.1        up   1.00000  1.00000 +
-  2    ssd   13.97260          osd.2        up   1.00000  1.00000 +
-  3    ssd   13.97260          osd.3        up   1.00000  1.00000 +
-  4    ssd   13.97260          osd.4        up   1.00000  1.00000 +
-  5    ssd   13.97260          osd.5        up   1.00000  1.00000 +
-  6    ssd   13.97260          osd.6        up   1.00000  1.00000 +
-  7    ssd   13.97260          osd.7        up   1.00000  1.00000 +
-  8    ssd   13.97260          osd.8        up   1.00000  1.00000 +
- -9         125.75336      host doli02                            +
- 12    ssd   13.97260          osd.12       up   1.00000  1.00000 +
- 14    ssd   13.97260          osd.14       up   1.00000  1.00000 +
- 21    ssd   13.97260          osd.21       up   1.00000  1.00000 +
- 26    ssd   13.97260          osd.26       up   1.00000  1.00000 +
- 31    ssd   13.97260          osd.31       up   1.00000  1.00000 +
- 36    ssd   13.97260          osd.36       up   1.00000  1.00000 +
- 41    ssd   13.97260          osd.41       up   1.00000  1.00000 +
- 46    ssd   13.97260          osd.46       up   1.00000  1.00000 +
- 50    ssd   13.97260          osd.50       up   1.00000  1.00000 +
--11         125.75336      host doli03                            +
- 11    ssd   13.97260          osd.11       up   1.00000  1.00000 +
- 17    ssd   13.97260          osd.17       up   1.00000  1.00000 +
- 22    ssd   13.97260          osd.22       up   1.00000  1.00000 +
- 28    ssd   13.97260          osd.28       up   1.00000  1.00000 +
- 33    ssd   13.97260          osd.33       up   1.00000  1.00000 +
- 40    ssd   13.97260          osd.40       up   1.00000  1.00000 +
- 44    ssd   13.97260          osd.44       up   1.00000  1.00000 +
- 49    ssd   13.97260          osd.49       up   1.00000  1.00000 +
- 52    ssd   13.97260          osd.52       up   1.00000  1.00000 +
- -7         125.75336      host doli04                            +
-  9    ssd   13.97260          osd.9        up   1.00000  1.00000 +
- 15    ssd   13.97260          osd.15       up   1.00000  1.00000 +
- 19    ssd   13.97260          osd.19       up   1.00000  1.00000 +
- 24    ssd   13.97260          osd.24       up   1.00000  1.00000 +
- 29    ssd   13.97260          osd.29       up   1.00000  1.00000 +
- 34    ssd   13.97260          osd.34       up   1.00000  1.00000 +
- 39    ssd   13.97260          osd.39       up   1.00000  1.00000 +
- 43    ssd   13.97260          osd.43       up   1.00000  1.00000 +
- 48    ssd   13.97260          osd.48       up   1.00000  1.00000 +
--13         125.75336      host doli05                            +
- 13    ssd   13.97260          osd.13       up   1.00000  1.00000 +
- 18    ssd   13.97260          osd.18       up   1.00000  1.00000 +
- 23    ssd   13.97260          osd.23       up   1.00000  1.00000 +
- 27    ssd   13.97260          osd.27       up   1.00000  1.00000 +
- 32    ssd   13.97260          osd.32       up   1.00000  1.00000 +
- 38    ssd   13.97260          osd.38       up   1.00000  1.00000 +
- 45    ssd   13.97260          osd.45       up   1.00000  1.00000 +
- 51    ssd   13.97260          osd.51       up   1.00000  1.00000 +
- 53    ssd   13.97260          osd.53       up   1.00000  1.00000 +
- -5         125.75336      host doli06                            +
- 10    ssd   13.97260          osd.10       up   1.00000  1.00000 +
- 16    ssd   13.97260          osd.16       up   1.00000  1.00000 +
- 20    ssd   13.97260          osd.20       up   1.00000  1.00000 +
- 25    ssd   13.97260          osd.25       up   1.00000  1.00000 +
- 30    ssd   13.97260          osd.30       up   1.00000  1.00000 +
- 35    ssd   13.97260          osd.35       up   1.00000  1.00000 +
- 37    ssd   13.97260          osd.37       up   1.00000  1.00000 +
- 42    ssd   13.97260          osd.42       up   1.00000  1.00000 +
- 47    ssd   13.97260          osd.47       up   1.00000  1.00000+
 </code> </code>
  
 +The ''ceph osd tree'' output is unchanged from the previous revision of this page: 6 hosts (doli01-06), 9 x 13.97 TiB ssd OSDs each, all up and in.
 +</code>
services/shortrunningcluster.txt · Last modified: by djgalloway