Sepia Lab Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Central Logging (Loki) ======
===== Summary =====
All lab logs land in one queryable store: a LokiStack in the ''openshift-logging'' namespace on the OpenShift cluster.  See https://github.com/ceph/sepia-openshift/tree/main/openshift-logging for the full deployment runbook.

Two tenants:
  * **infrastructure** - journald + auditd from ~205 baremetal lab hosts (pushed by Grafana Alloy, deployed via the ''alloy'' role in ceph-cm-ansible) plus the OCP node journals (collected in-cluster).  30d retention.
  * **audit** - the full OCP cluster audit trail including request bodies for writes.  Answers "who changed what."  90d retention.

OCP infra-compute/infra-storage nodes and testnodes are **excluded** from the Alloy play.  The OCP nodes are already covered by the in-cluster collector.

===== Reading logs =====
==== Grafana ====
https://ci-metrics.ceph.com has a ''Loki Infrastructure'' datasource.  Any dashboard built on it **must** go in the Private folder (''spec.folderRef: private'') or it's world-readable.

==== OpenShift console ====
**[[https://console-openshift-console.apps.pok.os.sepia.ceph.com/monitoring/logs?q=%7B+log_type%3D%22application%22+%7D+%7C+json|Observe -> Logs]]** in the console.  This is the easiest way to query the audit tenant.

===== Useful queries =====
==== Logs from a baremetal host ====
<code>
{host="soko01.front.sepia.ceph.com"} |= "nginx"
{job="auditd", host="soko03.front.sepia.ceph.com"}
</code>

==== Who changed something in OpenShift ====
<code>
{log_type="audit"} | json | objectRef_namespace="paddles" |= "requestObject"
</code>

==== Who shelled into a pod ====
<code>
{log_type="audit"} |= "pods/exec"
</code>

===== Troubleshooting =====
==== One host stopped shipping logs ====
Check the agent on the host.  Permission-denied on ''/var/log/audit/audit.log'' means the file-permission tasks in the role didn't take.
<code>
journalctl -u alloy
</code>

==== ALL hosts stopped shipping at once ====
The cluster's ingress cert rotated and the agents no longer trust the gateway.  Re-extract the CA and re-run the play.
<code>
oc get cm -n openshift-config-managed default-ingress-cert \
  -o jsonpath='{.data.ca-bundle\.crt}'
# -> ceph-sepia-secrets/ansible/files/lokistack-ingress-ca.crt, then
ansible-playbook alloy.yml
</code>

==== Audit tenant is empty ====
The ''collect-audit-logs'' ClusterRoleBinding is missing.  The CLF status stays green with no error, so this is easy to miss.
<code>
oc apply -f rbac-collector.yaml
</code>

==== Grafana/agent tokens stopped working ====
Don't use ''oc create token'' - it issues a 1-hour token.  Use the long-lived secrets created by the rbac yamls:
<code>
oc -n openshift-logging get secret sepia-hosts-token -o jsonpath='{.data.token}' | base64 -d
oc -n openshift-logging get secret grafana-reader-token -o jsonpath='{.data.token}' | base64 -d
</code>