Sepia Lab Wiki

This is an old revision of the document!

Grafana Agent Setup for Lab Nodes Monitoring

** Overview **

The grafana_agent Ansible role deploys Grafana Agent (static mode) on Lab Nodes to collect system metrics and forward them to a central Mimir instance.

This setup replaces or coexists with the previous Prometheus + node_exporter approach, providing a lightweight agent that pushes metrics into Mimir for long-term storage and Grafana dashboards.

** Architecture & Data Flow **

   Lab Nodes
      │
      ▼
node_exporter (on every lab node)
      │
      ▼
Grafana Agent (on every lab node)
      │
      ▼  (remote_write over HTTPS)
   Mimir (central instance)
      │
      ▼
 Grafana Dashboards

Components explained:

1. node_exporter

Installed on every host (prometheus-node-exporter on Debian/Ubuntu, node_exporter on RedHat via EPEL)
Exposes standard host metrics (CPU, memory, disk, network, etc.) on http://localhost:9100/metrics

2. Grafana Agent

Lightweight Prometheus-compatible agent
Scrapes `node_exporter` locally from localhost:9100
Relabels the instance using the host’s FQDN
Forwards all collected metrics to the central Mimir using the remote_write protocol

3. Mimir

Central long-term metrics storage backend (Prometheus-compatible)
Accepts metrics via the remote_write protocol with basic authentication

** Configuration Details **

Grafana Agent config (/etc/grafana-agent.yaml on target hosts):

- Global scrape interval: 60s (configurable via scrape_interval_global) - Scrapes the node job from localhost:9100 - Applies relabeling:

instance: <fqdn>:9100
nodename: <fqdn>

- Forwards metrics to Mimir with basic auth credentials stored in the secrets repository(in mimir_password.yml):

** Mimir Endpoint **

https://mimir-jenkins-monitoring.apps.pok.os.sepia.ceph.com/api/v1/push