This is an old revision of the document!
Grafana Agent Setup for Lab Nodes Monitoring
### Overview
The grafana_agent Ansible role deploys Grafana Agent (static mode) on Lab Nodes to collect system metrics and forward them to a central Mimir instance.
This setup replaces or coexists with the previous Prometheus + node_exporter approach, providing a lightweight agent that pushes metrics into Mimir for long-term storage and Grafana dashboards.
### Architecture & Data Flow
Lab Nodes
│
▼
node_exporter (on every lab node)
│
▼
Grafana Agent (on every lab node)
│
▼ (remote_write over HTTPS)
Mimir (central instance)
│
▼
Grafana Dashboards
Components explained:
1. node_exporter
- Installed on every host (`prometheus-node-exporter` on Debian/Ubuntu, `node_exporter` on RedHat via EPEL)
- Exposes standard host metrics (CPU, memory, disk, network, etc.) on `http://localhost:9100/metrics`
2. Grafana Agent
- Lightweight Prometheus-compatible agent
- Scrapes `node_exporter` locally from `localhost:9100`
- Relabels the instance using the host’s FQDN
- Forwards all collected metrics to the central Mimir using the `remote_write` protocol
3. Mimir
- Central long-term metrics storage backend (Prometheus-compatible)
- Accepts metrics via the remote_write protocol with basic authentication
### Configuration Details
Grafana Agent config (`/etc/grafana-agent.yaml` on target hosts):
- Global scrape interval: `60s` (configurable via `scrape_interval_global`) - Scrapes the `node` job from `localhost:9100` - Applies relabeling:
- `instance`: `<fqdn>:9100`
- `nodename`: `<fqdn>`
- Forwards metrics to Mimir with basic auth credentials stored in the secrets repository
Secrets required (in `mimir_password.yml`):
- `mimir_username` - `mimir_password` - Grafana repository URLs (`grafana_apt_repo_key_url`, `grafana_apt_repo_url`, `grafana_rpm_repo_url`, `grafana_rpm_repo_key_url`) - `agent_mimir_url`
### Mimir Endpoint
`https://mimir-jenkins-monitoring.apps.pok.os.sepia.ceph.com/api/v1/push`