**Grafana Agent Setup for Lab Nodes Monitoring**

** Overview **

The grafana_agent Ansible role deploys **Grafana Agent** (static mode) on Lab Nodes to collect system metrics and forward them to a central **Mimir** instance.

This setup replaces or coexists with the previous Prometheus + node_exporter approach, providing a lightweight agent that pushes metrics into Mimir for long-term storage and Grafana dashboards.

** Architecture & Data Flow **


     Lab Nodes
        │
        ▼
  node_exporter (on every lab node)
        │
        ▼
  Grafana Agent (on every lab node)
        │
        ▼  (remote_write over HTTPS)
     Mimir (central instance)
        │
        ▼
   Grafana Dashboards


**Components explained:**

1. **node_exporter**  
   - Installed on every host (prometheus-node-exporter on Debian/Ubuntu, node_exporter on RedHat via EPEL)  
   - Exposes standard host metrics (CPU, memory, disk, network, etc.) on http://localhost:9100/metrics

2. **Grafana Agent**  
   - Lightweight Prometheus-compatible agent  
   - Scrapes node_exporter locally from localhost:9100  
   - Relabels the instance using the host’s FQDN  
   - Forwards all collected metrics to the central Mimir using the remote_write protocol

3. **Mimir**  
   - Central long-term metrics storage backend (Prometheus-compatible)  
   - Accepts metrics via the remote_write protocol with basic authentication

** Configuration Details **

**Grafana Agent config** (/etc/grafana-agent.yaml on target hosts):

- Global scrape interval: 60s (configurable via scrape_interval_global)
- Scrapes the node job from localhost:9100
- Applies relabeling:
  - instance: <fqdn>:9100
  - nodename: <fqdn>
- Forwards metrics to Mimir with basic auth credentials stored in the secrets repository(in mimir_password.yml):

** Mimir Endpoints **
  Internal
  https://mimir-jenkins-monitoring.apps.pok.os.sepia.ceph.com/api/v1/push
  
  External
  https://mimir.ceph.com/api/v1/push