tasks:grafana_agent_setup_for_lab_nodes_monitoring
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
tasks:grafana_agent_setup_for_lab_nodes_monitoring [2026/04/16 12:08] akraitman |
tasks:grafana_agent_setup_for_lab_nodes_monitoring [2026/05/03 14:31] (current) akraitman |
||
|---|---|---|---|
| Line 4: | Line 4: | ||
| **Grafana Agent Setup for Lab Nodes Monitoring** | **Grafana Agent Setup for Lab Nodes Monitoring** | ||
| - | ### Overview | + | ** Overview ** |
| The grafana_agent Ansible role deploys **Grafana Agent** (static mode) on Lab Nodes to collect system metrics and forward them to a central **Mimir** instance. | The grafana_agent Ansible role deploys **Grafana Agent** (static mode) on Lab Nodes to collect system metrics and forward them to a central **Mimir** instance. | ||
| Line 10: | Line 10: | ||
| This setup replaces or coexists with the previous Prometheus + node_exporter approach, providing a lightweight agent that pushes metrics into Mimir for long-term storage and Grafana dashboards. | This setup replaces or coexists with the previous Prometheus + node_exporter approach, providing a lightweight agent that pushes metrics into Mimir for long-term storage and Grafana dashboards. | ||
| - | ### Architecture & Data Flow | + | ** Architecture & Data Flow ** |
| - | [ Lab Nodes ] | + | Lab Nodes |
| │ | │ | ||
| ▼ | ▼ | ||
| - | node_exporter (on every node) | + | node_exporter (on every lab node) |
| │ | │ | ||
| ▼ | ▼ | ||
| - | Grafana Agent (on every node) | + | Grafana Agent (on every lab node) |
| │ | │ | ||
| ▼ (remote_write over HTTPS) | ▼ (remote_write over HTTPS) | ||
| Line 26: | Line 26: | ||
| ▼ | ▼ | ||
| Grafana Dashboards | Grafana Dashboards | ||
| - | ``` | + | |
| **Components explained:** | **Components explained:** | ||
| 1. **node_exporter** | 1. **node_exporter** | ||
| - | - Installed on every host (`prometheus-node-exporter` on Debian/Ubuntu, `node_exporter` on RedHat via EPEL) | + | - Installed on every host (prometheus-node-exporter on Debian/Ubuntu, node_exporter on RedHat via EPEL) |
| - | - Exposes standard host metrics (CPU, memory, disk, network, etc.) on `http://localhost:9100/metrics` | + | - Exposes standard host metrics (CPU, memory, disk, network, etc.) on http://localhost:9100/metrics |
| 2. **Grafana Agent** | 2. **Grafana Agent** | ||
| - Lightweight Prometheus-compatible agent | - Lightweight Prometheus-compatible agent | ||
| - | - Scrapes `node_exporter` locally from `localhost:9100` | + | - Scrapes node_exporter locally from localhost:9100 |
| - Relabels the instance using the host’s FQDN | - Relabels the instance using the host’s FQDN | ||
| - | - Forwards all collected metrics to the central Mimir using the `remote_write` protocol | + | - Forwards all collected metrics to the central Mimir using the remote_write protocol |
| 3. **Mimir** | 3. **Mimir** | ||
| Line 44: | Line 44: | ||
| - Accepts metrics via the remote_write protocol with basic authentication | - Accepts metrics via the remote_write protocol with basic authentication | ||
| - | ### Configuration Details | + | ** Configuration Details ** |
| - | **Grafana Agent config** (`/etc/grafana-agent.yaml` on target hosts): | + | **Grafana Agent config** (/etc/grafana-agent.yaml on target hosts): |
| - | - Global scrape interval: `60s` (configurable via `scrape_interval_global`) | + | - Global scrape interval: 60s (configurable via scrape_interval_global) |
| - | - Scrapes the `node` job from `localhost:9100` | + | - Scrapes the node job from localhost:9100 |
| - Applies relabeling: | - Applies relabeling: | ||
| - | - `instance`: `<fqdn>:9100` | + | - instance: <fqdn>:9100 |
| - | - `nodename`: `<fqdn>` | + | - nodename: <fqdn> |
| - | - Forwards metrics to Mimir with basic auth credentials stored in the secrets repository | + | - Forwards metrics to Mimir with basic auth credentials stored in the secrets repository(in mimir_password.yml): |
| - | + | ||
| - | **Secrets required** (in `mimir_password.yml`): | + | |
| - | + | ||
| - | - `mimir_username` | + | |
| - | - `mimir_password` | + | |
| - | - Grafana repository URLs (`grafana_apt_repo_key_url`, `grafana_apt_repo_url`, `grafana_rpm_repo_url`, `grafana_rpm_repo_key_url`) | + | |
| - | - `agent_mimir_url` | + | |
| - | + | ||
| - | ### Mimir Endpoint | + | |
| - | `https://mimir-jenkins-monitoring.apps.pok.os.sepia.ceph.com/api/v1/push` | + | ** Mimir Endpoints ** |
| + | Internal | ||
| + | https://mimir-jenkins-monitoring.apps.pok.os.sepia.ceph.com/api/v1/push | ||
| + | |||
| + | External | ||
| + | https://mimir.ceph.com/api/v1/push | ||
tasks/grafana_agent_setup_for_lab_nodes_monitoring.1776341335.txt.gz · Last modified: 2026/04/16 12:08 by akraitman
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
