devops-observability
/devops-observability
Use when designing, writing, reviewing or debugging monitoring, logging or tracing - Prometheus (scrape config, service discovery, relabeling, recording and alerting rules, retention, remote write, cardinality), PromQL and histograms, Alertmanager routing/grouping/inhibition/silences, exporters (node, blackbox, cAdvisor, kube-state-metrics), kube-prometheus-stack and Prometheus Operator CRDs (ServiceMonitor, PodMonitor, PrometheusRule), long-term storage choices (Thanos, Mimir, VictoriaMetrics), Grafana provisioning as code, Loki with Grafana Alloy (Promtail is end of life), LogQL, OpenTelemet
- Author
- fusengine ↗
- Repository
- agents ↗
- Licence
- MIT
- Last repository change
- 13.11.2026
Assessment and security
- Files in this package
- 33
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 2
No signals detected. Repository changed on 13.11.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Use when designing, writing, reviewing or debugging monitoring, logging or tracing - Prometheus (scrape config, service discovery, relabeling, recording and alerting rules, retention, remote write, cardinality), PromQL and histograms, Alertmanager routing/grouping/inhibition/silences, exporters (node, blackbox, cAdvisor, kube-state-metrics), kube-prometheus-stack and Prometheus Operator CRDs (ServiceMonitor, PodMonitor, PrometheusRule), long-term storage choices (Thanos, Mimir, VictoriaMetrics), Grafana provisioning as code, Loki with Grafana Alloy (Promtail is end of life), LogQL, OpenTelemet
Original author description
Use when designing, writing, reviewing or debugging monitoring, logging or tracing - Prometheus (scrape config, service discovery, relabeling, recording and alerting rules, retention, remote write, cardinality), PromQL and histograms, Alertmanager routing/grouping/inhibition/silences, exporters (node, blackbox, cAdvisor, kube-state-metrics), kube-prometheus-stack and Prometheus Operator CRDs (ServiceMonitor, PodMonitor, PrometheusRule), long-term storage choices (Thanos, Mimir, VictoriaMetrics), Grafana provisioning as code, Loki with Grafana Alloy (Promtail is end of life), LogQL, OpenTelemet
How it works
- Agent Workflow (MANDATORY)
- Versions (checked 2026-11-10)
- Overview
- Critical Rules
- Architecture
- Reference Guide
- Quick Reference
- Best Practices
SKILL.md preview
---
name: devops-observability
description: Use when designing, writing, reviewing or debugging monitoring, logging or tracing - Prometheus (scrape config, service discovery, relabeling, recording and alerting rules, retention, remote write, cardinality), PromQL and histograms, Alertmanager routing/grouping/inhibition/silences, exporters (node, blackbox, cAdvisor, kube-state-metrics), kube-prometheus-stack and Prometheus Operator CRDs (ServiceMonitor, PodMonitor, PrometheusRule), long-term storage choices (Thanos, Mimir, VictoriaMetrics), Grafana provisioning as code, Loki with Grafana Alloy (Promtail is end of life), LogQL, OpenTelemetry Collector pipelines, sampling, Tempo/Jaeger, alert design and observability troubleshooting. Not for SLO and error-budget method or incident response (devops-sre-ops), Kubernetes workloads (devops-kubernetes), host OS tuning (devops-linux) or SaaS vendor products.
versions:
prometheus: "3.16.0 (2026-11-04); LTS 3.13 (supported to 2027-07-31)"
alertmanager: "0.35.0 (2026-11-09)"
grafana: "13.3.0"
loki: "3.7.9"
grafana-alloy: "1.21.3 (2026-10-28)"
opentelemetry-collector: "0.165.0 (core 1.71.0)"
prometheus-operator: "0.95.1 (2026-11-05); kube-prometheus-stack chart 93.1.1 (main)"
exporters: "node_exporter 1.12.1, blackbox_exporter 0.29.1, cAdvisor 0.60.6, kube-state-metrics 2.20.0"
user-invocable: true
references:
- references/signals-and-method.md
- references/prometheus-config.md
- references/promql-essentials.md
- references/alertmanager.md
- references/exporters.md
- references/prometheus-operator-k8s.md
- references/long-term-storage.md
- references/grafana-as-code.md
- references/logs-loki-alloy.md
- references/logs-other-stacks.md
- references/tracing-opentelemetry.md
- references/alert-design.md
- references/observability-safe-ops.md
- references/troubleshooting-playbook.md
- references/templates/prometheus.yml
- references/templates/file-sd-nodes.yml
- references/templates/blackbox.yml
- references/templates/rules-alerts.yml
- references/templates/rules-recording.yml
- references/templates/alertmanager.yml
- references/templates/servicemonitor.yaml
- references/templates/prometheusrule.yaml
- references/templates/kube-prometheus-stack-values.yaml
- references/templates/grafana-datasources.yaml
- references/templates/grafana-dashboards-provider.yaml
- references/templates/grafana-alerting-routing.yaml
- references/templates/loki-config.yaml
- references/templates/alloy-host.alloy
- references/templates/alloy-k8s-logs.alloy
- references/templates/alloy-otlp.alloy
- references/templates/otel-collector.yaml
- references/templates/docker-compose-monitoring.yaml
related-skills:
- devops-sre-ops
- devops-kubernetes
- devops-k8s-cluster
- devops-gitops-packaging
- devops-docker
- devops-linux
- devops-networking
- devops-storage-backup
- devops-virtualization
---
<objective>
Covers the three observability signals for an IT/DevOps estate: Prometheus metrics
and rules, Alertmanager routing, exporters, the Kubernetes Operator stack,
Grafana as code, Loki logs collected by Grafana Alloy, OpenTelemetry tracing and
sampling, alert design, and troubleshooting the pipeline itself. Does not cover
SLO/error-budget design and incident response (devops-sre-ops), workload
manifests (devops-kubernetes) or SaaS vendor products.
</objective>
# DevOps Observability
## Agent Workflow (MANDATORY)
Before ANY monitoring config or change, spawn in parallel, one `Agent` call each with a `name`:
1. **fuse-ai-pilot:explore-codebase** - find existing `prometheus.yml`, rule files, Alertmanager/Loki/Alloy/Collector configs, Helm values, dashboards, ServiceMonitors
2. **fuse-ai-pilot:research-expert** - verify option names for the deployed version at prometheus.io, grafana.com/docs, opentelemetry.io (the OpenTelemetry Collector renames components between 0.x releases)
3. **mcp__context7__query-docs** - Prometheus, Grafana, Loki, OpenTelemetry docs
Then discover read-only (running versions, `promtool check`, `amtool check-config`, `alloy validate`). After writing files, run **fuse-ai-pilot:sniper**. Live changes follow the agent's Operations Safety Protocol and [observability-safe-ops.md](references/observability-safe-ops.md).
## Versions (checked 2026-11-10)
Prometheus 3.16.0 is latest; **LTS 3.13** is the conservative choice. Alertmanager 0.35.0 has breaking flag/config changes (event recorder outputs need a `name`). **Promtail reached end of life on 2026-03-02**: collect logs with Grafana Alloy. The OpenTelemetry Collector renamed `otlp` to `otlp_grpc`, `otlphttp` to `otlp_http`, and added a `queue_batch` processor meant to replace `batch`; read its changelog before every 0.x bump.
## Overview
| Need | Decision in one line |
|------|----------------------|
| **What to instrument** | USE for resources, RED for services, golden signals at the edge; plus a blackbox probe of every user-facing URL |
| **Metrics store** | Prometheus (local TSDB, retention by time AND size flags); long-term via remote write to Mimir/Thanos/VictoriaMetrics only when needed |
| **Alerts** | Symptom-based, `for:` + `keep_firing_for`, severity `page`/`ticket`, every alert has `runbook_url`; routing and inhibition in Alertmanager |
| **Kubernetes** | kube-prometheus-stack; ServiceMonitor/PodMonitor/PrometheusRule; check the `release` label selectors and PVCs |
| **Logs** | Loki + Alloy; few static labels, rest in structured metadata; retention by the compactor |
| **Traces** | OpenTelemetry SDK -> Collector/Alloy (OTLP 4317/4318) -> Tempo; tail-sample errors and slow traces |
| **Dashboards/alerts as code** | Grafana provisioning files in Git; metric alerts live in Prometheus, not Grafana |
## Critical Rules
1. **Version first** - read the running versions and the config of the target before writing an option; renames and removals are frequent (OpenTelemetry Collector, Alertmanager, Prometheus 3.x)
2. **Validate offline before any reload** - `promtool check config|rules`, `amtool check-config`, `alloy validate`, `otelcol validate`; reloading production is a proposal in ACTION / EFFECT / PROOF / WAIT-FOR-GO format ([observability-safe-ops.md](references/observability-safe-ops.md))
3. **Cardinality is the budget** - no unbounded label values (user IDs, trace IDs, pod names as Loki labels); use `sample_limit`/`label_limit`, `metric_relabel_configs`, recording rules
4. **Destructive verbs are proposed, not run** - admin-API series deletion, shrinking retention, Loki delete requests, removing PVCs/buckets, mass silences, `helm uninstall`
5. **Persistent storage or it is not monitoring** - emptyDir/no volume loses metrics and silences on restart
6. **Never print secrets** - receivers use `*_file` options; Grafana `secureJsonData`; no tokens in configs or logs
7. **Monitor the monitoring** - Watchdog alert to an external dead man's switch, scrape and notification-failure alerts
8. **Never invent a key or metric name** - cite the page for the version; unknown = "not verified"
## Architecture
```
monitoring/
├── prometheus.yml # global, scrape_configs, alerting, (remote_write)
├── rules/ # alerts and recording rules (promtool-checked)
├── alertmanager.yml # route tree, inhibit_rules, receivers (secrets as files)
├── blackbox.yml # probe modules
├── alloy/*.alloy # logs (Loki), host metrics, OTLP gateway
├── otel-collector.yaml # OpenTelemetry Collector gateway (if used instead of Alloy)
├── loki-config.yaml # storage, schema, compactor retention
└── grafana/
├── provisioning/{datasources,dashboards,alerting}/
└── dashboards/*.json
```
## Reference Guide
### Concepts
| Topic | Reference | When to Consult |
|-------|-----------|-----------------|
| **Signals and method** | [signals-and-method.md](references/signals-and-method.md) | Deciding what to instrument first |
| **Prometheus config** | [prometheus-config.md](references/prometheus-config.md) | Scrape, SD, relabeling, rules, retention, remote write, cardinality |
| **PromQL** | [promql-essentials.md](references/promql-essentials.md) | Queries, histograms, native histograms, pitfalls |
| **Alertmanager** | [alertmanager.md](references/alertmanager.md) | Routing, grouping, inhibition, silences, receivers |
| **Exporters** | [exporters.md](references/exporters.md) | node, blackbox, cAdvisor, kube-state-metrics |
| **Operator / K8s** | [prometheus-operator-k8s.md](references/prometheus-operator-k8s.md) | CRDs, kube-prometheus-stack, missing targets |
| **Long-term storage** | [long-term-storage.md](references/long-term-storage.md) | Thanos, Mimir, VictoriaMetrics pointers |
| **Grafana** | [grafana-as-code.md](references/grafana-as-code.md) | Provisioning datasources, dashboards, alerting |
| **Loki and Alloy** | [logs-loki-alloy.md](references/logs-loki-alloy.md) | Collection, labels, LogQL, retention, Promtail migration |
| **Other log stacks** | [logs-other-stacks.md](references/logs-other-stacks.md) | OpenSearch/Elastic, Vector, Fluent Bit |
| **Tracing** | [tracing-opentelemetry.md](references/tracing-opentelemetry.md) | Collector pipelines, OTLP, sampling, Tempo/Jaeger |
| **Alert design** | [alert-design.md](references/alert-design.md) | Writing and pruning alerts |
| **Safe ops** | [observability-safe-ops.md](references/observability-safe-ops.md) | Any reload, restart, deletion, silence |
| **Troubleshooting** | [troubleshooting-playbook.md](references/troubleshooting-playbook.md) | Missing data, silent alerts, slow or full stores |
### Templates
| Template | When to Use |
|----------|-------------|
| [prometheus.yml](references/templates/prometheus.yml) | Server config with SD, relabeling, blackbox, limits |
| [file-sd-nodes.yml](references/templates/file-sd-nodes.yml) | File-based target list |
| [blackbox.yml](references/templates/blackbox.yml) | Probe modules |
| [rules-alerts.yml](references/templates/rules-alerts.yml) | Symptom alerts with runbook links, Watchdog |
| [rules-recording.yml](references/templates/rules-recording.yml) | Recording rules, level:metric:operations |
| [alertmanager.yml](references/templates/alertmanager.yml) | Route tree, inhibition, time intervals, receivers |
| [servicemonitor.yaml](references/templates/servicemonitor.yaml) | ServiceMonitor + PodMonitor |
| [prometheusrule.yaml](references/templates/prometheusrule.yaml) | PrometheusRule |
| [kube-prometheus-stack-values.yaml](references/templates/kube-prometheus-stack-values.yaml) | Helm values: retention, PVC, selectors |
| [grafana-datasources.yaml](references/templates/grafana-datasources.yaml) | Prometheus, Loki, Tempo, Alertmanager datasources |
| [grafana-dashboards-provider.yaml](references/templates/grafana-dashboards-provider.yaml) | Dashboard file provider |
| [grafana-alerting-routing.yaml](references/templates/grafana-alerting-routing.yaml) | Contact point + notification policy |
| [loki-config.yaml](references/templates/loki-config.yaml) | Single-binary Loki with compactor retention |
| [alloy-host.alloy](references/templates/alloy-host.alloy) | Host metrics + log files |
| [alloy-k8s-logs.alloy](references/templates/alloy-k8s-logs.alloy) | Pod logs on Kubernetes |
| [alloy-otlp.alloy](references/templates/alloy-otlp.alloy) | OTLP gateway in Alloy |
| [otel-collector.yaml](references/templates/otel-collector.yaml) | Collector gateway with tail sampling |
| [docker-compose-monitoring.yaml](references/templates/docker-compose-monitoring.yaml) | Single-host stack |
## Quick Reference
```bash
promtool check config prometheus.yml && promtool check rules rules-*.yml
amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml --tree severity=page team=sre
alloy validate /etc/alloy/config.alloy
otelcol-contrib validate --config=otel-collector.yaml
curl -s http://prometheus:9090/-/ready ; curl -s http://loki:3100/ready
```
## Best Practices
### DO
- Pin image and chart versions; read the release notes of each jump (Alertmanager 0.35, OpenTelemetry 0.x, kube-prometheus-stack majors)
- Alert on user-visible symptoms; link a runbook; group and inhibit in Alertmanager
- Keep dashboards, rules, datasources and Alloy/Collector configs in Git; deliver via GitOps (devops-gitops-packaging)
- Bind monitoring ports to localhost or an authenticated proxy; use TLS between sites
### DON'T
- Enable `--web.enable-admin-api` or `--web.enable-lifecycle` on an exposed port
- Use `irate` in alerts or label metrics/logs with unbounded values
- Run Promtail or a second host-metrics collector next to Alloy/node_exporter
- Duplicate the same alert in Prometheus and Grafana
Files in this package (33)
- SKILL.md 13 KB
- references/alert-design.md 4 KB
- references/alertmanager.md 5 KB
- references/exporters.md 5 KB
- references/grafana-as-code.md 5 KB
- references/logs-loki-alloy.md 7 KB
- references/logs-other-stacks.md 4 KB
- references/long-term-storage.md 4 KB
- references/observability-safe-ops.md 10 KB
- references/prometheus-config.md 7 KB
- references/prometheus-operator-k8s.md 6 KB
- references/promql-essentials.md 5 KB
- references/signals-and-method.md 4 KB
- references/tracing-opentelemetry.md 8 KB
- references/troubleshooting-playbook.md 8 KB
- references/templates/alertmanager.yml 3 KB
- references/templates/alloy-host.alloy 2 KB
- references/templates/alloy-k8s-logs.alloy 2 KB
- references/templates/alloy-otlp.alloy 2 KB
- references/templates/blackbox.yml 1 KB
- references/templates/docker-compose-monitoring.yaml 4 KB
- references/templates/file-sd-nodes.yml 1 KB
- references/templates/grafana-alerting-routing.yaml 1 KB
- references/templates/grafana-dashboards-provider.yaml 1 KB
- references/templates/grafana-datasources.yaml 2 KB
- references/templates/kube-prometheus-stack-values.yaml 3 KB
- references/templates/loki-config.yaml 2 KB
- references/templates/otel-collector.yaml 2 KB
- references/templates/prometheus.yml 3 KB
- references/templates/prometheusrule.yaml 1 KB
- references/templates/rules-alerts.yml 3 KB
- references/templates/rules-recording.yml 1 KB
- references/templates/servicemonitor.yaml 2 KB
Versions
The database has no other skills named devops-observability.
Database snapshot: 13.11.2026 · package version 8c20435c94e1 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90playbook-monitoring-stack
Playbook for deploying, upgrading or extending a self-hosted Kubernetes observability stack — kube-prometheus-stack (Prometheus Operator, Prometheus, Alertmanager, Grafana), Loki, Grafana Alloy as the collector, persistent storage on Longhorn or cloud disks, retention and… - 100prometheus-grafana
Use this skill when the user says 'prometheus grafana setup', 'kube-prometheus-stack', 'servicemonitor', 'expose /metrics', 'promql query', 'prometheus alert rule', 'alertmanager routing', 'grafana dashboard', 'red metrics', 'use method'. Sets up metrics monitoring with… - 100prometheus-and-grafana-monitoring-stack
Guides standing up and operating the Prometheus + Grafana monitoring stack in Kubernetes and hybrid environments — scrape configuration and service discovery, writing PromQL for dashboards and alerts, defining Alertmanager routing/silencing, and provisioning Grafana datasources… - 100k8s-prometheus-troubleshooting
Use when Prometheus on Kubernetes does not collect or alert as expected: a ServiceMonitor or PodMonitor whose targets never appear, targets that are down ("server returned HTTP status 404 Not Found", "connection refused", "context deadline exceeded"), a PrometheusRule that does… - 91prometheus
Prometheus monitoring and alerting for cloud-native observability. Use when implementing metrics collection, PromQL queries, alerting rules, service discovery, recording rules, and scrape config. - 95monitoring
Use this skill when configuring observability and monitoring stack — Prometheus, Grafana, Loki, ELK Stack. This skill enforces: Prometheus scrape configs with relabeling, Grafana dashboard folder hierarchy, Loki log label strategy, Alertmanager routing by severity, SLO tracking…