Skip to content
Skills Library
DevOps & cloud · Learning & explanation

prometheus

/prometheus

Expert Prometheus monitoring assistance covering PromQL, metric types (Counter, Gauge, Histogram), scrape configs, and Alertmanager. Use when collecting and querying operational metrics across microservices.

Licence
MIT
Last repository change
01.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
2
Deprecation

No signals detected. Repository changed on 01.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Expert Prometheus monitoring assistance covering PromQL, metric types (Counter, Gauge, Histogram), scrape configs, and Alertmanager. Use when collecting and querying operational metrics across microservices.

Original author description

Expert Prometheus monitoring assistance covering PromQL, metric types (Counter, Gauge, Histogram), scrape configs, and Alertmanager. Use when collecting and querying operational metrics across microservices.

How it works

  1. When to Use
  2. Quick Start
  3. Core Concepts
  4. Common Patterns
  5. Best Practices
  6. Troubleshooting
  7. References
Original SKILL.md headings · 170 lines, 582 words
SKILL.md preview
---
name: prometheus
description: Expert Prometheus monitoring assistance covering PromQL, metric types (Counter, Gauge, Histogram), scrape configs, and Alertmanager. Use when collecting and querying operational metrics across microservices.
---

# Prometheus

Prometheus is the CNCF graduated monitoring and alerting toolkit, featuring a multi-dimensional data model with time-series metrics, PromQL, and native OpenTelemetry ingestion.

## When to Use

- **Cloud-Native Time-Series Monitoring**: Scraping, storing, and querying numerical metrics from infrastructure and applications.
- **PromQL Quantitative Querying**: Computing request rates, 99th percentile latencies, error ratios, and saturations.
- **Automated Service Discovery**: Dynamically discovering Kubernetes pods, EC2 instances, and Consul nodes to scrape.
- **Alerting Rules & Alertmanager Integration**: Evaluating threshold rules and dispatching alerts to on-call engineers.

## Quick Start

```yaml
# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]
```

## Core Concepts

### Prometheus Scrape Configuration (prometheus.yml)

Configuring scrape jobs with relabeling:

```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "api-microservices"
    scrape_interval: 10s
    metrics_path: "/metrics"
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: ["production"]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_container_port_number]
        action: keep
        regex: "8080"
```

### Production Alerting Rules (alert_rules.yml)

Defining SLO-based alert thresholds:

```yaml
groups:
  - name: API_SLO_Alerts
    rules:
      - alert: HighHttpErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) * 100 > 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High HTTP 5xx error rate on {{ $labels.service }}"
          description: 'Service {{ $labels.service }} error rate is {{ $value | printf "%.2f" }}% (> 5% SLO threshold).'
```

### Exposing Custom Metrics in Python

Instrumentation using official Prometheus Python client:

```python
from prometheus_client import start_http_server, Counter, Histogram
import time

REQUEST_COUNTER = Counter(
    'http_requests_total',
    'Total count of HTTP requests processed',
    ['method', 'endpoint', 'status']
)

REQUEST_DURATION = Histogram(
    'http_request_duration_seconds',
    'Histogram of HTTP request latency in seconds',
    ['endpoint'],
    buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0)
)

@REQUEST_DURATION.labels(endpoint='/api/orders').time()
def process_order():
    time.sleep(0.08)
    REQUEST_COUNTER.labels(method='POST', endpoint='/api/orders', status='200').inc()

# Start metrics endpoint server on port 8000
start_http_server(8000)
```

## Common Patterns

### High-Throughput Scrape Config with Kubernetes Service Discovery

**Problem**: Static IP scraping configurations fail as pods scale dynamically in Kubernetes.

**Solution**:
Use `kubernetes_sd_configs` with metric path annotations:

```yaml
scrape_configs:
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels:
          [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
```

## Best Practices

**Do**:

- Adhere strictly to the Four Golden Signals (Latency, Traffic, Errors, Saturation) when building PromQL alert rules.
- Configure appropriate histogram buckets around your service SLO targets (e.g. 100ms, 200ms, 500ms).
- Use recording rules (`record: job:metric:rate5m`) for complex PromQL expressions queried by dashboards.
- Store long-term historical metrics using remote-write storage (Thanos, Cortex, or VictoriaMetrics).

**Don't**:

- Introduce high-cardinality label values (user IDs, email addresses, order IDs) into metric labels.
- Set scrape intervals too low (< 5s) across large fleets; it overloads target scrapers and storage.
- Alert on simple transient spikes; always use `for: 2m` or `for: 5m` to filter temporary noise.

## Troubleshooting

| Error                                        | Cause                                                                             | Solution                                                               |
| :------------------------------------------- | :-------------------------------------------------------------------------------- | :--------------------------------------------------------------------- |
| `Target shows DOWN in Prometheus Targets UI` | Scrape target port firewalled or application metrics path crashing.               | Test target endpoint manually: `curl http://<pod-ip>:<port>/metrics`.  |
| `High memory usage / Prometheus OOM`         | High churn in metric series (high cardinality tags like UUIDs or user emails).    | Audit and drop unbounded labels using `metric_relabel_configs`.        |
| `PromQL: many-to-many matching not allowed`  | Vector matching join (`on(...)`) without `group_left` or `group_right` modifiers. | Add `group_left` or `group_right` to specify many-to-one relationship. |

## References

- [Prometheus Documentation](https://prometheus.io/)

Versions

RepositoryScoreLicence
This page · G1Joshi/Agent-Skills100MIT
xobotyi/cc-foundry100MIT
grafana/skills100Apache-2.0
julianobarbosa/claude-code-skills100MIT
dkbnull/hello-skill96MIT
antgroup/agent3sigma-canary95Apache-2.0
CloudChef/atlasclaw-providers91Apache-2.0
wilsonchenghy/prometheus-pcb-harness86Apache-2.0

Database snapshot: 13.11.2026 · package version ecc6ae904b98 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100prometheus-grafana
    Use this skill when the user says 'prometheus grafana setup', 'kube-prometheus-stack', 'servicemonitor', 'expose /metrics', 'promql query', 'prometheus alert rule', 'alertmanager routing', 'grafana dashboard', 'red metrics', 'use method'. Sets up metrics monitoring with…
  • 100prometheus-grafana
    Set up metrics collection and visualization with Prometheus and Grafana. Configure scrape targets, create PromQL queries, build dashboards, and implement alerting. Use when implementing monitoring, metrics collection, or visualization for applications and infrastructure.
  • 100prometheus-expert
    Expert-level Prometheus monitoring, metrics collection, PromQL queries, alerting, and production operations. Use when the user mentions monitoring, metrics, observability, alerting, or PromQL, or when the task involves Prometheus Architecture, Installation on Kubernetes,…
  • 100prometheus-analysis
    This skill should be used when the user asks to "query Prometheus", "analyze Prometheus metrics", "check Prometheus alerts", "write PromQL", "interpret Prometheus data", "fetch metrics", or mentions Prometheus querying, alerting, or metrics analysis. Provides guidance for…
  • 100prometheus-and-grafana-monitoring-stack
    Guides standing up and operating the Prometheus + Grafana monitoring stack in Kubernetes and hybrid environments — scrape configuration and service discovery, writing PromQL for dashboards and alerts, defining Alertmanager routing/silencing, and provisioning Grafana datasources…
  • 100add-metric
    TRIGGER when user asks to add or modify a metric, counter, gauge, or histogram, or to track/measure an operation.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.