alerting

---
name: alerting
description: >
  Use this skill when the user asks about alert rules, Alertmanager, Prometheus
  alerts, Grafana alerts, notification routing, PagerDuty, on-call, escalation,
  alert fatigue, or threshold tuning.
version: "1.0.0"
author: "j4flmao"
license: "MIT"
compatibility:
  claude-code: true
  cursor: true
  codex: true
  windsuf: true
tags: [management, alerting, phase-8]
---

# Alert Rules

## Purpose
Design alert rules following the RED/USE method with severity classification, threshold tuning, notification routing, and escalation paths. Ensure every alert is actionable, documented with a runbook, and tuned to balance detection speed against fatigue.

## Framework and Methodology

### Alerting Philosophy
Four principles guide alert rule design:

1. **Actionability** — if no action can be taken, it belongs on a dashboard, not as an alert.
2. **Signal over noise** — every firing alert represents a real issue requiring human intervention.
3. **Tiered response** — severity dictates speed of response, not importance of the metric.
4. **Continuous tuning** — thresholds degrade over time as systems evolve; review monthly.

### RED Method (Applications)
Rate, Errors, Duration — the three golden signals for application monitoring.

```
Rate: How much traffic is flowing through the system?
  Query: rate(http_requests_total[5m])
  Alert: sudden drop (possible outage) or unexpected surge (traffic anomaly)

Errors: What fraction of requests are failing?
  Query: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
  Alert: > 1% warning, > 5% critical

Duration: How long do requests take?
  Query: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
  Alert: p99 > 500ms warning, > 2s critical
```

### USE Method (Infrastructure)
Utilization, Saturation, Errors — resource-focused monitoring.

```
Utilization: What percentage of capacity is being used?
  CPU: node_cpu_seconds_total{mode="idle"} < 0.1
  Memory: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
  Disk: node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.1

Saturation: How much extra work is queued?
  CPU run queue length > 4 * core count
  Disk I/O queue depth > disk I/O capacity
  Network interface drops > 0

Errors: How many errors are occurring?
  OOM kills: increase(node_vmstat_oom_kill[5m]) > 0
  Disk errors: increase(node_disk_io_time_seconds_total[5m]) > 0
  Network errors: increase(node_network_transmit_errors_total[5m]) > 0
```

### Business Metrics
Domain-specific alerts on revenue-impacting signals.

```
- Order failure rate > 10%
- Payment processing delay > 30s p95
- User signup completion rate < 90%
- API integration partner error rate > 5%
- Cart abandonment rate spike > 20% above baseline
```

### Alert Fatigue Calculation
```
Alert Fatigue Risk = (Total Alerts / Week) / (Actionable Alerts / Week)

  Healthy: < 3 alerts per actionable alert
  Warning: 3-10 alerts per actionable (investigate thresholds)
  Critical: > 10 alerts per actionable (urgent tuning needed)

  Target: < 5 total alerts per on-call person per shift
  Maximum: 10 alerts per shift before burnout risk
```

### Decision Tree: Alert or Dashboard?

```
Will someone take immediate action when this fires?
  ├── Yes → What action?
  │   ├── Rollback, restart, failover
  │   │   └── P0/P1 alert with runbook
  │   ├── Investigate during business hours
  │   │   └── P2 alert with runbook
  │   └── Investigate when someone has time
  │       └── P3 alert or ticket, not pager
  └── No → Is it useful for post-mortem analysis?
      ├── Yes → Dashboard panel + log
      └── No → Delete the metric or reduce collection frequency
```

## Agent Protocol

### Trigger
User request includes: `alert rule`, `alertmanager`, `prometheus alert`, `grafana alert`, `notification`, `pagerduty`, `on-call`, `escalation`, `alert fatigue`, `threshold`.

### Input Context
- Monitoring platform (Prometheus, Grafana, ELK, Datadog, New Relic)
- On-call tool (PagerDuty, Opsgenie, Slack)
- Current alert volume and known pain points
- Service criticality tiers

### Output Artifact
A markdown document containing:
- Alert severity definitions with response SLAs
- Alert rule templates per signal type (latency, errors, saturation, throughput)
- Threshold tuning guidelines (avoid alert fatigue)
- Notification routing rules (severity to channel to escalation)
- Silence/maintenance window policy
- Runbook template for common alerts

### Response Format
Produce the artifact directly. No preamble. No postamble. No explanations.

### Completion Criteria
- Severity levels defined with clear escalation paths
- Alert rules cover USE/RED method
- Thresholds tuned with tuning guidelines
- Notification routing defined per severity
- Runbook template included

### Max Response Length
4096 tokens

## Workflow

### Step 1: Define Severity Levels
Map severity (P0-P3) to response SLA, notification channels, and escalation paths.

```
P0: Critical -- 5 min acknowledge, 1 hr fix -- Phone + Slack + PagerDuty
     Escalation: EM -> CTO -- Example: total service outage, data loss
P1: High -- 15 min acknowledge, 4 hr fix -- Slack + PagerDuty
     Escalation: Team lead -- Example: high error rate, degraded performance
P2: Warning -- 1 hr acknowledge, 24 hr fix -- Slack
     Escalation: None -- Example: disk usage rising, latency increasing
P3: Info -- Next business day -- Dashboard only
     Escalation: None -- Example: certificate expiring in 30 days
```

### Step 2: Create RED Method Alerts
Define rate, error, and duration alerts for application services.

```
HighErrorRate: rate(5xx) / rate(total) > 0.05 for 5m -> P1
HighLatency: p99 latency > 2s for 5m -> P1
ZeroTraffic: rate(requests) == 0 for 5m -> P0
HighTrafficSurge: rate(requests) > 3x baseline for 10m -> P2
```

### Step 3: Create USE Method Alerts
Define utilization, saturation, and error alerts for infrastructure resources.

```
HighCpuUsage: idle < 10% for 10m -> P2
DiskSpaceRunningOut: avail < 10% for 5m -> P2
HighMemoryUsage: avail < 10% for 5m -> P2
OOMKillDetected: oom_kill > 0 -> P1
DiskIOLatency: avg_queue_latency > 100ms for 10m -> P2
NetworkPacketLoss: tx_errors > 5% for 5m -> P2
```

### Step 4: Add Business Alerts
Create domain-specific alerts for order failures, payment delays, and other business metrics.

```
OrderFailureRate: failure rate > 10% for 10m -> P1
PaymentProcessingDelay: p95 > 30s for 5m -> P1
SignupCompletionDrop: completion rate < 90% for 15m -> P2
```

### Step 5: Tune Thresholds
Apply threshold tuning rules to balance detection speed against alert fatigue.

```
- Start with conservative thresholds (higher tolerance).
- Review monthly: are alerts firing? Are they actionable?
- Adjust based on: false positive rate, detection latency, business impact.
- Use seasonal baselines: peak vs off-peak thresholds.
- Implement dynamic thresholds for predictable traffic patterns.
```

### Step 6: Configure Notification Routing
Map severity to notification channels with escalation matrix.

```
P0 -> Phone + Slack + PagerDuty -> EM after 5 min no ack -> CTO after 15 min
P1 -> Slack + PagerDuty -> Team lead after 15 min no ack
P2 -> Slack channel -> Escalate after 1 hr if no response
P3 -> Dashboard + email digest -> No escalation
```

### Step 7: Create Runbooks
Every alert must have a documented runbook with diagnosis and resolution steps.

```
Runbook template per alert:
  1. Symptom description
  2. Possible causes (ordered by likelihood)
  3. Diagnostic steps (commands to run, metrics to check)
  4. Resolution steps
  5. Verification procedure
  6. Escalation path
```

### Step 8: Implement Maintenance Windows
Define policy for silencing alerts during planned operations.

```
- Pre-approved maintenance window: silence P2/P3 alerts automatically.
- Post notification in on-call channel before starting.
- Maximum window: 4 hours (extend only with manager approval).
- Auto-expire: alerts resume when window ends.
- Exception: P0 alerts never silenced during maintenance.
```

### Step 9: Monitor and Improve
Track alert health metrics and implement improvements.

```
- Alert volume per severity per service (target: < 5 P0/month per service).
- Time-to-acknowledge per severity (target: < 5 min P0).
- False positive rate (target: < 20%).
- MTTA (mean time to acknowledge) per team.
- MTTR (mean time to resolve) per alert type.
```

### Step 10: On-Call Rotation Design
Design sustainable on-call rotations to prevent burnout.

```
- Team size for on-call: minimum 4 people for weekly rotations
- Rotation length: 1 week (shorter = better, 1 day for high-volume teams)
- Secondary on-call: always have a backup
- Follow-the-sun: preferred for global teams
- Compensation: time off in lieu or on-call pay
- Handoff: scheduled 30-min overlap on rotation day
- Escalation path documented in the rotation schedule
```

## Common Pitfalls

1. **Alert fatigue from oversensitive thresholds**: Start conservative and tighten gradually.
2. **Cascading alerts without grouping**: One root cause fires 20 related alerts. Use alert grouping and inhibition rules.
3. **Missing runbooks**: Teams waste time diagnosing without documented procedures.
4. **P0 alerts for non-urgent issues**: Devalues severity system. Reserve P0 for actual critical failures.
5. **Noisy infrastructure alerts**: CPU spikes that self-resolve add noise. Use longer `for:` durations.
6. **Thresholds never reviewed**: Six-month-old thresholds are irrelevant for growing systems.
7. **Missing `for:` duration**: Single data point flaps cause false alerts.
8. **Ignoring cardinality**: Labels with unbounded values break Prometheus.
9. **Alerting on symptoms, not causes**: Alert on high latency, not on high CPU that causes high latency.
10. **No alert on silence**: Silent alert system = broken monitoring. Set up Deadman's switch.
11. **All alerts go to pager**: Every alert is P2+. Fix: route P3+ to dashboard, only page for P0-P1.
12. **No correlation between alerts**: Same incident fires 5 different alert rules. Fix: use alert grouping and composite rules.

## Best Practices

- Every alert must have a runbook linked in its annotation
- Use `for:` duration equal to at least 2-3 scrape intervals
- Group related alerts into a single notification to reduce noise
- Define severity by impact, not by metric type
- Label every alert with team, service, environment, and severity
- Test alerts in staging before deploying to production
- Implement Deadman's switch: alert if no data received for expected interval
- Review alert rules during monthly SRE retrospectives
- Track and trend alert volume by service
- Rotate on-call frequently to reduce burnout
- Set up different routing for business hours vs after hours
- Use composite alerts to reduce noise from correlated symptoms

## Compared With

| Approach | Strengths | Weaknesses |
|---|---|---|
| RED method (this skill) | Covers user-facing signals, simple | Misses infrastructure saturation |
| USE method (this skill) | Covers resource health | Needs RED for full picture |
| Four golden signals | Complete coverage (RED + USE) | More alerts to manage |
| Single metric alerts | Simple to implement | Misses composite conditions |
| Anomaly detection | Catches unknown unknowns | High false positive rate |
| SLO-based alerting | Business-aligned | Requires SLO definition effort |
| Log-based alerting | Deep context | High volume, slow |
| Tracing-based alerting | Precise root cause | High overhead, sampling tradeoff |

## Templates and Tools

### Alert Rule Template (Prometheus)
```yaml
alert: {{ALERT_NAME}}
expr: {{PROMQL_EXPRESSION}}
for: {{DURATION}}
labels:
  severity: {{P0|P1|P2|P3}}
  team: {{TEAM_NAME}}
  service: {{SERVICE_NAME}}
annotations:
  summary: "{{BRIEF_DESCRIPTION}}"
  description: "{{DETAILED_DESCRIPTION}}"
  runbook: "https://runbooks.team.com/{{ALERT_NAME}}"
```

### Alertmanager Configuration Template
```yaml
route:
  receiver: "default"
  routes:
    - match: {severity: "P0"}
      receiver: "pagerduty-critical"
      repeat_interval: 5m
    - match: {severity: "P1"}
      receiver: "pagerduty-high"
      repeat_interval: 30m
    - match: {severity: "P2"}
      receiver: "slack-warnings"
      repeat_interval: 4h

receivers:
  - name: "pagerduty-critical"
    pagerduty_configs:
      - routing_key: "{{PAGERDUTY_KEY}}"
        severity: "critical"
  - name: "slack-warnings"
    slack_configs:
      - channel: "#alerts"
        title: "{{ .GroupLabels.alertname }}"
```

### Alert Health Dashboard
```
Metric              | Current | Target | Status
Total alerts/week   | 42      | < 25   | Needs tuning
P0 alerts/month     | 3       | < 5    | Healthy
False positive rate | 28%     | < 20%  | Review thresholds
MTTA P0             | 3 min   | < 5 min| Healthy
MTTR P0             | 25 min  | < 60 min| Healthy
Alerts per on-call  | 6/shift | < 5    | Near limit
```

### On-Call Rotation Template
```
Team: {name} | Week: {date} - {date}
Primary: {name} (acknowledge within 5 min)
Secondary: {name} (backup if primary unresponsive)
Handoff scheduled: Monday 09:00 (30 min overlap)

Escalation path:
  Primary unresponsive > 5 min → Secondary
  Secondary unresponsive > 5 min → Engineering Manager
  EM unresponsive > 5 min → CTO
```

## Rules
- Every alert must be actionable — if no action can be taken, it is a dashboard metric
- No alert without a runbook — every alert routes to a document explaining diagnosis and fix
- No duplicate alerts — group related alerts and suppress cascade alerts
- Require minimum `for:` duration before firing to prevent flapping
- Review and adjust thresholds monthly as part of seasonal tuning
- Scheduled maintenance periods must automatically silence alerts
- Alert labels must have bounded cardinality
- P0 alerts go to phone + Slack + PagerDuty with immediate escalation
- Severity is defined by user impact, not by metric type
- Every alert must have a team label for ownership
- Alerts must be tested in staging before production deployment
- Deadman's switch alert for metric pipelines that stop producing data
- Acknowledge alert within SLA before investigating root cause
- Escalate if first responder does not acknowledge within SLA window
- Monthly alert review as part of incident management improvement
- Auto-resolve alerts when condition clears
- Distinguish between warning (potential issue) and critical (active problem)
- Alert routing must consider time of day and day of week
- On-call shift handover includes alert status review

## References
  - references/alert-rules-catalog.md — Alert Rules Catalog
  - references/alert-rules-templates.md — Alert Rules Templates
  - references/alerting-advanced.md — Alerting Advanced Topics
  - references/alerting-fundamentals.md — Alerting Fundamentals
  - references/escalation-policies.md — Escalation Policies
  - references/notification-routing.md — Notification Routing Reference
  - references/alerting-rule-design.md — Alert Rule Design Patterns
  - references/alerting-oncall-rotation.md — On-Call Rotation Design

## Handoff
Hand off to `devops/monitoring/SKILL.md` for monitoring stack setup.
Hand off to `management/team-rules/SKILL.md` for on-call schedule and incident response.

## Architecture Decision Trees

### Alert Routing
| Decision Point | Option A | Option B | Decision Criteria |
|---|---|---|---|
| Alert channel | Email (archival) | PagerDuty/Opsgenie (real-time) | Urgency, response SLA |
| Notification platform | Built-in (Grafana) | Dedicated (PagerDuty) | Cost vs flexibility, existing stack |
| Escalation strategy | Linear escalation | Tiered/broadcast escalation | Team size, criticality tiers |

### Threshold Definition
- Static threshold: Simple, good for stable systems with known baselines.
- Dynamic/seasonal threshold: Uses ML on historical data. Better for systems with periodic traffic patterns.
- Anomaly-based: Flags deviations from predicted behavior. Ideal for systems without clear fixed thresholds.

## Implementation Patterns

### Alert Rule Template (Prometheus)
`yaml
groups:
  - name: service_alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 3m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error rate on {{ \.service }}"
          description: "Error rate {{ \ | humanizePercentage }} exceeded 5% for 3 minutes"
          runbook: "https://runbooks.internal/error-rate"
`

### Escalation Policy Template
`yaml
escalation_policy:
  name: "Backend On-Call"
  rules:
    - targets: [primary_on_call]
      timeout: 15m
    - targets: [secondary_on_call]
      timeout: 10m
    - targets: [engineering_manager]
      timeout: 5m
    - targets: [incident_command]
      notify: immediate
`

## Production Considerations

### Alert Fatigue Prevention
- **Noise reduction**: Use alert aggregation and deduplication. Silence recurring known issues.
- **Alert severity**: Distinguish critical (page) vs warning (notification). Aim for < 5 pages per on-call shift.
- **Maintenance windows**: Suppress alerts during planned maintenance. Automatically create maintenance windows from deployment schedules.

### Reliability
- **Redundant notification**: Alert through multiple channels (PagerDuty + Slack + SMS). Have fallback if primary channel is down.
- **Heartbeat alerts**: Monitor the monitoring system itself. Alert if alertmanager/prometheus is unreachable.
- **On-call handoff**: Document active incidents during shift handoff. Require 15min overlap for knowledge transfer.

## Anti-Patterns

| Anti-Pattern | Symptom | Solution |
|---|---|---|
| Alerting everything | 500+ alerts/day, all ignored | Alert on symptoms, not causes. Start with < 10 rules |
| No runbook | On-call wakes up and doesn't know what to do | Every alert must link to a runbook |
| Friday evening deploys | Weekend pages | Restrict deploys to business hours |
| Silent threshold increase | Decreased sensitivity masks real issues | Track MTTA/MTTR alongside alert volume |
| No alert testing | False sense of security | Weekly alert simulation drills |

## Performance Optimization

### Response Time
- **Runbook automation**: Embed auto-remediation scripts in runbooks. Use webhook-triggered lambda functions for common fixes.
- **Alert enrichment**: Augment alerts with links to dashboards, logs, and trace IDs. Pre-compute likely impact scope.
- **Intelligent routing**: Route alerts based on current on-call, time of day, and expertise. Use ML to predict best responder.

### Monitoring Efficiency
- **Reduced alert storm**: Implement dependency-aware alerting. Suppress downstream alerts when root cause is identified upstream.
- **Aggregation rules**: Group related alerts into incidents. Deduplicate identical alerts within time windows.
- **Auto-closure**: Auto-resolve alerts when metric returns to normal. Require manual ack only for acknowledged pages.

## Security Considerations

### Authentication & Access
- **Alert modification**: Restrict who can silence/modify alert rules. Audit all alert configuration changes.
- **On-call roster access**: Protect on-call schedules as PII (staff location, rotation patterns). Use SSO with MFA for alerting platforms.
- **Alert channel security**: Use private Slack channels for alert notifications. Never forward alerts to public channels.

### Incident Response Security
- **Alert data leakage**: Sanitize alert payloads to exclude secrets. Mask sensitive values in alert annotations.
- **Secure runbooks**: Host runbooks in authenticated internal wikis. Never embed API keys or passwords in runbook steps.
- **Post-incident review**: Conduct blameless postmortems. Preserve alert data and timelines for forensic analysis.

- **Audit trail maintenance**: Retain alert and incident records per compliance requirements. Archive resolved incidents for trend analysis.
- **On-call shift logging**: Log all on-call actions, escalations, and handoffs. Use time-stamped entries for postmortem reconstruction.
- **Notification delivery confirmation**: Verify all critical alerts are acknowledged. Escalate unacknowledged pages after timeout.
- **Alert volume trending**: Track weekly alert volume per service. Set targets for reducing noise-to-signal ratio.
- **Cross-team incident coordination**: Document handoff procedures between primary and secondary on-call teams. Maintain shared incident timeline.

• SKILL.md 20 KB
• references/alert-rules-catalog.md 2 KB
• references/alert-rules-templates.md 5 KB
• references/alerting-advanced.md 8 KB
• references/alerting-fundamentals.md 7 KB
• references/alerting-oncall-rotation.md 22 KB
• references/alerting-rule-design.md 17 KB
• references/escalation-policies.md 4 KB
• references/notification-routing.md 1 KB