Skip to content
Skills Library
DevOps & cloud · Analysis

debug-prod

/debug-prod

Systematic production investigation - pods, logs, metrics, alerts, deployments, Kibana, Slack context. Suggests likely causes from patterns. Use when user says "debug prod", "investigate production issue", or escalated from investigate_alert.

Repository
skills ↗
Licence
Apache-2.0
Last repository change
27.10.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Systematic production investigation - pods, logs, metrics, alerts, deployments, Kibana, Slack context. Suggests likely causes from patterns. Use when user says "debug prod", "investigate production issue", or escalated from investigate_alert.

Triggers from the description

debug prodinvestigate production issue
Original author description

Systematic production investigation - pods, logs, metrics, alerts, deployments, Kibana, Slack context. Suggests likely causes from patterns. Use when user says "debug prod", "investigate production issue", or escalated from investigate_alert.

How it works

  1. Kubeconfig Rules
  2. Inputs
  3. Persona
  4. Workflow
  5. Output
  6. Chains To
Original SKILL.md headings · 124 lines, 447 words
SKILL.md preview
---
name: debug-prod
description: Systematic production investigation - pods, logs, metrics, alerts, deployments, Kibana, Slack context. Suggests likely causes from patterns. Use when user says "debug prod", "investigate production issue", or escalated from investigate_alert.
---

# Debug Production

Deep investigation of production issues in Automation Analytics. Gathers pod status, logs, metrics, alerts, recent deployments, and suggests likely causes.

## Kubeconfig Rules

**NEVER copy kubeconfig files.**

| File               | Environment |
| ------------------ | ----------- |
| `~/.kube/config.s` | Stage       |
| `~/.kube/config.p` | Production  |

Use `--kubeconfig=~/.kube/config.p` for prod.

## Inputs

| Input        | Type   | Default | Purpose                                |
| ------------ | ------ | ------- | -------------------------------------- |
| `namespace`  | string | -       | `main` or `billing` (ask if missing)   |
| `alert_name` | string | -       | Prometheus alert if triggered by alert |
| `pod_filter` | string | -       | Filter pods (e.g., fastapi, processor) |
| `time_range` | string | 1h      | Log search window (15m, 1h, 6h, 24h)   |
| `issue_key`  | string | -       | Jira key to attach session context     |

## Persona

Load **incident** persona (prometheus, alertmanager, kibana, k8s, grafana).

## Workflow

### 1. Bootstrap

- `persona_load("incident")`
- `check_known_issues("kubectl_get_pods")`
- `check_known_issues("kibana_search_logs")`
- If no namespace: ask user (main vs billing)

### 2. Load Patterns

- `memory_read("learned/patterns")` → error_patterns

### 3. Pod Status

- `kubectl_get_pods(namespace, environment="production")`
- `kubectl_top_pods(namespace, environment="production")`
- Parse: CrashLoopBackOff, OOMKilled, Error, Pending, ImagePullBackOff, high restarts

### 4. Events

- `kubectl_get_events(namespace, environment="production")`
- Filter: warning, error, failed, killed, backoff, unhealthy

### 5. Pod Logs

- For unhealthy pods (up to 3): `kubectl_describe_pod`, `kubectl_logs(pod, namespace, tail=500, since=time_range)`
- Extract error/exception/traceback lines

### 6. Alerts

- `alertmanager_alerts(environment="production")`
- If alert_name: search app-interface alert YAML for definition

### 7. Kibana Logs

- `kibana_search_logs(query="error OR exception OR traceback", namespace, environment="production", limit=20)`
- `kibana_get_errors(namespace, environment="production")`
- `kibana_trace_request` — if request_id in summary

### 8. Slack Context

- `persona_load("slack")` → `slack_find_channel`, `slack_list_messages` (team-automation-analytics)
- `persona_load("incident")` — restore

### 9. Metrics

- `prometheus_query` — 5xx error rate
- `prometheus_query_range` — error rate trend, memory trend
- `prometheus_pod_health`, `prometheus_rules`
- `grafana_dashboard_list`, `grafana_dashboard_get`, `prometheus_grafana_link`

### 10. Deployments

- `kubectl_get_deployments`, `kubectl_get(resource="replicasets")`
- Check app-interface CICD config for deployed SHA
- Check namespace config from app-interface

### 11. Knowledge & Code Search

- `code_search(query=error_from_logs, project="automation-analytics-backend")`
- `knowledge_query(project="automation-analytics-backend", persona="devops", section="patterns.deployment")`
- `knowledge_query(..., section="gotchas")`

### 12. Match Patterns

Match pod issues against known_patterns. Add generic suggestions: OOMKilled, CrashLoopBackOff, ImagePullBackOff.

### 13. Attach to Jira

- If `issue_key`: `jira_attach_session(issue_key, include_transcript=true)`

### 14. Log & Update

- `memory_session_log("Debugged production", ...)`
- Update `state/environments`

### 15. Failure Recovery

- k8s "unauthorized" → `kube_login(cluster="prod")`
- "no route to host" → `vpn_connect()`
- Prometheus "connection refused" → VPN/auth
- Kibana 401 → check credentials
- `learn_tool_fix(...)` for patterns

## Output

Report: pod status, resource usage, events, error logs, Kibana errors, metrics, deployments, pattern matches, Grafana/Kibana links, suggested Prometheus queries.

## Chains To

- `hotfix`, `rollout_restart`, `scale_deployment`
- `silence_alert`, `create_jira_issue`, `learn_pattern`

Versions

The database has no other skills named debug-prod.

Database snapshot: 13.11.2026 · package version 75c2213fabe8 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100tierzero-investigate
    The entry point for any production or infrastructure problem — deployment failures, errors, latency regressions, traffic anomalies, alerts, or anything misbehaving in live systems. Use when the user wants to understand what's happening in prod, check the status of an…
  • 100kit-debug-prod
    Investigate production bugs using logs, traces, error trackers, and metrics. Use when the user reports errors, user/session/trace IDs, timeframes, or status codes. Requires connected tools and/or a team `wire-observability.md`.
  • 90investigate-alert
    Quick triage of firing Prometheus alerts. Gets alerts, pod health, events, Kibana logs, known patterns. Escalates to debug_prod for serious issues. Use when user says "what's firing?", "check alerts", "any alerts?", or "investigate alert".
  • 95groundcover-cli
    Invoke the `groundcover` Go CLI to manage Groundcover resources (dashboards, monitors, silences, connected apps, notification routes, API keys, policies, integrations, pipelines, workflows) AND to answer production observability questions by querying logs, traces, metrics, k8s…
  • 84investigate-incident
    Investigate production incidents using Datadog. Use when user says "what's causing this alert", "investigate production incident", "root cause analysis for production error", "debug production issue", "what caused this outage", "analyze production logs", "triage production…
  • 100prod-readiness
    Scans a codebase and config for 24 production readiness items and produces a scored report. Use when the user asks to check production readiness, audit for prod, review operational concerns, scan for health checks, verify deployment readiness, or assess if code is ready for production.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.