debug-prod
/debug-prod
Systematic production investigation - pods, logs, metrics, alerts, deployments, Kibana, Slack context. Suggests likely causes from patterns. Use when user says "debug prod", "investigate production issue", or escalated from investigate_alert.
- Author
- dmzoneill ↗
- Repository
- skills ↗
- Licence
- Apache-2.0
- Last repository change
- 27.10.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Systematic production investigation - pods, logs, metrics, alerts, deployments, Kibana, Slack context. Suggests likely causes from patterns. Use when user says "debug prod", "investigate production issue", or escalated from investigate_alert.
Triggers from the description
Original author description
Systematic production investigation - pods, logs, metrics, alerts, deployments, Kibana, Slack context. Suggests likely causes from patterns. Use when user says "debug prod", "investigate production issue", or escalated from investigate_alert.
How it works
- Kubeconfig Rules
- Inputs
- Persona
- Workflow
- Output
- Chains To
SKILL.md preview
---
name: debug-prod
description: Systematic production investigation - pods, logs, metrics, alerts, deployments, Kibana, Slack context. Suggests likely causes from patterns. Use when user says "debug prod", "investigate production issue", or escalated from investigate_alert.
---
# Debug Production
Deep investigation of production issues in Automation Analytics. Gathers pod status, logs, metrics, alerts, recent deployments, and suggests likely causes.
## Kubeconfig Rules
**NEVER copy kubeconfig files.**
| File | Environment |
| ------------------ | ----------- |
| `~/.kube/config.s` | Stage |
| `~/.kube/config.p` | Production |
Use `--kubeconfig=~/.kube/config.p` for prod.
## Inputs
| Input | Type | Default | Purpose |
| ------------ | ------ | ------- | -------------------------------------- |
| `namespace` | string | - | `main` or `billing` (ask if missing) |
| `alert_name` | string | - | Prometheus alert if triggered by alert |
| `pod_filter` | string | - | Filter pods (e.g., fastapi, processor) |
| `time_range` | string | 1h | Log search window (15m, 1h, 6h, 24h) |
| `issue_key` | string | - | Jira key to attach session context |
## Persona
Load **incident** persona (prometheus, alertmanager, kibana, k8s, grafana).
## Workflow
### 1. Bootstrap
- `persona_load("incident")`
- `check_known_issues("kubectl_get_pods")`
- `check_known_issues("kibana_search_logs")`
- If no namespace: ask user (main vs billing)
### 2. Load Patterns
- `memory_read("learned/patterns")` → error_patterns
### 3. Pod Status
- `kubectl_get_pods(namespace, environment="production")`
- `kubectl_top_pods(namespace, environment="production")`
- Parse: CrashLoopBackOff, OOMKilled, Error, Pending, ImagePullBackOff, high restarts
### 4. Events
- `kubectl_get_events(namespace, environment="production")`
- Filter: warning, error, failed, killed, backoff, unhealthy
### 5. Pod Logs
- For unhealthy pods (up to 3): `kubectl_describe_pod`, `kubectl_logs(pod, namespace, tail=500, since=time_range)`
- Extract error/exception/traceback lines
### 6. Alerts
- `alertmanager_alerts(environment="production")`
- If alert_name: search app-interface alert YAML for definition
### 7. Kibana Logs
- `kibana_search_logs(query="error OR exception OR traceback", namespace, environment="production", limit=20)`
- `kibana_get_errors(namespace, environment="production")`
- `kibana_trace_request` — if request_id in summary
### 8. Slack Context
- `persona_load("slack")` → `slack_find_channel`, `slack_list_messages` (team-automation-analytics)
- `persona_load("incident")` — restore
### 9. Metrics
- `prometheus_query` — 5xx error rate
- `prometheus_query_range` — error rate trend, memory trend
- `prometheus_pod_health`, `prometheus_rules`
- `grafana_dashboard_list`, `grafana_dashboard_get`, `prometheus_grafana_link`
### 10. Deployments
- `kubectl_get_deployments`, `kubectl_get(resource="replicasets")`
- Check app-interface CICD config for deployed SHA
- Check namespace config from app-interface
### 11. Knowledge & Code Search
- `code_search(query=error_from_logs, project="automation-analytics-backend")`
- `knowledge_query(project="automation-analytics-backend", persona="devops", section="patterns.deployment")`
- `knowledge_query(..., section="gotchas")`
### 12. Match Patterns
Match pod issues against known_patterns. Add generic suggestions: OOMKilled, CrashLoopBackOff, ImagePullBackOff.
### 13. Attach to Jira
- If `issue_key`: `jira_attach_session(issue_key, include_transcript=true)`
### 14. Log & Update
- `memory_session_log("Debugged production", ...)`
- Update `state/environments`
### 15. Failure Recovery
- k8s "unauthorized" → `kube_login(cluster="prod")`
- "no route to host" → `vpn_connect()`
- Prometheus "connection refused" → VPN/auth
- Kibana 401 → check credentials
- `learn_tool_fix(...)` for patterns
## Output
Report: pod status, resource usage, events, error logs, Kibana errors, metrics, deployments, pattern matches, Grafana/Kibana links, suggested Prometheus queries.
## Chains To
- `hotfix`, `rollout_restart`, `scale_deployment`
- `silence_alert`, `create_jira_issue`, `learn_pattern`
Versions
The database has no other skills named debug-prod.
Database snapshot: 13.11.2026 · package version 75c2213fabe8 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100tierzero-investigate
The entry point for any production or infrastructure problem — deployment failures, errors, latency regressions, traffic anomalies, alerts, or anything misbehaving in live systems. Use when the user wants to understand what's happening in prod, check the status of an… - 100kit-debug-prod
Investigate production bugs using logs, traces, error trackers, and metrics. Use when the user reports errors, user/session/trace IDs, timeframes, or status codes. Requires connected tools and/or a team `wire-observability.md`. - 90investigate-alert
Quick triage of firing Prometheus alerts. Gets alerts, pod health, events, Kibana logs, known patterns. Escalates to debug_prod for serious issues. Use when user says "what's firing?", "check alerts", "any alerts?", or "investigate alert". - 95groundcover-cli
Invoke the `groundcover` Go CLI to manage Groundcover resources (dashboards, monitors, silences, connected apps, notification routes, API keys, policies, integrations, pipelines, workflows) AND to answer production observability questions by querying logs, traces, metrics, k8s… - 84investigate-incident
Investigate production incidents using Datadog. Use when user says "what's causing this alert", "investigate production incident", "root cause analysis for production error", "debug production issue", "what caused this outage", "analyze production logs", "triage production… - 100prod-readiness
Scans a codebase and config for 24 production readiness items and produces a scored report. Use when the user asks to check production readiness, audit for prod, review operational concerns, scan for health checks, verify deployment readiness, or assess if code is ready for production.