Skip to content
Skills Library
DevOps & cloud · Analysis

production-debugging

/production-debugging

Debug a live system without making the incident worse by staying read-only first, gating changes behind flags, and testing on mirrored traffic. Use when a bug only manifests in production and you must investigate against real users and real data.

Repository
AI-SKILLS ↗
Licence
MIT
Last repository change
05.09.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 05.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Debug a live system without making the incident worse by staying read-only first, gating changes behind flags, and testing on mirrored traffic. Use when a bug only manifests in production and you must investigate against real users and real data.

Original author description

Debug a live system without making the incident worse by staying read-only first, gating changes behind flags, and testing on mirrored traffic. Use when a bug only manifests in production and you must investigate against real users and real data.

How it works

  1. Method
  2. Checks
  3. Boundaries
Original SKILL.md headings · 55 lines, 481 words
SKILL.md preview
---
name: production-debugging
description: Debug a live system without making the incident worse by staying read-only first, gating changes behind flags, and testing on mirrored traffic. Use when a bug only manifests in production and you must investigate against real users and real data.
---

# Production debugging

Production is a running system with real users, and every debugging action
is also a change to that system. The instinct that works locally, edit and
re-run, is the instinct that turns an incident into an outage here. The
discipline is to learn as much as possible while changing as little as
possible, escalating from pure observation to reversible action to writes.

## Method

1. **Read the signals before touching anything.** Pull the logs, traces, and
   dashboards for the failing requests: error rate, latency percentiles, and
   the correlation id of a known-bad request. Most production causes are
   visible in existing telemetry, and reading is the only zero-risk step.
   Exhaust it first.
2. **Reproduce off the hot path.** Replay the failing request against a
   staging replica, a shadow environment, or a mirror of production traffic
   (Envoy/Istio traffic mirroring, `tcpreplay`, GoReplay) so your probing
   hits a copy, not live users. If you must touch production, use read
   replicas and read-only queries with a `LIMIT` and a statement timeout.
3. **Change behavior behind a flag, never behind a deploy.** Gate any fix or
   diagnostic branch behind a feature flag you can flip for one internal
   user, then a small percentage, then everyone. A flag rolls back in
   seconds; a deploy rolls back in minutes and drags unrelated changes with
   it. Test the fix on your own account first.
4. **Add targeted observability instead of guessing.** When the existing
   signals fall short, add a scoped log line or span for the specific code
   path and ship it, or attach a sampling profiler (`py-spy`,
   `async-profiler`, `perf`) that reads a live process without stopping
   it. Never attach a breakpoint debugger to a serving instance: it
   freezes every request on that node.
5. **Isolate the blast radius before you write.** If a write is truly
   required, scope it to one tenant or one record, wrap it in a transaction
   you can roll back, and confirm the row count before commit. Take a backup
   or snapshot first. Announce the change in the incident channel so no one
   else moves the same ground.
6. **Verify with metrics, then remove the scaffolding.** Confirm the fix by
   the same dashboards that showed the fault, not by one lucky manual
   request. Then roll the flag to 100, delete the temporary logging, and
   record the timeline and root cause for the postmortem.

## Checks

- Did every step so far read state, or did some step change it? Know which.
- Can the last change you made be reverted in seconds, without a deploy?
- Is the fix confirmed by a falling error-rate curve, not a single retry?

## Boundaries

This is investigation discipline, not incident command; declaring severity,
paging owners, and coordinating comms follow the team's incident-response
runbook. Data-fix scripts that touch many rows, schema changes, and anything
irreversible need review and a rollback plan, not a live console session.

Versions

RepositoryScoreLicence
This page · Amey-Thakur/AI-SKILLS90MIT
Idfoxes/keelson100Apache-2.0
Weaverse/.agents90MIT

Database snapshot: 13.11.2026 · package version 86c06f3b509d (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 90testing-in-production
    Verify changes against real traffic without risking users, using canaries, shadow traffic, and feature gates with automatic rollback. Use when a behavior cannot be trusted from staging alone and needs production load or data to prove out.
  • 100incident
    Triage a production bug or incident report (log, error, tracker URL, user complaint) without touching production. Classify severity, route to /debug, /fix, /rollback, or /feature, and write an incident record. Use for /incident, production is broken, user reports an error, or…
  • 84investigate-incident
    Investigate production incidents using Datadog. Use when user says "what's causing this alert", "investigate production incident", "root cause analysis for production error", "debug production issue", "what caused this outage", "analyze production logs", "triage production…
  • 90production-incident-diagnostic
    Use when diagnosing a production incident - "prod is down", "la prod est down", an API responding 502/500/slow, abnormal behavior in prod, a user reporting something broken in production, alerting gone silent. Triggers on localizing the faulty layer of a live system.
  • 100production-testing
    When the user wants to design or operate production-side testing — synthetic monitoring, canary analysis, real user monitoring, error-budget-driven testing, shift-right verification. Use when the user mentions "production testing," "testing in production," "TiP," "synthetic…
  • 100qe-shift-right-testing
    Testing in production with feature flags, canary deployments, synthetic monitoring, and chaos engineering. Use when implementing production observability or progressive delivery.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.