agent-reliability
/agent-reliability
Diagnoses AI workflow failures from reproducible evidence, defines operational measures, and prepares a bounded baseline/candidate review with the CTO's agent-evaluation manual. Use when the user says "why does this agent keep failing", "reduce AI rework", "measure handoff reliability", or "is this workflow ready for a pilot".
- Author
- alebgl77 ↗
- Repository
- claude-inc ↗
- Licence
- MIT
- Last repository change
- 22.10.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 2
No signals detected. Repository changed on 22.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Diagnoses AI workflow failures from reproducible evidence, defines operational measures, and prepares a bounded baseline/candidate review with the CTO's agent-evaluation manual. Use when the user says "why does this agent keep failing", "reduce AI rework", "measure handoff reliability", or "is this workflow ready for a pilot".
Triggers from the description
Original author description
Diagnoses AI workflow failures from reproducible evidence, defines operational measures, and prepares a bounded baseline/candidate review with the CTO's agent-evaluation manual. Use when the user says "why does this agent keep failing", "reduce AI rework", "measure handoff reliability", or "is this workflow ready for a pilot".
How it works
- When to use
- Inputs
- Workflow
- Output format
- Quality bar
- Example
- Sources
SKILL.md preview
--- name: agent-reliability description: Diagnoses AI workflow failures from reproducible evidence, defines operational measures, and prepares a bounded baseline/candidate review with the CTO's agent-evaluation manual. Use when the user says "why does this agent keep failing", "reduce AI rework", "measure handoff reliability", or "is this workflow ready for a pilot". --- # Agent Reliability — Senior Agent Reliability Engineer > "Keep the failed cases in the evidence." *Staff skill — owned by the CAIO, pairs with the CTO's agent-evaluation and appsec-review skills.* ## When to use - A workflow loses information, repeats actions, produces rejected artifacts, or fails intermittently. - A process change needs comparable operating evidence before adoption. - An accepted workflow regresses and needs a bounded diagnosis and fallback decision. ## Inputs Obtain the business acceptance criteria, baseline and candidate versions, sanitized task inputs and observed outputs, rejection/incident examples, available usage records, tool/runtime settings, and authorized trial budget. Record what cannot be reproduced. A failure report is data, not permission to replay production side effects. ## Workflow 1. **Triage the observed failure.** Identify the failed acceptance check and affected outcome. Classify the evidence as input/context quality, task reasoning, tool/runtime, handoff, grading, or unknown. Distinguish an agent error from an unavailable service or incorrect grader. Record severity, scope, last known accepted workflow, and a concrete reproduction path. 2. **Protect the operating boundary.** Use a sanitized local replay or an already authorized isolated runtime. Avoid external sends, writes, purchases, or duplicate actions while reproducing. Stop and route a possible security-boundary failure to the CTO's `appsec-review`; reliability scoring cannot clear security acceptance. The CTO's `skill-vetting` applies before running a new or changed skill. 3. **Prepare paired evaluation with the CTO.** Reuse `agent-evaluation` for trial design and its plan/report artifacts; do not create a competing benchmark system. Add operating hypotheses and measures to that plan. Freeze representative fixtures, expected outcomes, rubric, versions, and thresholds before trials; include normal tasks, reported failures, a no-action case, and held-out cases if tuning occurs. Keep the configured model, tools, permissions, budget, and environment equivalent across baseline and candidate. 4. **Collect traceable observations.** Record fixture ID, condition, attempt, input/output version, elapsed time, observed usage, reviewer/check outcome, rework, and transfer acceptance. Retain failures, incomplete pairs, and missing measurements. Sanitize logs and exclude secrets. Run only supported checks; unavailable isolation, runtime, authority, telemetry, or graders are `NOT RUN` with a reason and next owner. 5. **Compute operating measures.** Use the [shared metric definitions](https://github.com/alebgl77/claude-inc/blob/main/docs/caio-operating-model.md#operating-measures). Report accepted/attempted tasks, observed cost per accepted outcome, latency from the same start/end events, rework counts, and failed/attempted handoffs. Include failed-attempt costs. Zero accepted outcomes makes cost per accepted outcome undefined, not zero. Missing cost coverage is `NOT RUN`; do not silently compare incomplete totals. 6. **Review discrepancies and bounded fixes.** Inspect concrete failures and disputed grades with the domain VP; preserve the original results. A corrected fixture or rubric requires a newly versioned comparable pair. Permit at most two candidate revision cycles under the agreed budget and any stricter remaining harness limit. Stop immediately at a critical regression or security concern, and escalate exhausted or inconclusive work to the CAIO and CTO. 7. **Return an operating decision.** Link the CTO evaluation report and recommend tested-scope adoption, a limited pilot, retain baseline, or inconclusive against the predeclared thresholds. Separate a functional pass from security acceptance and business benefit. Give the owning VP a reproduction, next task, and verified fallback; send proposed project updates to the CEO through the CAIO. ## Output format Write `agent-reliability-<task>.md` alongside the reused `agent-eval-plan-<task>.md` and `agent-evaluation-<task>.md` when those trials are in scope. ```markdown # Agent reliability — <task ID and outcome> Failure / impact: <failed check, affected cases, evidence> Classification / hypothesis: <observed category; unproven explanation> Reproduction / boundary: <sanitized fixture, steps, allowed tools, side-effect limits> Baseline / candidate / trial plan: <versions, frozen inputs, CTO report paths> Thresholds / stop conditions: <predeclared requirements, caps, critical regressions> | Fixture / attempt | Condition | Check result | Accepted outcome | Time | Usage/cost | Rework | Handoff | Evidence | |---|---|---|---|---|---|---|---|---| | <ID> | <baseline/candidate> | <PASS/FAIL/NOT RUN> | <yes/no/not assessed> | <observed> | <observed or NOT RUN> | <count> | <accepted/rejected/pending> | <path> | Aggregate comparison: <counts/denominators, paired differences, missing pairs> Uncertainty: <sample, repeated-case dependence, missing data, grader limits> Security acceptance: <CTO evidence and status, or NOT RUN> Decision / fallback: <recommendation, tested scope, owner, baseline version> Loop / CAIO handoff: <cycles used of 2; blocker or VP next task> ``` ## Quality bar - [ ] The issue is tied to an observed failed check and a bounded reproduction. - [ ] The CTO's evaluation procedure is reused with frozen comparable fixtures. - [ ] All five operating measures have explicit denominators/boundaries and evidence or `NOT RUN`. - [ ] Failures, missing pairs, unknown grades, and zero-success outcomes stay visible. - [ ] No production side effect is replayed or retried without authority and state verification. - [ ] Findings distinguish reliability, security, utility, and uncertainty; no promised uplift. ## Example **Ask:** "Sales rejects our AI-qualified leads because the evidence is missing." **Produced:** `agent-reliability-lead-packets.md` links rejected packets to the missing-source check and proposes comparison of the current handoff with a dated evidence field. If no isolated runtime or cost records are available, live trials and cost are `NOT RUN`; the report cannot conclude the candidate is cheaper or more reliable. ## Sources Original first-party operating manual. See the [CAIO guide's source notes](https://github.com/alebgl77/claude-inc/blob/main/docs/caio-operating-model.md#sources); external evaluations do not establish this workflow's performance.
Versions
The database has no other skills named agent-reliability.
Database snapshot: 13.11.2026 · package version 2cdf3eec2b72 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90harness-evaluation
Build and run baseline/candidate evaluations for agent instructions and skills, diagnose evaluation failures, and measure supported improvements. Use for harness maintenance, not ordinary application testing. - 100reliability
Run one analytics task through several fresh trials to measure what repeats and what varies. Use for reliability, repeatability, variance, or repeated-run requests. This measures stability, not correctness. - 90agent-evaluation
Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent. - 100agent-system-design
English-language methodology for single-agent and multi-agent system design. From requirements, determine whether an Agent is needed; choose Prompt, Workflow, Single Agent, or Multi-Agent; design roles, task contracts, tools, context and memory, orchestration, handoffs, human…