Przejdź do treści
Biblioteka skilli
AI i agenci · Testowanie

agent-accuracy

/agent-accuracy

Naprawia agenta LLM albo system RAG, który podaje błędne, sprzeczne lub zmyślone wyniki: odtwarza problem, izoluje komponent i mierzy poprawkę.

Repozytorium
agent-engineer ↗
Licencja
MIT
Ostatnia zmiana repozytorium
13.11.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
5
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 13.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Naprawia agenta LLM albo system RAG, który podaje błędne, sprzeczne lub zmyślone wyniki: odtwarza problem, izoluje komponent i mierzy poprawkę.

Wyzwalacze z opisu

it hallucinateswrong answersaccuracy droppedflaky / right only sometimesworse after the model upgradeshould I add RAG / a bigger model / fine-tune?
Oryginalny opis (EN)

Debug and fix an LLM agent or RAG system that gives wrong, inconsistent, hallucinated or low-quality results: reproduce, localise the failing component, one hypothesis, the least-powerful fix, measure, check regressions, log. Use when the user says "it hallucinates", "wrong answers", "accuracy dropped", "flaky / right only sometimes", "worse after the model upgrade", "should I add RAG / a bigger model / fine-tune?", or wants a pass rate up. Not for building evals (agent-evals), errors/timeouts/migrations (agent-production), retrieval internals (agent-rag), prompt rewrites (agent-prompting) or

Jak działa

  1. Core rules
  2. Workflow
  3. Decision rules
  4. Anti-patterns
  5. Outputs
  6. References
Sekcje z nagłówków SKILL.md, w oryginale · 105 linii, 852 słów
Podgląd SKILL.md
---
name: agent-accuracy
description: >-
  Debug and fix an LLM agent or RAG system that gives wrong, inconsistent, hallucinated or
  low-quality results: reproduce, localise the failing component, one hypothesis, the
  least-powerful fix, measure, check regressions, log. Use when the user says "it
  hallucinates", "wrong answers", "accuracy dropped", "flaky / right only sometimes",
  "worse after the model upgrade", "should I add RAG / a bigger model / fine-tune?", or
  wants a pass rate up. Not for building evals (agent-evals), errors/timeouts/migrations
  (agent-production), retrieval internals (agent-rag), prompt rewrites (agent-prompting)
  or tool redesign (agent-tools).
license: MIT
metadata:
  version: 0.4.0
---

# Agent accuracy

Take one failure mode at a time to a measured, logged fix that breaks nothing else, using
the cheapest change that works.

## Core rules

1. **Measure before and after.** A fix needs a baseline. No eval set → propose 20–50
   cases from real failures via `agent-evals` (*Minimal eval*); if the user declines, say
   the fix is unverified.
2. **Reproduce first:** k ≈ 3 / expected failure rate runs (≈ 5 at 50%, ≈ 30 at 10%;
   20–30 if unknown). Five clean runs only rule out failure rates above ~45%.
3. **Fix the first upstream failure.** Later errors cascade from it.
4. **One hypothesis, one change per experiment.**
5. **Least-powerful fix first:** code/config/data → prompt → examples → tools → retrieval
   → architecture → model → fine-tune. Climb only when the lower rung failed and
   localisation still points higher.
6. **Fix categories, not traces.** A single bad trace goes into the dataset, not the
   prompt.
7. **Paired comparison on dev with repeats; confirm once on held-out test;** full suite
   for regressions, cost, latency.
8. **Log every experiment,** failures included, in `experiment-log.md`.
9. **0/k = capability (something missing); some/k = reliability.** Different fixes.

## Workflow

1. **Intake.** Find the failures yourself before touching code: logged conversations,
   traces or eval output in the repo (`logs/`, `data/`, `*.jsonl`) or the tracing backend.
   Read the failing ones; they become the reproduction cases. Also read
   `failure-taxonomy.md` and `experiment-log.md` if present, and state what "fixed" means.
2. **Reproduce.** Each failure becomes an eval case (multi-turn → single turn or N-1).
   Run k times; record per-case pass rate.
3. **Rule out the eval.** Is the reference, grader or task wrong?
4. **Localise** (`references/localising-failures.md`): first wrong step, confirmed by a
   counterfactual probe. With subagents: many traces → `trace-analyst`; retrieval →
   `rag-diagnostician`.
5. **Quantify.** How many cases share the mode? Work down by rate × severity.
6. **Hypothesise** in the log before running: "M fails because C; changing X moves M from
   a% to b% without hurting N."
7. **Fix** at the lowest rung that addresses the cause (`references/intervention-ladder.md`).
8. **Measure** (`references/experiment-discipline.md`; procedure: `agent-evals` →
   *Comparing two versions*).
9. **Regression check:** full suite, cost, latency. Reproduced cases join the regression
   set.
10. **Log and decide:** keep, revert or iterate. Next mode, or stop.
11. **Next:** `agent-evals` for new gates; `agent-production` to roll out;
    `agent-observability` to watch the mode online.

## Decision rules

| Symptom | First move |
|---|---|
| Confident wrong fact | Search the trace's context for the right fact. Absent → retrieval/data; present → generation |
| Right context, wrong answer | Answer only from context, quote first, allow "I don't know"; trim distractors |
| Wrong tool or arguments | Tool names, descriptions, schemas, examples (`agent-tools`) |
| Format breaks | Structured output + validation in code |
| Answers half of a multi-part question | Decompose; completeness check per part |
| Confident answer after empty/`null` tool result | Silent tool failure: explicit tool errors (`agent-tools`) |
| Varies run to run | Remove ambiguity, examples, rules into code, verification step; measure pass^k |
| Worse after model/prompt change | Paired diff by mode vs last good version; read the log |
| Degrades in long runs | `agent-context` |
| Correct but slow/expensive | Not accuracy → `agent-production` |

- **Add RAG** only if the needed fact is absent from context on failing traces; for
  format, policy or consistency it can hurt.
- **Bigger model** only when cases fail 0/k with oracle context, clean instructions and
  good tools, and a stronger-model probe passes. Sweep effort first.
- **Fine-tune** only for consistent behaviour/format, with 50+ examples and a stable eval.

**Stop** when the target is met with the CI's lower bound above it; when the last 2–3
experiments are within noise and the next rung costs more than the remaining failures; or
when what's left needs a product decision. Tell the user which.

## Anti-patterns

- Tweak, eyeball two examples, ship.
- Several changes in one experiment; one prompt line per bad trace.
- Iterating on the test split; eval cases in few-shot examples.
- RAG, a bigger model, multi-agent or fine-tuning before localising.
- Fixing the downstream symptom instead of the first upstream failure.
- Trusting one run of a stochastic agent.
- Running a prompt optimiser on an unvalidated eval.
- Blaming the model when the grader or reference is wrong.

## Outputs

Write these to `agent-engineering/` only if the project keeps one (see `agent-engineer` → *Project record (optional)*); otherwise put them in your reply or the PR description.

- One `experiment-log.md` entry per experiment: change, hypothesis, eval and dataset
  version, before → after with CI, regressions, cost/latency, decision.
- `failure-taxonomy.md`: updated counts and `Component` column.
- New regression cases in `datasets/` with provenance.

## References

- `references/localising-failures.md` — trace reading order, symptom → component →
  confirmation, counterfactual probes, capability vs reliability. Read before choosing a
  fix.
- `references/intervention-ladder.md` — each rung's when/how/pitfalls, when popular fixes
  are wrong, reliability fixes. Read when choosing a fix.
- `references/experiment-discipline.md` — hypothesis, run counts, keep/revert,
  overfitting, log entry, stopping. Comparison mechanics: `agent-evals` →
  `references/quick-paths.md`.
Pliki w paczce (5)
  • SKILL.md 6 KB
  • assets/experiment-log.md 1 KB
  • references/experiment-discipline.md 3 KB
  • references/intervention-ladder.md 7 KB
  • references/localising-failures.md 6 KB

Wersje

W bazie nie ma innych skilli o nazwie agent-accuracy.

Migawka bazy z 13.11.2026 · wersja paczki efa1c5513962 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100agent-evals
    Mierzy agentów LLM i systemy RAG: analizę błędów, taksonomię, zbiory testowe, sędziów LLM, ścieżki wywołań i bramki CI.
  • 90monitor-rag-quality
    Mierzy jakość potoku RAG i rozdziela błędy wyszukiwania od błędów generowania odpowiedzi.
  • 100add-llm-evals
    Dodaje ewaluację do aplikacji LLM lub agenta: mierzy poprawność, wierność, trafność i bezpieczeństwo wyników, offline w CI oraz online przez LLM-as-a-judge w produkcji.
  • 90production-ai-agents
    Podpowiada, jak budować agenta LLM, RAG i funkcje z wywoływaniem narzędzi: model, prompty, kontekst, koszty tokenów i ewaluacje.
  • 100rag-systems
    Projektuje, buduje i naprawia systemy RAG: parsowanie, fragmentowanie, wyszukiwanie hybrydowe, kontrolę dostępu i ewaluację odpowiedzi.
  • 100agent-rag
    Ustala, czy agent potrzebuje wyszukiwania, a potem projektuje, mierzy i naprawia potok RAG: podział tekstu, osadzenia, hybrydowe wyszukiwanie i cytowania.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.