rag-debugger
/rag-debugger
Use when a RAG system is returning bad answers — hallucinations, wrong content, missing exact terms, low recall, slow responses, stale data, citation errors, tenant data leakage, or any production retrieval failure. Walks the 9-layer diagnostic chain backward from output to root cause. Triggers on phrases like "RAG returning wrong answers", "hallucinating", "can't find document", "wrong tenant data", "low recall", "RAG is slow", "missing citations".
- Author
- kumaran-is ↗
- Repository
- claude-code-onboarding ↗
- Licence
- MIT
- Last repository change
- 30.06.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 30.06.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Use when a RAG system is returning bad answers — hallucinations, wrong content, missing exact terms, low recall, slow responses, stale data, citation errors, tenant data leakage, or any production retrieval failure. Walks the 9-layer diagnostic chain backward from output to root cause. Triggers on phrases like "RAG returning wrong answers", "hallucinating", "can't find document", "wrong tenant data", "low recall", "RAG is slow", "missing citations".
Triggers from the description
Original author description
Use when a RAG system is returning bad answers — hallucinations, wrong content, missing exact terms, low recall, slow responses, stale data, citation errors, tenant data leakage, or any production retrieval failure. Walks the 9-layer diagnostic chain backward from output to root cause. Triggers on phrases like "RAG returning wrong answers", "hallucinating", "can't find document", "wrong tenant data", "low recall", "RAG is slow", "missing citations".
How it works
- The diagnostic chain
- Failure modes by layer
- Common failure patterns (signature → layer)
- How to apply
- Anti-patterns to watch for
SKILL.md preview
---
name: rag-debugger
description: Use when a RAG system is returning bad answers — hallucinations, wrong content, missing exact terms, low recall, slow responses, stale data, citation errors, tenant data leakage, or any production retrieval failure. Walks the 9-layer diagnostic chain backward from output to root cause. Triggers on phrases like "RAG returning wrong answers", "hallucinating", "can't find document", "wrong tenant data", "low recall", "RAG is slow", "missing citations".
---
# RAG Failure Diagnosis
When a RAG answer is wrong, **walk the layers backward** from output to root cause. Most "the LLM hallucinated" reports are actually upstream failures that surface at generation.
**Critical principle:** don't fix downstream when upstream is broken. Tuning the reranker won't help if the parser dropped the table.
## The diagnostic chain
For any wrong answer, walk in this order. **The first layer where you find a problem is your root cause** — stop walking there.
```
Wrong answer
│
├─ Layer 9: LLM generation
│ Did the LLM ignore evidence it had?
│ check: was the correct chunk in the context window?
│
├─ Layer 8: Context packing
│ Did the right chunk reach the LLM?
│ check: was it retrieved but dropped before packing?
│
├─ Layer 7: Reranker
│ Did rerank bury the right chunk?
│ check: was the right chunk in pre-rerank candidates but not top-k?
│
├─ Layer 6: Fusion (RRF)
│ Did fusion lose a strong single-retriever hit?
│ check: was it top-3 in BM25 or dense alone but buried after RRF?
│
├─ Layer 5: Retrieval (dense)
│ Did dense find the right chunk?
│ check: similarity score; close to top hits?
│
├─ Layer 4: Retrieval (sparse / BM25)
│ Did BM25 find the right chunk?
│ check: does the chunk contain query terms? Stop-words eating query?
│
├─ Layer 3: Filtering
│ Did filters wrongly exclude the right chunk?
│ check: replay query without filters; does it appear?
│
├─ Layer 2: Chunking
│ Was the answer split across chunk boundaries?
│ check: does the answer text exist in any single chunk?
│
└─ Layer 1: Parsing / Ingestion
Was the source content captured at all?
check: does the raw document contain the answer?
```
## Failure modes by layer
| Layer | Symptom | Likely cause | Fix direction |
|---|---|---|---|
| 1 Parsing | Answer not in any chunk; raw doc has it | Parser dropped table, lost footer, OCR failed | Better parser (Unstructured.io / LlamaParse), OCR quality gates, chunk quality scoring |
| 1 Ingestion | Document not indexed | Failed job, deletion bug, tombstone | Audit ingestion logs, idempotency check |
| 2 Chunking | Condition separated from rule | Fixed-size split mid-statement | Document-aware chunking, parent-child, overlap |
| 2 Chunking | Table broken across chunks | Naive splitter cut the table | Adaptive chunking, preserve tables whole |
| 3 Filtering | Right chunk filtered out | Metadata mismatch, tenant filter too aggressive, version filter | Audit filter logic; check effective_to / version |
| 3 Filtering | Post-filter killed recall | Top-k all filtered after retrieval | Switch to pre-filtered ANN |
| 4 Sparse | Exact term not matched | Tokenizer issue, stop-words too aggressive | Check tokenization; rebuild with correct analyzer |
| 5 Dense | Paraphrase not matched | Domain-weak embedder, chunk size wrong | Test domain embedder; adjust chunk size |
| 5 Dense | Proper noun missed | Model never saw the proper noun | Hybrid (BM25 catches it) |
| 6 Fusion | Strong hit buried | One retriever had 1 great hit, other had 50 mediocre | Inspect rank-by-rank; consider weighted RRF |
| 7 Reranker | Wrong chunk preferred | Reranker mismatched to domain | Try different reranker; eval on labeled pairs |
| 7 Reranker | Length bias | Long irrelevant beat short relevant | Length normalization; switch reranker |
| 8 Packing | Right chunk dropped | Aggressive top-k, dedup removed unique answer | Loosen top-k; smarter dedup |
| 8 Packing | Lost in the middle | Right chunk at end of long context | Move highest-scored first |
| 9 Generation | LLM ignored evidence | Weak grounding prompt; model priors | Strict grounding; structured output; abstention threshold |
| 9 Generation | Hallucinated citation | Citation present, chunk doesn't support claim | Per-claim citation checking; answer contract validation |
## Common failure patterns (signature → layer)
| Pattern | Signature | Likely layer |
|---|---|---|
| Plausible but wrong | Coherent answer, no citation matches | 9 or 8 |
| "I don't know" when answer exists | Right chunk never retrieved | 4–6 |
| Outdated answer | Newer version exists, old retrieved | 3 (version filter) or 5 (no freshness signal) |
| Wrong tenant data | Info from another customer | 3 — **security incident, page someone** |
| Repeated / redundant | Same content cited multiple times | 8 (no dedup) |
| Right facts, wrong combination | Each fact cited, combination invented | 9 or 2 (lost connector across chunks) |
## How to apply
When the user reports a RAG failure:
1. **Get the trace.** Ask for: query, classification (if any), filters applied, all retriever outputs with scores, RRF result, rerank result, context sent to LLM, LLM response. If they don't have this — that's the first problem to fix (audit logs per §39.2).
2. **Check Layer 1 first.** Does the answer exist in any chunk? If no, stop. Nothing downstream matters. Recommend parser / ingestion fixes.
3. **Walk forward layer by layer.** At each layer, ask: was the right chunk present? The first layer where it disappears is the root cause.
4. **Match against common patterns** — they short-circuit the walk.
5. **For tenant data leakage: page immediately.** Pre-filter chokepoint is broken; this is a security incident, not a quality issue.
## Anti-patterns to watch for
- "It's just hallucinating, raise the temperature lower" → almost never the answer; check upstream first
- "Add more chunks to context" → if retrieval is wrong, more wrong chunks won't help
- "Try a bigger model" → bigger model with same broken retrieval is still broken
- "Tune the reranker" before checking that the right chunk is in the candidate pool
Reference: full playbook §40 (failure taxonomy), §39.2 (audit log schema).
Versions
The database has no other skills named rag-debugger.
Database snapshot: 13.11.2026 · package version 5d8f8608fffa (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100rag-systems
Designs, builds and debugs retrieval-augmented generation (RAG) - ingestion and parsing, chunking, embeddings, hybrid search with reranking, access control at retrieval time, grounded answers with citations, and retrieval and answer evaluation. Use when building search or Q&A… - 100rag-auditor
Evaluates RAG pipeline quality across retrieval (precision, recall, MRR) and generation (groundedness, hallucination rate). Triggers on: "audit RAG pipeline", "RAG quality", "hallucination detection", "why is RAG failing", "grounding check". NOT for general architecture audits,… - 90monitor-rag-quality
Use this to measure and monitor the quality of a RAG (retrieval-augmented generation) pipeline - whether it retrieves the right context and answers faithfully. Trigger on "my RAG gives wrong answers", "is my retrieval any good", "the chatbot makes things up", "evaluate my RAG",… - 100rag
Use when building retrieval-augmented generation. Covers chunking, embedding and hybrid search, reranking, grounding and citation, and diagnosing whether a bad answer is a retrieval failure or a generation failure. Trigger on "our rag keeps hallucinating", "chat with our docs",… - 95rag-eval
Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality,… - 100retrieval-evaluation
Evaluate a RAG or search feature in two stages, retrieval then generation. Use when answers are wrong or cite the wrong document, before changing the LLM, or for a new embedding model or re-index.