Skip to content
Skills Library
AI & agents · Testing

evaluate-agentic-rag

/evaluate-agentic-rag

Evaluate Agentic RAG — measure retrieval quality, source selection accuracy, iteration efficiency, groundedness, citation accuracy, cost per query. Use when: evaluate, benchmark.

Repository
frootai ↗
Licence
MIT
Last repository change
26.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Evaluate Agentic RAG — measure retrieval quality, source selection accuracy, iteration efficiency, groundedness, citation accuracy, cost per query. Use when: evaluate, benchmark.

Original author description

Evaluate Agentic RAG — measure retrieval quality, source selection accuracy, iteration efficiency, groundedness, citation accuracy, cost per query. Use when: evaluate, benchmark.

How it works

  1. When to Use
  2. Agentic RAG Metrics & Targets
  3. Step 1: Prepare Multi-Source Test Set
  4. Step 2: Evaluate Source Selection
  5. Step 3: Evaluate Iteration Efficiency
  6. Step 4: Evaluate Self-Evaluation Quality
  7. Step 5: Compare vs Standard RAG (Play 01)
  8. Step 6: Generate Report
  9. Evaluation Cadence
  10. Common Failure Patterns
Original SKILL.md headings · 101 lines, 731 words
SKILL.md preview
---
name: evaluate-agentic-rag
description: "Evaluate Agentic RAG — measure retrieval quality, source selection accuracy, iteration efficiency, groundedness, citation accuracy, cost per query. Use when: evaluate, benchmark."
---

# Evaluate Agentic RAG

## When to Use
- Evaluate autonomous retrieval quality across query types
- Measure source selection accuracy (correct source chosen?)
- Assess iteration efficiency (fewer hops = faster, cheaper)
- Validate groundedness and citation accuracy
- Gate deployments with agentic retrieval thresholds

## Agentic RAG Metrics & Targets

| Metric | Target | How to Measure |
|--------|--------|----------------|
| Groundedness | ≥ 0.90 | Azure AI Evaluation SDK |
| Relevance | ≥ 0.85 | Retrieved docs relevance to query |
| Source selection accuracy | ≥ 90% | Correct source chosen for query type |
| Avg retrieval hops | < 2.0 | Average iterations per query |
| Iteration reduction | ≥ 30% vs max | Queries resolved in 1 hop vs needing max |
| Citation accuracy | ≥ 95% | Citations link to correct source doc |
| Cache hit rate | ≥ 40% | Cached responses / total queries |
| Cost per query | < $0.05 | Tokens across all iterations |
| Self-eval accuracy | ≥ 85% | Agent correctly knows when to stop |

## Step 1: Prepare Multi-Source Test Set
```json
{"query": "What's our refund policy?", "expected_source": "knowledge-base", "expected_hops": 1}
{"query": "Latest Azure pricing changes", "expected_source": "web-search", "expected_hops": 1}
{"query": "Order status for customer 12345", "expected_source": "product-db", "expected_hops": 1}
{"query": "Compare our warranty vs competitor X", "expected_source": "knowledge-base+web-search", "expected_hops": 2}
```
Minimum: 50 test queries across single-source, multi-source, and iterative scenarios.

## Step 2: Evaluate Source Selection
```bash
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics source_selection
```
- Did the agent choose the right source for each query?
- Track: correct source, wrong source, unnecessary multi-source

## Step 3: Evaluate Iteration Efficiency
- Queries that resolve in 1 hop (ideal) vs 2-3 hops (acceptable) vs max hops (concern)
- Compare: iterative agent cost vs fixed RAG cost for same queries
- Target: 70% of queries resolve in ≤ 2 hops

## Step 4: Evaluate Self-Evaluation Quality
- Does the agent correctly assess when it has enough context?
- False positive: agent thinks it has enough but answer is wrong
- False negative: agent keeps searching when answer is already in context
- Track: self-eval triggered vs actual groundedness score

## Step 5: Compare vs Standard RAG (Play 01)
Run same test set through both Play 01 (fixed RAG) and Play 21 (agentic RAG):
| Metric | Standard RAG | Agentic RAG | Delta |
|--------|-------------|-------------|-------|
| Groundedness | Measure | Measure | Improvement |
| Multi-source coverage | N/A (single) | Measure | New capability |
| Cost per query | Measure | Measure | Cost overhead |
| Complex query accuracy | Measure | Measure | Key differentiator |

## Step 6: Generate Report
```bash
python evaluation/eval.py --all --output evaluation/report.json --ci-gate
```

### Quality Gate Decision
| Result | Action |
|--------|--------|
| All PASS | Deploy to production |
| Groundedness < 0.85 | Improve retrieval prompts, add more sources |
| Source selection < 80% | Refine tool descriptions, add routing examples |
| Avg hops > 3 | Raise self-eval threshold, improve first-hop quality |
| Cache hit < 20% | Lower similarity threshold, increase TTL |

## Evaluation Cadence
- **Pre-deployment**: Full multi-source evaluation suite
- **Weekly**: Source selection accuracy, cache hit rate
- **Monthly**: Full comparison vs standard RAG baseline
- **On source change**: Re-evaluate when adding/removing data sources

## Common Failure Patterns

| Symptom | Root Cause | Fix |
|---------|-----------|-----|
| Agent ignores available sources | Poor tool descriptions | Rewrite tool descriptions with clear use cases |
| Always uses max iterations | Self-eval threshold too high (0.99) | Lower to 0.85 |
| Bad citations | Source metadata not returned from tools | Include doc title + URL in tool response |
| Cache returns wrong answer | Similarity threshold too low | Raise from 0.85 to 0.92 |
| High cost on simple queries | No routing shortcut for FAQ | Add direct-answer path for high-confidence KB matches |
| Multi-source synthesis poor | No merge strategy | Add explicit "synthesize from multiple sources" in system prompt |

## CI/CD Integration
```yaml
- name: Agentic RAG Quality Gate
  run: python evaluation/eval.py --all --ci-gate
- name: Source Selection Gate
  run: python evaluation/eval.py --metrics source_selection --ci-gate --threshold 0.90
- name: Cost Gate
  run: python evaluation/eval.py --metrics cost_per_query --ci-gate --max-cost 0.05
```
Files in this package (3)
  • SKILL.lean.md 5 KB
  • SKILL.md 5 KB
  • agents/openai.yaml 1 KB

Versions

The database has no other skills named evaluate-agentic-rag.

Database snapshot: 13.11.2026 · package version be8fe0a77edf (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100evaluate-enterprise-rag
    Evaluate Enterprise RAG quality — measure groundedness, relevance, coherence, fluency, content safety. Run evaluation pipeline, analyze gaps, set quality gates. Use when: evaluate, quality, groundedness, relevance, metrics, test, audit.
  • 90tune-agentic-rag
    Tune Agentic RAG — optimize source routing weights, iteration thresholds, self-evaluation sensitivity, semantic cache, cost per query. Use when: tune, optimize retrieval.
  • 100rag-evaluator
    Generates tailored giskard.checks evaluation suites for RAG (Retrieval-Augmented Generation) systems. Use whenever a user describes a Q&A bot grounded in documents, a knowledge-base chatbot, a retrieval system, or wants to evaluate answer groundedness, faithfulness,…
  • 98agentic-rag
    Build agentic RAG systems with autonomous planning, reflection, and tool use. Use this skill when implementing agent-driven retrieval, query decomposition, iterative refinement, or multi-source RAG orchestration. Activate when: agentic RAG, A-RAG, autonomous retrieval, query…
  • 90monitor-rag-quality
    Use this to measure and monitor the quality of a RAG (retrieval-augmented generation) pipeline - whether it retrieves the right context and answers faithfully. Trigger on "my RAG gives wrong answers", "is my retrieval any good", "the chatbot makes things up", "evaluate my RAG",…
  • 100evaluate-langchain4j
    Sets up evaluation of LangChain4j applications and RAG pipelines using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark a LangChain4j app, RAG pipeline, or AI service. Also use when the user mentions LangChain4j evaluation, RAG testing, retrieval…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.