evaluate-agentic-rag
/evaluate-agentic-rag
Evaluate Agentic RAG — measure retrieval quality, source selection accuracy, iteration efficiency, groundedness, citation accuracy, cost per query. Use when: evaluate, benchmark.
Assessment and security
- Files in this package
- 3
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Evaluate Agentic RAG — measure retrieval quality, source selection accuracy, iteration efficiency, groundedness, citation accuracy, cost per query. Use when: evaluate, benchmark.
Original author description
Evaluate Agentic RAG — measure retrieval quality, source selection accuracy, iteration efficiency, groundedness, citation accuracy, cost per query. Use when: evaluate, benchmark.
How it works
- When to Use
- Agentic RAG Metrics & Targets
- Step 1: Prepare Multi-Source Test Set
- Step 2: Evaluate Source Selection
- Step 3: Evaluate Iteration Efficiency
- Step 4: Evaluate Self-Evaluation Quality
- Step 5: Compare vs Standard RAG (Play 01)
- Step 6: Generate Report
- Evaluation Cadence
- Common Failure Patterns
SKILL.md preview
---
name: evaluate-agentic-rag
description: "Evaluate Agentic RAG — measure retrieval quality, source selection accuracy, iteration efficiency, groundedness, citation accuracy, cost per query. Use when: evaluate, benchmark."
---
# Evaluate Agentic RAG
## When to Use
- Evaluate autonomous retrieval quality across query types
- Measure source selection accuracy (correct source chosen?)
- Assess iteration efficiency (fewer hops = faster, cheaper)
- Validate groundedness and citation accuracy
- Gate deployments with agentic retrieval thresholds
## Agentic RAG Metrics & Targets
| Metric | Target | How to Measure |
|--------|--------|----------------|
| Groundedness | ≥ 0.90 | Azure AI Evaluation SDK |
| Relevance | ≥ 0.85 | Retrieved docs relevance to query |
| Source selection accuracy | ≥ 90% | Correct source chosen for query type |
| Avg retrieval hops | < 2.0 | Average iterations per query |
| Iteration reduction | ≥ 30% vs max | Queries resolved in 1 hop vs needing max |
| Citation accuracy | ≥ 95% | Citations link to correct source doc |
| Cache hit rate | ≥ 40% | Cached responses / total queries |
| Cost per query | < $0.05 | Tokens across all iterations |
| Self-eval accuracy | ≥ 85% | Agent correctly knows when to stop |
## Step 1: Prepare Multi-Source Test Set
```json
{"query": "What's our refund policy?", "expected_source": "knowledge-base", "expected_hops": 1}
{"query": "Latest Azure pricing changes", "expected_source": "web-search", "expected_hops": 1}
{"query": "Order status for customer 12345", "expected_source": "product-db", "expected_hops": 1}
{"query": "Compare our warranty vs competitor X", "expected_source": "knowledge-base+web-search", "expected_hops": 2}
```
Minimum: 50 test queries across single-source, multi-source, and iterative scenarios.
## Step 2: Evaluate Source Selection
```bash
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics source_selection
```
- Did the agent choose the right source for each query?
- Track: correct source, wrong source, unnecessary multi-source
## Step 3: Evaluate Iteration Efficiency
- Queries that resolve in 1 hop (ideal) vs 2-3 hops (acceptable) vs max hops (concern)
- Compare: iterative agent cost vs fixed RAG cost for same queries
- Target: 70% of queries resolve in ≤ 2 hops
## Step 4: Evaluate Self-Evaluation Quality
- Does the agent correctly assess when it has enough context?
- False positive: agent thinks it has enough but answer is wrong
- False negative: agent keeps searching when answer is already in context
- Track: self-eval triggered vs actual groundedness score
## Step 5: Compare vs Standard RAG (Play 01)
Run same test set through both Play 01 (fixed RAG) and Play 21 (agentic RAG):
| Metric | Standard RAG | Agentic RAG | Delta |
|--------|-------------|-------------|-------|
| Groundedness | Measure | Measure | Improvement |
| Multi-source coverage | N/A (single) | Measure | New capability |
| Cost per query | Measure | Measure | Cost overhead |
| Complex query accuracy | Measure | Measure | Key differentiator |
## Step 6: Generate Report
```bash
python evaluation/eval.py --all --output evaluation/report.json --ci-gate
```
### Quality Gate Decision
| Result | Action |
|--------|--------|
| All PASS | Deploy to production |
| Groundedness < 0.85 | Improve retrieval prompts, add more sources |
| Source selection < 80% | Refine tool descriptions, add routing examples |
| Avg hops > 3 | Raise self-eval threshold, improve first-hop quality |
| Cache hit < 20% | Lower similarity threshold, increase TTL |
## Evaluation Cadence
- **Pre-deployment**: Full multi-source evaluation suite
- **Weekly**: Source selection accuracy, cache hit rate
- **Monthly**: Full comparison vs standard RAG baseline
- **On source change**: Re-evaluate when adding/removing data sources
## Common Failure Patterns
| Symptom | Root Cause | Fix |
|---------|-----------|-----|
| Agent ignores available sources | Poor tool descriptions | Rewrite tool descriptions with clear use cases |
| Always uses max iterations | Self-eval threshold too high (0.99) | Lower to 0.85 |
| Bad citations | Source metadata not returned from tools | Include doc title + URL in tool response |
| Cache returns wrong answer | Similarity threshold too low | Raise from 0.85 to 0.92 |
| High cost on simple queries | No routing shortcut for FAQ | Add direct-answer path for high-confidence KB matches |
| Multi-source synthesis poor | No merge strategy | Add explicit "synthesize from multiple sources" in system prompt |
## CI/CD Integration
```yaml
- name: Agentic RAG Quality Gate
run: python evaluation/eval.py --all --ci-gate
- name: Source Selection Gate
run: python evaluation/eval.py --metrics source_selection --ci-gate --threshold 0.90
- name: Cost Gate
run: python evaluation/eval.py --metrics cost_per_query --ci-gate --max-cost 0.05
```
Files in this package (3)
- SKILL.lean.md 5 KB
- SKILL.md 5 KB
- agents/openai.yaml 1 KB
Versions
The database has no other skills named evaluate-agentic-rag.
Database snapshot: 13.11.2026 · package version be8fe0a77edf (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100evaluate-enterprise-rag
Evaluate Enterprise RAG quality — measure groundedness, relevance, coherence, fluency, content safety. Run evaluation pipeline, analyze gaps, set quality gates. Use when: evaluate, quality, groundedness, relevance, metrics, test, audit. - 90tune-agentic-rag
Tune Agentic RAG — optimize source routing weights, iteration thresholds, self-evaluation sensitivity, semantic cache, cost per query. Use when: tune, optimize retrieval. - 100rag-evaluator
Generates tailored giskard.checks evaluation suites for RAG (Retrieval-Augmented Generation) systems. Use whenever a user describes a Q&A bot grounded in documents, a knowledge-base chatbot, a retrieval system, or wants to evaluate answer groundedness, faithfulness,… - 98agentic-rag
Build agentic RAG systems with autonomous planning, reflection, and tool use. Use this skill when implementing agent-driven retrieval, query decomposition, iterative refinement, or multi-source RAG orchestration. Activate when: agentic RAG, A-RAG, autonomous retrieval, query… - 90monitor-rag-quality
Use this to measure and monitor the quality of a RAG (retrieval-augmented generation) pipeline - whether it retrieves the right context and answers faithfully. Trigger on "my RAG gives wrong answers", "is my retrieval any good", "the chatbot makes things up", "evaluate my RAG",… - 100evaluate-langchain4j
Sets up evaluation of LangChain4j applications and RAG pipelines using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark a LangChain4j app, RAG pipeline, or AI service. Also use when the user mentions LangChain4j evaluation, RAG testing, retrieval…