Skip to content
Skills Library
AI & agents · Testing

evaluate-knowledge-graph-rag

/evaluate-knowledge-graph-rag

Evaluate Knowledge Graph RAG — measure entity extraction accuracy, relationship precision, graph traversal quality, hybrid retrieval effectiveness, multi-hop reasoning. Use when: evaluate, benchmark graph RAG.

Repository
frootai ↗
Licence
MIT
Last repository change
26.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Evaluate Knowledge Graph RAG — measure entity extraction accuracy, relationship precision, graph traversal quality, hybrid retrieval effectiveness, multi-hop reasoning. Use when: evaluate, benchmark graph RAG.

Original author description

Evaluate Knowledge Graph RAG — measure entity extraction accuracy, relationship precision, graph traversal quality, hybrid retrieval effectiveness, multi-hop reasoning. Use when: evaluate, benchmark graph RAG.

How it works

  1. When to Use
  2. Graph RAG Metrics & Targets
  3. Step 1: Prepare Graph Test Dataset
  4. Step 2: Evaluate Entity Extraction
  5. Step 3: Evaluate Relationship Mapping
  6. Step 4: Evaluate Multi-Hop Queries
  7. Step 5: Compare Graph RAG vs Vector RAG
  8. Step 6: Generate Report
  9. Evaluation Cadence
  10. Common Failure Patterns
Original SKILL.md headings · 102 lines, 698 words
SKILL.md preview
---
name: evaluate-knowledge-graph-rag
description: "Evaluate Knowledge Graph RAG — measure entity extraction accuracy, relationship precision, graph traversal quality, hybrid retrieval effectiveness, multi-hop reasoning. Use when: evaluate, benchmark graph RAG."
---

# Evaluate Knowledge Graph RAG

## When to Use
- Evaluate entity extraction accuracy (correct entities found?)
- Measure relationship mapping precision (correct edges?)
- Assess graph traversal retrieval quality for multi-hop queries
- Compare graph RAG vs vector RAG on relationship questions
- Gate deployments with graph quality thresholds

## Graph RAG Metrics & Targets

| Metric | Target | How to Measure |
|--------|--------|----------------|
| Entity extraction F1 | ≥ 0.85 | Compare extracted vs ground truth entities |
| Relationship precision | ≥ 0.80 | Correct edges / total extracted edges |
| Entity resolution accuracy | ≥ 0.90 | Merged duplicates correctly identified |
| Multi-hop retrieval success | ≥ 75% | 2-3 hop queries returning correct path |
| Groundedness (graph context) | ≥ 0.85 | Answers grounded in graph-retrieved content |
| Graph vs vector lift | ≥ 15% | Quality improvement on relationship queries |
| Graph traversal latency | < 500ms | Gremlin query execution time |
| Entity count accuracy | ±10% of expected | Total entities vs document content |

## Step 1: Prepare Graph Test Dataset
```json
{"doc": "Alice (CTO) reports to Bob (CEO) at Acme Corp. They launched Widget Pro in Q4.",
 "expected_entities": [{"name":"Alice","type":"Person"},{"name":"Bob","type":"Person"},{"name":"Acme Corp","type":"Organization"},{"name":"Widget Pro","type":"Product"}],
 "expected_relationships": [{"from":"Alice","to":"Bob","type":"reports_to"},{"from":"Alice","to":"Acme Corp","type":"works_at"}]}
```
Minimum: 50 documents with labeled entities and relationships.

## Step 2: Evaluate Entity Extraction
```bash
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics entity_extraction
```
- Precision: extracted entities that are correct
- Recall: ground truth entities that were extracted
- Per-type breakdown (Person, Org, Product — which types are weakest?)

## Step 3: Evaluate Relationship Mapping
- Are extracted relationships correct? (Alice reports_to Bob, not Bob reports_to Alice)
- Direction matters for hierarchical relationships
- Track: false relationships (hallucinated connections)
- Track: missed relationships (stated but not extracted)

## Step 4: Evaluate Multi-Hop Queries
Test queries requiring 2-3 relationship hops:
```json
{"query": "Who does Alice's manager work for?", "expected_path": ["Alice → Bob → Acme Corp"], "hops": 2}
{"query": "What products were launched by Alice's company?", "expected_path": ["Alice → Acme Corp → Widget Pro"], "hops": 2}
```

## Step 5: Compare Graph RAG vs Vector RAG
| Query Type | Vector RAG Score | Graph RAG Score | Winner |
|-----------|-----------------|----------------|--------|
| Factual lookup | Measure | Measure | Usually vector |
| Relationship query | Measure | Measure | Usually graph |
| Multi-hop reasoning | Measure | Measure | Always graph |
| Similarity search | Measure | Measure | Always vector |
| Hybrid query | Measure | Measure | Hybrid best |

## Step 6: Generate Report
```bash
python evaluation/eval.py --all --output evaluation/graph-report.json --ci-gate
```

### Quality Gate Decision
| Result | Action |
|--------|--------|
| All PASS | Deploy graph RAG to production |
| Entity F1 < 0.75 | Improve extraction prompt, add entity type examples |
| Relationships < 0.70 | Add relationship type constraints to prompt |
| Multi-hop < 60% | Increase max traversal depth, improve graph density |
| Graph vs vector no lift | Graph not adding value — use vector RAG instead |

## Evaluation Cadence
- **Pre-deployment**: Full entity + relationship evaluation
- **Weekly**: Spot-check entity extraction on new documents
- **Monthly**: Full multi-hop query evaluation
- **On extraction prompt change**: Re-evaluate entity/relationship accuracy

## Common Failure Patterns

| Symptom | Root Cause | Fix |
|---------|-----------|-----|
| Too many entities extracted | No entity type constraints | Limit to "Person, Org, Product, Event" |
| Wrong relationship direction | Prompt doesn't specify direction | Add "from → to" examples |
| Duplicate entities in graph | No entity resolution | Add fuzzy + alias matching |
| Multi-hop returns noise | Too many hops (4+) | Limit to 2-3 hops max |
| Graph too sparse | Only extracting from some docs | Process all documents |
| Vector outperforms graph | Domain has few relationships | Consider vector-only (Play 01) |

## CI/CD Quality Gates
```yaml
- name: Entity Extraction Gate
  run: python evaluation/eval.py --metrics entity_extraction --ci-gate --f1-threshold 0.85
- name: Relationship Gate
  run: python evaluation/eval.py --metrics relationships --ci-gate --precision-threshold 0.80
- name: Multi-Hop Gate
  run: python evaluation/eval.py --metrics multi_hop --ci-gate --success-threshold 0.75
```
Files in this package (3)
  • SKILL.lean.md 5 KB
  • SKILL.md 5 KB
  • agents/openai.yaml 1 KB

Versions

The database has no other skills named evaluate-knowledge-graph-rag.

Database snapshot: 13.11.2026 · package version f2940c09b5db (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100deploy-knowledge-graph-rag
    Deploy Knowledge Graph RAG — configure entity extraction, relationship mapping, Cosmos DB Gremlin graph, graph traversal retrieval, hybrid graph+vector search. Use when: deploy, provision graph RAG.
  • 100knowledge-graph-builder
    Activates KnowledgeGraph — an expert in building, querying, and reasoning over knowledge graphs. Use when you need entity extraction, relationship mapping, ontology design, Neo4j/RDF graph construction, graph-RAG pipelines, or complex multi-hop reasoning over structured knowledge.
  • 100tune-knowledge-graph-rag
    Tune Knowledge Graph RAG — optimize graph traversal depth, entity resolution, hybrid graph+vector weights, extraction prompt, Gremlin query performance, cost. Use when: tune, optimize graph RAG.
  • 100graphrag-patterns
    Implement GraphRAG combining knowledge graphs with RAG for multi-hop reasoning. Use this skill when building knowledge graph RAG, implementing multi-hop queries, using Neo4j with RAG, or connecting entities across documents. Activate when: GraphRAG, knowledge graph, multi-hop…
  • 100graph-rag
    Knowledge-graph-augmented retrieval. Entity and triple extraction, graph construction (Neo4j, LlamaIndex PropertyGraphIndex), hierarchical community summarization (Microsoft GraphRAG), personalized PageRank (HippoRAG), multi-hop traversal retrieval, and hybrid graph + vector…
  • 90rag-evaluation
    Evaluate retrieval-augmented-generation pipelines with reference-free metrics to detect silent retrieval and grounding failures. Use when: evaluate rag pipeline, measure faithfulness, context precision recall, detect rag hallucination, reference-free rag metrics, score retrieval quality.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.