docs-grounding-verifier
/docs-grounding-verifier
Weryfikuje na poziomie twierdzeń, czy strony dokumentacji zgadzają się z kodem źródłowym, sprawdza fakty w sekcjach i wskazuje twierdzenia, które odeszły od źródła.
- Autor
- danielmeppiel ↗
- Repozytorium
- apm ↗
- Licencja
- Apache-2.0
- Ostatnia zmiana repozytorium
- 13.11.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 101
- Skrypty
- 4
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 13.11.2026, bez odwołań do wycofanych modeli i API.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
- 4 skrypty w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.
Skrypty: evals/run-evals.sh, scripts/extract-claims.py, scripts/retrieve-evidence.sh, scripts/verify-page.sh
Co robi i kiedy
Weryfikuje na poziomie twierdzeń, czy strony dokumentacji zgadzają się z kodem źródłowym, sprawdza fakty w sekcjach i wskazuje twierdzenia, które odeszły od źródła.
Wyzwalacze z opisu
Oryginalny opis (EN)
Use this skill to verify CLAIM-LEVEL grounding of a documentation page (or set of pages) against the source code. Activate when you have specific pages to check for factual accuracy -- not when sweeping a whole corpus (use docs-corpus-audit for that) and not when triaging a PR diff (use docs-sync for that). Trigger nouns: "is this doc accurate", "verify the page against the code", "fact-check this section", "any claims that drifted from source", "fact-checking", "grounding audit", "drift hunt", "claim verification". Returns per-claim verdicts (GROUNDED | PARTIAL | CONTRADICTED | UNSUPPORTED) w
Jak działa
- Sibling contract
- When to activate
- When NOT to activate
- Architecture (PIPELINE-of-PANELS)
- Phase 1: SCOPE
- Phase 2: EXTRACT (parallel)
- Phase 3: RETRIEVE (deterministic, batched)
- Phase 4: JUDGE (parallel)
- Phase 5: SYNTHESIZE
- Phase 6: ALIGNMENT LOOP (A8)
Podgląd SKILL.md
---
name: docs-grounding-verifier
description: 'Use this skill to verify CLAIM-LEVEL grounding of a documentation page (or set of pages) against the source code. Activate when you have specific pages to check for factual accuracy -- not when sweeping a whole corpus (use docs-corpus-audit for that) and not when triaging a PR diff (use docs-sync for that). Trigger nouns: "is this doc accurate", "verify the page against the code", "fact-check this section", "any claims that drifted from source", "fact-checking", "grounding audit", "drift hunt", "claim verification". Returns per-claim verdicts (GROUNDED | PARTIAL | CONTRADICTED | UNSUPPORTED) with file:line evidence citations. Catches paragraph-level inaccuracies that page-level audit averages over -- e.g. a paragraph with 5 claims where 4 are grounded and 1 is fabricated. Does NOT modify files (returns advisory only); does NOT re-architect the docs; does NOT triage PRs.'
---
# docs-grounding-verifier
CLAIM-LEVEL grounding verification. Adapts the RAGAS faithfulness-eval
pattern (proven in RAG literature) to docs/code instead of generated-
answers/retrieved-context. Source code is the ground truth; docs
paragraphs are the candidate text under audit.
[python-architect persona](../../personas/python-architect.persona.md)
[doc-writer persona](../../personas/doc-writer.persona.md)
## Sibling contract
This skill is a SIBLING of `docs-corpus-audit` and `docs-sync`. The
boundary is load-bearing:
| Skill | Trigger | Scope | Granularity |
| ---------------------- | -------------------------------------- | ---------------- | --------------- |
| docs-sync | PR opened/synchronized | PR diff only | Page-level |
| docs-corpus-audit | Maintainer asks for whole-corpus pass | Entire corpus | Page-level |
| **docs-grounding-verifier** | **Verify specific pages factually** | **1..N pages** | **CLAIM-level** |
`docs-corpus-audit` invokes this skill in its VERIFY phase on the
highest-risk pages of each wave. `docs-sync` can invoke it on the
specific pages in a PR diff. The skill is also runnable standalone.
## When to activate
- Maintainer says "verify <page> against the code".
- An audit wave wants per-claim grounding scores for its highest-risk pages.
- A PR review wants to confirm that prose changes are not just plausible
but actually consistent with the implementation.
- A "fact-check" or "grounding" or "drift hunt" request.
## When NOT to activate
- Whole-corpus sweep with no specific page list -> use `docs-corpus-audit`.
- PR review with mixed code+docs diff -> use `docs-sync`.
- Editorial / tone review -> use `editorial-owner` persona directly.
## Architecture (PIPELINE-of-PANELS)
```
PARENT
-> [Stage 1: EXTRACT claims, fan-out PANEL]
per page -> LLM extracts atomic factual claims as JSON
script: scripts/extract-claims.py
-> [Stage 2: RETRIEVE evidence, deterministic S7]
per claim -> grep over src/ via keywords + hints
script: scripts/retrieve-evidence.sh (NO LLM)
-> [Stage 3: JUDGE grounding, adversarial A7]
per (claim, evidence) -> LLM rules GROUNDED|PARTIAL|CONTRADICTED|UNSUPPORTED
asset: assets/judge-prompt.md
-> [Stage 4: SYNTHESIZE]
aggregate ungrounded -> doc-writer for fix
re-verify after fix (A8 ALIGNMENT LOOP)
```
Stage 2 is the load-bearing design choice: evidence retrieval is
DETERMINISTIC (grep + AST hints), not LLM. The judge in Stage 3 can
only rule on evidence it actually receives -- it cannot hallucinate
support that the retriever did not find. This is the structural
guard against the failure mode "the LLM convinces itself the docs
match the code."
## Phase 1: SCOPE
Input: list of page paths to verify (1..N). If a `risk_class` is
attached (e.g. "high-stakes"), prefer it; otherwise treat all as equal.
Out-of-scope:
- Pages outside `docs/src/content/docs/` or
`packages/apm-guide/.apm/skills/apm-usage/`.
- Pages with no factual claims (pure editorial / landing). Skip
rather than force-extract.
## Phase 2: EXTRACT (parallel)
For each page, dispatch ONE claim-extractor agent:
- Prompt template: `scripts/extract-claims.py <page>` produces the
prompt and embeds the page content.
- Returns: JSON `{"page", "claims":[{"id","text","section","keywords",
"expected_source_areas"}]}` capped at 15 claims per page.
Parallel safe; no shared state between extractors.
## Phase 3: RETRIEVE (deterministic, batched)
For each claim, pipe to `scripts/retrieve-evidence.sh`:
- Uses keywords + expected_source_areas to grep src/.
- Returns one-line JSON: `{"claim_id","claim_text","evidence":[...],
"evidence_count"}`.
Sequential is fine (grep is fast). No LLM. Diagnostics on stderr,
data on stdout.
## Phase 4: JUDGE (parallel)
For each (claim, evidence) tuple, dispatch ONE grounding-judge agent:
- Load `assets/judge-prompt.md`.
- Send the prompt + the tuple.
- Returns: JSON verdict per the schema in `judge-prompt.md`.
Batching across claims-of-one-page into a single judge call is fine
(prompt with all tuples at once). Across pages, fan out.
## Phase 5: SYNTHESIZE
Aggregate verdicts. Materialize the report:
```
{
"summary": {
"pages_verified": N,
"claims_total": N,
"grounded": N, "partial": N, "contradicted": N, "unsupported": N,
"grounding_rate": N/total
},
"actionable": [
{"page", "claim", "verdict", "evidence_cited", "fix_suggestion"}
]
}
```
CONTRADICTED and PARTIAL are doc-writer work items. UNSUPPORTED is
split: if `retrieval_fix_suggestion` is plausible, retry retrieval
with the suggested keywords; if still empty, treat as CONTRADICTED.
## Phase 6: ALIGNMENT LOOP (A8)
Hand actionable items to doc-writer (one subagent per page). After
edits, RE-RUN the pipeline on the same pages. The grounding_rate
must MONOTONICALLY INCREASE between iterations or the loop has
diverged -- stop and escalate to the operator.
## Ship gate
- grounding_rate >= 0.9 on each verified page after the alignment loop.
- Every CONTRADICTED claim cited a specific code file:line that
disproves it -- not vague "the code doesn't say that".
- The eval-runner (see `evals/`) passes on the trigger evals and
the content evals before the skill is treated as production-ready.
## Bundled assets
- `scripts/extract-claims.py` -- Stage 1 prompt builder. `--help`, `--schema`.
- `scripts/retrieve-evidence.sh` -- Stage 2 retriever. Deterministic. `--help`.
- `scripts/verify-page.sh` -- end-to-end orchestrator. `--help`.
- `assets/judge-prompt.md` -- Stage 3 adversarial judge prompt.
- `evals/trigger-evals.json` -- 20 dispatch queries (10 should, 10 shouldn't).
- `evals/content-evals.json` -- seeded-drift recall scenarios.
- `evals/run-evals.sh` -- the eval-runner that turns JSON into metrics.
## Failure modes guarded against
- **Hallucinated grounding**: Stage 2 is deterministic; judge sees only
real evidence.
- **Adversarial weakness**: Stage 3 prompt defaults to SKEPTICAL.
- **Page-level averaging**: claim-level granularity surfaces partials.
- **Bundle leakage**: design notes / one-time scripts stay in session
state, never in `references/`.
- **Phantom dependency**: SKILL.md links its persona deps via relative
paths; A9 PROBE before invoking docs-corpus-audit's substrate.
- **Dispatch collision** with sibling skills: trigger-eval validation
split is the ship gate (must distinguish from docs-sync /
docs-corpus-audit triggers).
Pliki w paczce (101)
- SKILL.md 7 KB
- assets/judge-prompt.md 3 KB
- evals/content-evals.json 2 KB
- evals/run-evals.sh 5 KB
- evals/trigger-evals.json 2 KB
- evals/runs/20260527-194228/trigger-prompts.txt 2 KB
- evals/runs/20260527-194228/seeded-corpus/drift-install-flag/page.md 14 KB
- evals/runs/20260527-194228/seeded-corpus/drift-install-flag/scenario.json 1 KB
- evals/runs/20260527-194228/seeded-corpus/drift-policy-reject/page.md 7 KB
- evals/runs/20260527-194228/seeded-corpus/drift-policy-reject/scenario.json 1 KB
- evals/runs/20260527-194228/seeded-corpus/drift-registry-resolver/page.md 21 KB
- evals/runs/20260527-194228/seeded-corpus/drift-registry-resolver/scenario.json 1 KB
- evals/runs/proof/trigger-responses.json 1 KB
- evals/runs/proof/claims/copilot.json 4 KB
- evals/runs/proof/claims/install.json 5 KB
- evals/runs/proof/claims/pkgtypes.json 4 KB
- evals/runs/proof/claims/policy.json 4 KB
- evals/runs/proof/claims/registries.json 5 KB
- evals/runs/proof/copilot/judge-batch-docs_src_content_docs_integrations_copilot-app_md.txt 45 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c1.json 3 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c10.json 2 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c11.json 2 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c12.json 3 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c13.json 2 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c14.json 2 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c15.json 2 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c2.json 3 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c3.json 3 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c4.json 3 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c5.json 3 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c6.json 2 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c7.json 2 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c8.json 3 KB
- evals/runs/proof/copilot/evidence/docs_src_content_docs_integrations_copilot-app_md_c9.json 3 KB
- evals/runs/proof/install/judge-batch-docs_src_content_docs_reference_cli_install_md.txt 38 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c1.json 3 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c10.json 2 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c11.json 1 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c12.json 2 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c13.json 2 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c14.json 3 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c15.json 3 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c2.json 1 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c3.json 2 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c4.json 1 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c5.json 2 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c6.json 2 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c7.json 5 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c8.json 1 KB
- evals/runs/proof/install/evidence/docs_src_content_docs_reference_cli_install_md_c9.json 1 KB
- evals/runs/proof/pkgtypes/judge-batch-docs_src_content_docs_reference_package-types_md.txt 51 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c1.json 3 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c10.json 2 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c11.json 2 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c12.json 2 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c13.json 3 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c14.json 2 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c15.json 4 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c2.json 4 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c3.json 3 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c4.json 4 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c5.json 3 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c6.json 3 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c7.json 3 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c8.json 2 KB
- evals/runs/proof/pkgtypes/evidence/docs_src_content_docs_reference_package-types_md_c9.json 2 KB
- evals/runs/proof/policy/judge-batch-docs_src_content_docs_enterprise_apm-policy_md.txt 51 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c1.json 4 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c10.json 2 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c11.json 2 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c12.json 2 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c13.json 3 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c14.json 3 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c15.json 4 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c2.json 3 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c3.json 3 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c4.json 3 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c5.json 2 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c6.json 3 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c7.json 2 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c8.json 2 KB
- evals/runs/proof/policy/evidence/docs_src_content_docs_enterprise_apm-policy_md_c9.json 1 KB
- evals/runs/proof/registries/judge-batch-docs_src_content_docs_guides_registries_md.txt 49 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c1.json 3 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c10.json 2 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c11.json 3 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c12.json 3 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c13.json 2 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c14.json 3 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c15.json 3 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c2.json 3 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c3.json 2 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c4.json 3 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c5.json 4 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c6.json 3 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c7.json 1 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c8.json 3 KB
- evals/runs/proof/registries/evidence/docs_src_content_docs_guides_registries_md_c9.json 2 KB
- scripts/extract-claims.py 3 KB
- scripts/retrieve-evidence.sh 3 KB
- scripts/verify-page.sh 2 KB
Wersje
W bazie nie ma innych skilli o nazwie docs-grounding-verifier.
Migawka bazy z 13.11.2026 · wersja paczki ffec794d5653 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100verify-openspec-docs
Sprawdza fakty w dokumentacji użytkownika OpenSpec przez subagenta ze świeżym kontekstem, który ponownie uruchamia polecenia i porównuje twierdzenia ze źródłem; uruchamiany ręcznie. - 100docs-corpus-audit
Przeprowadza całościowy przegląd dokumentacji microsoft/apm względem bieżącego kodu, strona po stronie, i wskazuje punktowe poprawki nieaktualnych twierdzeń, np. przed wydaniem. - 100docs-corpus-audit
Przeprowadza całościowy przegląd dokumentacji microsoft/apm względem bieżącego kodu, strona po stronie, i wskazuje punktowe poprawki nieaktualnych twierdzeń, np. przed wydaniem. - 90epistemic-grounding
Sprawdza, czy twierdzenia mają pokrycie w dowodach: przypisuje poziomy pewności, weryfikuje fakty i odróżnia wskazówkę od ustalenia. - 100verify-both-ways
Sprawdza twierdzenie w obie strony (fact-check) lub poprawia na miejscu twierdzenia w artefakcie albo dokumencie HTML, w trybach claim i artifact. - 100fact-checking
Weryfikuje twierdzenia techniczne w kodzie, dokumentacji i komentarzach przed scaleniem zmian, sprawdzając, czy zapisane stwierdzenia są prawdziwe.