audit-consensus
/audit-consensus
Łączy kilka niezależnych audytów powtarzalności tej samej publikacji w jeden raport konsensusu z potwierdzonymi ustaleniami.
- Autor
- poldracklab ↗
- Repozytorium
- scientific-audit-skills ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 09.11.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 8
- Skrypty
- 5
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 09.11.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
- 5 skryptów w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.
Skrypty: scripts/consensus.py, scripts/consensus.sh, scripts/discover.py, scripts/redact.py, scripts/unblind.py
Co robi i kiedy
Łączy kilka niezależnych audytów powtarzalności tej samej publikacji w jeden raport konsensusu z potwierdzonymi ustaleniami.
Oryginalny opis (EN)
Reconcile several independent, blinded reproducibility audits of the same paper into one consensus report. Use when a consensus folder made by consensus.sh (inputs.json present) needs reconciling, or when the user asks which audit findings are confirmed across runs, how severe they are, and whether they change the paper's major conclusions. Matches claims and findings across audits by meaning. Distinguishes converged, disputed and untested findings, and runs whatever additional analysis against the paper's data and code is needed to settle disagreements. Compare the included audits' accuracy a
Jak działa
- Intake
- Stage 1 — Extract
- Stage 2 — Cross-walk
- Stage 3 — Classify
- Stage 4 — Adjudicate
- Published critiques and power
- Finding classification
- Stage 5 — Severity and conclusion change
- Stage 6 — Compare the audits
- Stage 7 — Report
Podgląd SKILL.md
---
name: audit-consensus
description: >
Reconcile several independent, blinded reproducibility audits of the same paper into one
consensus report. Use when a consensus folder made by consensus.sh (inputs.json present)
needs reconciling, or when the user asks which audit findings are confirmed across runs,
how severe they are, and whether they change the paper's major conclusions. Matches claims
and findings across audits by meaning. Distinguishes converged, disputed and untested
findings, and runs whatever additional analysis against the paper's data and code is
needed to settle disagreements. Compare the included audits' accuracy and usefulness,
allowing ties. Not a fresh audit.
---
# Consensus of independent audits
Goal: a **private, self-contained rendered Quarto report** (`consensus.html`) that answers,
for one paper: which issues raised by the audits are confirmed, how severe each is, and
whether any changes a major conclusion. Settle every disagreement with evidence, never by
counting audits or by trusting the more confident report.
Then judge the relative accuracy and usefulness of the included audits from that evidence.
Work through the stages in order. Start from `templates/consensus_template.qmd`.
## Intake
1. Your working directory is a consensus folder `runs/<paper_id>/consensus/<consensus_id>/`
created by `consensus.sh`. Read `inputs.json`: it lists the completed runs to reconcile
(`runs[].run_id`) and any runs skipped because their audit was unfinished. Each source
run is at `../../<run_id>/`. Skipped runs are not inputs; do not read them. Never read
other folders under `consensus/`: earlier consensus runs hold the model mapping and
earlier verdicts, and would break blinding and anchor your adjudication.
2. Refer to audits by run ID only. Never read `registry.jsonl`, never guess which model
wrote an audit, and never write a model name into any file. Model names in the source
runs have been replaced with run IDs or `[redacted-model]`. The operator attaches
the run-ID → model mapping after you finish.
3. Treat the source runs as read-only. You may read anything in them: `paper/`, `data/`,
`manifest.json`, `README.md`, and each run's `audit/` (report, code, caches, results).
To re-run a source audit's code, copy it into your folder first and run the copy.
4. Write everything under the consensus folder: `consensus.qmd`, `R/` for adjudication
code, `results/` for its outputs, `additional-evidence/` for retrieved material.
## Stage 1 — Extract
For each run, read `audit/audit.qmd` (and `audit.html` where inline values matter) and list:
- its **claims** with their verdicts;
- its **findings** (ID, classification and rationale when present, status line, severity, affected claim, key numbers);
- what it **checked and found clean**: rows of its robustness table marked as not changing
the verdict, its *Clean checks* and *Verification performed* appendices, and robustness
output it reports.
The third list decides whether silence means "checked, no problem" or "never looked".
## Stage 2 — Cross-walk
Match claims across runs by meaning, not by ID. The runs number claims and findings
independently, and one run's claim may split or merge another's. Produce one consensus
claim list. Then match every finding from every run to a consensus item. Two findings
match when they identify the same problem in the same analysis, even when they reach it
by different routes (for example estimator arithmetic versus simulation). Every finding ID
from every run appears in the cross-walk table; nothing is dropped silently.
## Stage 3 — Classify
Each consensus finding gets exactly one status:
- **Converged**: two or more audits raised it independently and none tested the same
point and found it clean.
- **Disputed**: at least one audit raised it, and **any** audit explicitly tested the same
point and found no problem. This holds however many audits raised it: one explicit clean
result makes it disputed.
- **Single-audit, untested**: one audit raised it and no other audit looked.
Audits that never looked are recorded in the cross-walk but do not change the status.
Never classify by majority, and never by which report sounds more certain.
## Stage 4 — Adjudicate
- **Disputed** and **untested** findings: run the check yourself against the paper's data
and code, in `R/` with output in `results/`. Find out *why* the audits differ (different
baseline, different replication unit, a code error in one run, a different reading of the
paper). Give an outcome: **confirmed**, **partly confirmed**, or **refuted**. If an
existing output in a source run already settles the question, cite that exact output
instead of recomputing it, and say why it suffices.
- **Converged** findings: apply the multiplicity rule below even for converged findings.
For other converged findings, accept them, but spot-check any whose audits disagree on
magnitude or severity, and report which magnitude holds.
- **Claim verdicts that differ** between audits: decide the consensus verdict from the
adjudicated findings and state why each audit's verdict was or was not right.
- Cache heavy computations (e.g. `results/*.rds`) and read them in the report. The
operator re-renders the report after you finish, and every number must come out the
same.
Scope: adjudicate what the audits raised and the verdicts they disagree on. Do not hunt for
new issues. A new problem you notice while adjudicating goes in *Noticed during
adjudication*, unadjudicated, one line each, but only when it could change a verdict.
Leave out anything smaller. Additional evidence follows the
reproducibility-audit rules: save it under `additional-evidence/` with a provenance row,
with no large downloads, no paid jobs, and no contacting authors.
### Justify the multiple-comparison family
Before alleging missing multiple-comparison correction, establish family membership
from the scientific question, the authors' inferential claims, and the analysis plan,
preregistration, methods, and code where available. List the included tests and explain
why they offer parallel opportunities to support the same inference. Sharing the same
figure, table, dataset, or paper does not establish a family; distinct scientific
questions can legitimately be tested separately.
Focus correction checks primarily on large collections of essentially identical tests
performed in parallel, such as correlations across a set of variables or tests across
many brain voxels. This is a priority, not a minimum family size: smaller families also
need justification from their shared inferential purpose. Check any existing correction,
hierarchical testing, or omnibus procedure before alleging an omission.
Do not apply Holm automatically. Justify both the family and the error criterion
(e.g. family-wise error versus false discovery rate), and assess whether the authors'
method addresses their claim. Report all tests in the justified family, including null
results and effects in the opposite direction.
If family membership is uncertain, describe that uncertainty. An alternative grouping
may be a sensitivity analysis, but do not downgrade a claim solely because it fails
correction under a debatable grouping. A justified correction that changes significance
must be interpreted alongside effect size, uncertainty, and the actual claim; it does
not establish that the effect is absent.
## Published critiques and power
Two topics are an exception to the no-new-issues scope above, because the audits
are required to cover them and an omission there is a gap in the consensus, not a
new issue.
**Published critiques.** Build the union of the audits' critique ledgers, merging
the same point from different audits. If no audit searched, or the searches
differ in what they found, run the critique search yourself following the
reproducibility-audit *Published critiques* rules (commentaries, letters, replies,
failed replications and reanalyses, PubPeer, methodological papers using the paper
as an example), save what you find under `additional-evidence/`, and record the
search. Assess every critique point that no audit assessed, and adjudicate points
the audits assessed differently, like any other disputed finding. A supported point
that affects a claim enters the cross-walk as a consensus finding.
**Power and precision.** For every main claim, confirm that the effective sample
size, the interval for the deciding estimate, and the smallest effect the design
could detect with 80% power at the paper's alpha are recorded. Where the audits
omitted it or disagree, compute it yourself (a sensitivity power analysis, never
observed power) in `R/`. When the precision is inadequate for what a main claim
asserts (its magnitude, generalization, comparison between effects, or the absence
of an effect), discuss it in that claim's *Evidence* and *If weakened* and let it
set the verdict, even when the source audits treated it as minor or as a disclosed
limitation. Insufficient power that limits several central claims is a paper-wide
problem. A small sample alone is not a Defect; classify the inference it supports.
## Finding classification
Classify every retained finding as **Defect**, **Contested choice**, or
**Undetermined**, separate from evidence status and severity. Give a brief
classification rationale beside the finding, including minor findings.
- **Defect:** something wrong under any defensible standard applicable to the
stated claim and design. Identify the violated arithmetic, logical, factual,
or methodological requirement and the evidence establishing the violation.
Consider the strongest defensible rationale for the authors' approach before
using this label; preference for another method is insufficient.
- **Contested choice:** researchers can reasonably disagree because more than
one approach is defensible. State the authors' strongest defensible rationale,
the alternative, and the assumptions or scientific aims that distinguish them.
Explain any demonstrated consequence without calling the alternative a
correction. A sensitivity analysis can qualify robustness, but sensitivity
alone does not establish a defect.
- **Undetermined:** available evidence cannot settle this distinction. Specify
what would resolve it; missing evidence is not proof of a defect. Use this
for an unverified implementation concern, not Contested choice unless there
are identifiable defensible alternatives.
A wrong denominator is a Defect; different results from two defensible covariate
models are a Contested choice; unavailable code that prevents checking clustering
is Undetermined. A confirmed sensitivity result can remain a Contested choice,
and a Defect can be minor. A preregistration deviation alone does not prove the
chosen method invalid; assess disclosure and claims of adherence separately.
Split mixed findings so a factual error does not turn a defensible choice into a
defect. Summaries and recommendations must preserve these distinctions: explain
whether a claim changes because of a defect, dependence on contested assumptions,
or unresolved evidence. Do not count all retained findings as errors.
## Stage 5 — Severity and conclusion change
Converged does not mean Defect; Disputed does not mean Contested choice. Agreement
between auditors is distinct from whether the authors' choice is defensible.
Independently reassess the classification and its rationale even for converged
findings. For older audits without these labels, derive them from the underlying
evidence; do not infer Defect from severity or an error label. Record source and
consensus classifications and reasons for changes in adjudication notes. Refuted
allegations remain in the cross-walk as refuted; mark their classification not
applicable rather than presenting them as retained defects.
In the main body, label each paper-wide or claim-specific finding and explain its
classification rationale where its consequence is discussed. Include classification
and rationale in minor/cosmetic issue tables and the finding cross-walk.
For every confirmed or partly confirmed finding, state its **severity** on the audits'
scale (major / moderate / minor: the effect on the named claim). Then state **whether it
changes a major conclusion**. It does so **only when it moves that claim's consensus
verdict to weak or unsupported**. "Hypothesis-generating" for a claim the paper presents as
established counts. A downgrade such as strong → moderate is reported, but it does not
count as changing the conclusion. Apply this to every claim in the report, explaining any weakening even when it does not
meet this threshold.
## Stage 6 — Compare the audits
After adjudication, assess **every included run**, using run IDs only. Keep two judgments
separate:
- **Accuracy**: correctness of the audit's numerical results, methodological criticisms,
clean checks, severity assessments, and claim verdicts. Identify material errors,
refuted allegations, overstatements, and appropriately expressed uncertainty.
- **Usefulness**: how much the audit helps a reader understand what holds and what changes
in the paper. Credit consequential confirmed findings, informative clean checks,
reproducible evidence, clear explanations, and actionable qualifications. Explain the
cost of misleading allegations or distracting detail.
Base comparisons on **adjudicated evidence**, citing cross-walk items, claim verdicts,
and exact outputs where relevant. Weight scientific consequence rather than raw finding
counts, length, confidence of tone, or agreement with the majority. Compare scope and access
to materials before treating an omission as a weakness; silence alone is not an error.
Do not treat unresolved findings as refuted or an untested clean check as established.
Give a relative judgment for accuracy and another for usefulness. Name the leading run
or tied group and explain material differences among the remaining runs. Use a **tie**
whenever the evidence does not clearly distinguish runs; distinguish demonstrated similar
performance from insufficient evidence to separate them. Accuracy and usefulness may have
different leaders. Do not force an overall winner or numerical score.
State the limits: this comparison concerns these audits of this paper, not a general model
ranking. Consensus adjudication is itself fallible, and accepted converged findings are
not necessarily independently verified. Explain when these limits prevent a distinction.
## Stage 7 — Report
Write for a reader who has not read the audits. Use plain language and short, connected
paragraphs. Be concise, but give enough context to understand each claim, its evidence,
and what the audit changes. There is no fixed word cap; length should follow the number
and complexity of the main claims. Start from the template and use this order:
1. **Paper summary**: briefly explain the research question, study design, sample or
data, main analyses, and the paper's central conclusions.
2. **Paper-wide problems**: if confirmed or partly confirmed major problems affect the
paper as a whole, briefly explain each problem, the evidence establishing it, and
its impact on the paper's conclusions. Distinguish demonstrated impact from uncertainty
that prevents verification. If there are no such problems, omit this section.
A serious problem confined to one claim belongs with that claim.
3. **Main claims**: give each main claim a descriptive subsection heading and these
four elements, in order:
- **Claim**: explain what the authors assert, including the population or scope.
- **Evidence**: describe the evidence the paper offers: relevant design, measurements,
comparisons, and results. Include deciding estimates and uncertainty when available,
the effective sample size, and whether the design could detect the claimed effect,
with a figure, table, or analysis reference. Do this for claims that survive too.
- **Holds after audit**: explicitly say whether it holds, holds with qualifications,
is weakened, is unsupported, or cannot be determined with the available evidence.
Give the consensus strength verdict and explain its meaning for this claim.
An untested claim is not a verified claim.
- **If weakened**: explain the audit evidence that weakens it and the scientific
impact: what narrower conclusion remains defensible, what no longer follows, and
what is still unresolved. Distinguish reanalysis results from plausible concerns.
If it is unchanged, say so briefly; do not invent a weakness.
Describe a paper-wide problem once in full, then explain its specific impact on each
affected claim. Readers must not need to decode an issue ID or consult the appendix to
understand a verdict. Brief repetition that makes a claim understandable is appropriate.
Appendix:
- **Relative audit accuracy and usefulness**: one row per included run, with separate
accuracy and usefulness assessments and deciding evidence; then separate comparative
judgments, including ties and their reasons, and the limits of the comparison.
- **Published critiques**: the search record and one row per critique point, with its
assessment, the finding or check that tested it, and its disposition.
- **Minor issues**: changes to a reported number, test, or result label that do not
change a claim verdict; give the location and effect.
- **Noticed during adjudication**: unadjudicated concerns that could change a verdict,
clearly labelled as unconfirmed; otherwise "None."
- **Claim verdicts by audit**: per-run claim IDs and verdicts, with reasons for differences
from the consensus.
- **Finding cross-walk**: every finding ID of every run, consensus item, status, outcome,
classification and rationale, severity, affected claims, and whether it changes a major conclusion.
- **Adjudication notes**: evidence settling disputed or refuted findings, revised
magnitudes or severities, and why audits differed. Include the family rationale for
multiplicity findings, even when converged.
- **Cosmetic issues**: typos, labels, rounding or missing details that change no result
or conclusion, with locations.
- **Adjudication analyses**, provenance, and session info.
Run IDs and finding IDs stay in the appendix. Consensus IDs may be used as optional
links to detailed evidence, but never as substitutes for explanations in the main body.
## Stage 8 — Render and verify
Render `consensus.qmd` to `consensus.html` with embedded resources. Check every number in
the prose against the rendered output. Read the main body as a standalone explanation:
does it summarize the paper, explain any paper-wide problems and their impact, and give
each main claim its evidence, audit verdict, and the evidence and impact of any weakening?
Remove unexplained shorthand and unnecessary detail without losing those elements.
Confirm the cross-walk lists every finding ID of every run in `inputs.json`, and that
multiplicity criticisms justify the family and account for the authors' correction.
Confirm every published critique point has a disposition and every main claim has a
power and precision row.
Confirm every included run has an evidence-based accuracy and usefulness assessment,
and that both relative judgments state any ties and limitations.
Give the operator the file path. Do not publish it.
Pliki w paczce (8)
- README.md 1 KB
- SKILL.md 19 KB
- scripts/consensus.py 6 KB
- scripts/consensus.sh 6 KB
- scripts/discover.py 2 KB
- scripts/redact.py 4 KB
- scripts/unblind.py 1 KB
- templates/consensus_template.qmd 7 KB
Wersje
W bazie nie ma innych skilli o nazwie audit-consensus.
Migawka bazy z 13.11.2026 · wersja paczki 987c95c3ae02 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 90audit-summary
Podsumowuje zakończone audyty naukowe wielu prac, z liczbami przejrzystości oraz potwierdzonymi ustaleniami technicznymi i interpretacyjnymi. - 100audit-prep
Przygotowuje foldery artykułów do audytów powtarzalności i trafności badań neuroobrazowych z izolowanymi przebiegami. - 100paper-verification
Weryfikuje twierdzenia artykułu względem kodu i danych, sprawdza zgodność liczb, wzorów z kodem oraz poprawność cytowań. - 100reproducibility-audit
Prowadzi audyt odtwarzalności publikacji: pobiera dane i kod autorów, odtwarza wyniki w R i Quarto oraz ocenia siłę dowodów. - 95bio-consensus-sequences
Generuje sekwencje konsensusowe FASTA przez nakładanie wariantów z VCF na referencję (bcftools consensus) lub budowę konsensusu wirusowego i amplikonowego przez iVar. - 100paper-finder
Znajduje zweryfikowane DOI prac naukowych potrzebnych badaczowi, bez wymyślania publikacji, i sprawdza je w Crossref.