Przejdź do treści
Biblioteka skilli
Nauka i badania · Analiza

audit-consensus

/audit-consensus

Łączy kilka niezależnych audytów powtarzalności tej samej publikacji w jeden raport konsensusu z potwierdzonymi ustaleniami.

Licencja
MIT
Ostatnia zmiana repozytorium
09.11.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
8
Skrypty
5
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 09.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co sprawdzić przed instalacją
  • 5 skryptów w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.

Skrypty: scripts/consensus.py, scripts/consensus.sh, scripts/discover.py, scripts/redact.py, scripts/unblind.py

Co robi i kiedy

Łączy kilka niezależnych audytów powtarzalności tej samej publikacji w jeden raport konsensusu z potwierdzonymi ustaleniami.

Oryginalny opis (EN)

Reconcile several independent, blinded reproducibility audits of the same paper into one consensus report. Use when a consensus folder made by consensus.sh (inputs.json present) needs reconciling, or when the user asks which audit findings are confirmed across runs, how severe they are, and whether they change the paper's major conclusions. Matches claims and findings across audits by meaning. Distinguishes converged, disputed and untested findings, and runs whatever additional analysis against the paper's data and code is needed to settle disagreements. Compare the included audits' accuracy a

Jak działa

  1. Intake
  2. Stage 1 — Extract
  3. Stage 2 — Cross-walk
  4. Stage 3 — Classify
  5. Stage 4 — Adjudicate
  6. Published critiques and power
  7. Finding classification
  8. Stage 5 — Severity and conclusion change
  9. Stage 6 — Compare the audits
  10. Stage 7 — Report
Sekcje z nagłówków SKILL.md, w oryginale · 299 linii, 2 749 słów
Podgląd SKILL.md
---
name: audit-consensus
description: >
  Reconcile several independent, blinded reproducibility audits of the same paper into one
  consensus report. Use when a consensus folder made by consensus.sh (inputs.json present)
  needs reconciling, or when the user asks which audit findings are confirmed across runs,
  how severe they are, and whether they change the paper's major conclusions. Matches claims
  and findings across audits by meaning. Distinguishes converged, disputed and untested
  findings, and runs whatever additional analysis against the paper's data and code is
  needed to settle disagreements. Compare the included audits' accuracy and usefulness,
  allowing ties. Not a fresh audit.
---

# Consensus of independent audits

Goal: a **private, self-contained rendered Quarto report** (`consensus.html`) that answers,
for one paper: which issues raised by the audits are confirmed, how severe each is, and
whether any changes a major conclusion. Settle every disagreement with evidence, never by
counting audits or by trusting the more confident report.
Then judge the relative accuracy and usefulness of the included audits from that evidence.

Work through the stages in order. Start from `templates/consensus_template.qmd`.

## Intake

1. Your working directory is a consensus folder `runs/<paper_id>/consensus/<consensus_id>/`
   created by `consensus.sh`. Read `inputs.json`: it lists the completed runs to reconcile
   (`runs[].run_id`) and any runs skipped because their audit was unfinished. Each source
   run is at `../../<run_id>/`. Skipped runs are not inputs; do not read them. Never read
   other folders under `consensus/`: earlier consensus runs hold the model mapping and
   earlier verdicts, and would break blinding and anchor your adjudication.
2. Refer to audits by run ID only. Never read `registry.jsonl`, never guess which model
   wrote an audit, and never write a model name into any file. Model names in the source
   runs have been replaced with run IDs or `[redacted-model]`. The operator attaches
   the run-ID → model mapping after you finish.
3. Treat the source runs as read-only. You may read anything in them: `paper/`, `data/`,
   `manifest.json`, `README.md`, and each run's `audit/` (report, code, caches, results).
   To re-run a source audit's code, copy it into your folder first and run the copy.
4. Write everything under the consensus folder: `consensus.qmd`, `R/` for adjudication
   code, `results/` for its outputs, `additional-evidence/` for retrieved material.

## Stage 1 — Extract

For each run, read `audit/audit.qmd` (and `audit.html` where inline values matter) and list:

- its **claims** with their verdicts;
- its **findings** (ID, classification and rationale when present, status line, severity, affected claim, key numbers);
- what it **checked and found clean**: rows of its robustness table marked as not changing
  the verdict, its *Clean checks* and *Verification performed* appendices, and robustness
  output it reports.

The third list decides whether silence means "checked, no problem" or "never looked".

## Stage 2 — Cross-walk

Match claims across runs by meaning, not by ID. The runs number claims and findings
independently, and one run's claim may split or merge another's. Produce one consensus
claim list. Then match every finding from every run to a consensus item. Two findings
match when they identify the same problem in the same analysis, even when they reach it
by different routes (for example estimator arithmetic versus simulation). Every finding ID
from every run appears in the cross-walk table; nothing is dropped silently.

## Stage 3 — Classify

Each consensus finding gets exactly one status:

- **Converged**: two or more audits raised it independently and none tested the same
  point and found it clean.
- **Disputed**: at least one audit raised it, and **any** audit explicitly tested the same
  point and found no problem. This holds however many audits raised it: one explicit clean
  result makes it disputed.
- **Single-audit, untested**: one audit raised it and no other audit looked.

Audits that never looked are recorded in the cross-walk but do not change the status.
Never classify by majority, and never by which report sounds more certain.

## Stage 4 — Adjudicate

- **Disputed** and **untested** findings: run the check yourself against the paper's data
  and code, in `R/` with output in `results/`. Find out *why* the audits differ (different
  baseline, different replication unit, a code error in one run, a different reading of the
  paper). Give an outcome: **confirmed**, **partly confirmed**, or **refuted**. If an
  existing output in a source run already settles the question, cite that exact output
  instead of recomputing it, and say why it suffices.
- **Converged** findings: apply the multiplicity rule below even for converged findings.
  For other converged findings, accept them, but spot-check any whose audits disagree on
  magnitude or severity, and report which magnitude holds.
- **Claim verdicts that differ** between audits: decide the consensus verdict from the
  adjudicated findings and state why each audit's verdict was or was not right.
- Cache heavy computations (e.g. `results/*.rds`) and read them in the report. The
  operator re-renders the report after you finish, and every number must come out the
  same.

Scope: adjudicate what the audits raised and the verdicts they disagree on. Do not hunt for
new issues. A new problem you notice while adjudicating goes in *Noticed during
adjudication*, unadjudicated, one line each, but only when it could change a verdict.
Leave out anything smaller. Additional evidence follows the
reproducibility-audit rules: save it under `additional-evidence/` with a provenance row,
with no large downloads, no paid jobs, and no contacting authors.

### Justify the multiple-comparison family

Before alleging missing multiple-comparison correction, establish family membership
from the scientific question, the authors' inferential claims, and the analysis plan,
preregistration, methods, and code where available. List the included tests and explain
why they offer parallel opportunities to support the same inference. Sharing the same
figure, table, dataset, or paper does not establish a family; distinct scientific
questions can legitimately be tested separately.

Focus correction checks primarily on large collections of essentially identical tests
performed in parallel, such as correlations across a set of variables or tests across
many brain voxels. This is a priority, not a minimum family size: smaller families also
need justification from their shared inferential purpose. Check any existing correction,
hierarchical testing, or omnibus procedure before alleging an omission.

Do not apply Holm automatically. Justify both the family and the error criterion
(e.g. family-wise error versus false discovery rate), and assess whether the authors'
method addresses their claim. Report all tests in the justified family, including null
results and effects in the opposite direction.

If family membership is uncertain, describe that uncertainty. An alternative grouping
may be a sensitivity analysis, but do not downgrade a claim solely because it fails
correction under a debatable grouping. A justified correction that changes significance
must be interpreted alongside effect size, uncertainty, and the actual claim; it does
not establish that the effect is absent.

## Published critiques and power

Two topics are an exception to the no-new-issues scope above, because the audits
are required to cover them and an omission there is a gap in the consensus, not a
new issue.

**Published critiques.** Build the union of the audits' critique ledgers, merging
the same point from different audits. If no audit searched, or the searches
differ in what they found, run the critique search yourself following the
reproducibility-audit *Published critiques* rules (commentaries, letters, replies,
failed replications and reanalyses, PubPeer, methodological papers using the paper
as an example), save what you find under `additional-evidence/`, and record the
search. Assess every critique point that no audit assessed, and adjudicate points
the audits assessed differently, like any other disputed finding. A supported point
that affects a claim enters the cross-walk as a consensus finding.

**Power and precision.** For every main claim, confirm that the effective sample
size, the interval for the deciding estimate, and the smallest effect the design
could detect with 80% power at the paper's alpha are recorded. Where the audits
omitted it or disagree, compute it yourself (a sensitivity power analysis, never
observed power) in `R/`. When the precision is inadequate for what a main claim
asserts (its magnitude, generalization, comparison between effects, or the absence
of an effect), discuss it in that claim's *Evidence* and *If weakened* and let it
set the verdict, even when the source audits treated it as minor or as a disclosed
limitation. Insufficient power that limits several central claims is a paper-wide
problem. A small sample alone is not a Defect; classify the inference it supports.

## Finding classification

Classify every retained finding as **Defect**, **Contested choice**, or
**Undetermined**, separate from evidence status and severity. Give a brief
classification rationale beside the finding, including minor findings.

- **Defect:** something wrong under any defensible standard applicable to the
  stated claim and design. Identify the violated arithmetic, logical, factual,
  or methodological requirement and the evidence establishing the violation.
  Consider the strongest defensible rationale for the authors' approach before
  using this label; preference for another method is insufficient.
- **Contested choice:** researchers can reasonably disagree because more than
  one approach is defensible. State the authors' strongest defensible rationale,
  the alternative, and the assumptions or scientific aims that distinguish them.
  Explain any demonstrated consequence without calling the alternative a
  correction. A sensitivity analysis can qualify robustness, but sensitivity
  alone does not establish a defect.
- **Undetermined:** available evidence cannot settle this distinction. Specify
  what would resolve it; missing evidence is not proof of a defect. Use this
  for an unverified implementation concern, not Contested choice unless there
  are identifiable defensible alternatives.

A wrong denominator is a Defect; different results from two defensible covariate
models are a Contested choice; unavailable code that prevents checking clustering
is Undetermined. A confirmed sensitivity result can remain a Contested choice,
and a Defect can be minor. A preregistration deviation alone does not prove the
chosen method invalid; assess disclosure and claims of adherence separately.
Split mixed findings so a factual error does not turn a defensible choice into a
defect. Summaries and recommendations must preserve these distinctions: explain
whether a claim changes because of a defect, dependence on contested assumptions,
or unresolved evidence. Do not count all retained findings as errors.

## Stage 5 — Severity and conclusion change

Converged does not mean Defect; Disputed does not mean Contested choice. Agreement
between auditors is distinct from whether the authors' choice is defensible.
Independently reassess the classification and its rationale even for converged
findings. For older audits without these labels, derive them from the underlying
evidence; do not infer Defect from severity or an error label. Record source and
consensus classifications and reasons for changes in adjudication notes. Refuted
allegations remain in the cross-walk as refuted; mark their classification not
applicable rather than presenting them as retained defects.

In the main body, label each paper-wide or claim-specific finding and explain its
classification rationale where its consequence is discussed. Include classification
and rationale in minor/cosmetic issue tables and the finding cross-walk.

For every confirmed or partly confirmed finding, state its **severity** on the audits'
scale (major / moderate / minor: the effect on the named claim). Then state **whether it
changes a major conclusion**. It does so **only when it moves that claim's consensus
verdict to weak or unsupported**. "Hypothesis-generating" for a claim the paper presents as
established counts. A downgrade such as strong → moderate is reported, but it does not
count as changing the conclusion. Apply this to every claim in the report, explaining any weakening even when it does not
meet this threshold.

## Stage 6 — Compare the audits

After adjudication, assess **every included run**, using run IDs only. Keep two judgments
separate:

- **Accuracy**: correctness of the audit's numerical results, methodological criticisms,
  clean checks, severity assessments, and claim verdicts. Identify material errors,
  refuted allegations, overstatements, and appropriately expressed uncertainty.
- **Usefulness**: how much the audit helps a reader understand what holds and what changes
  in the paper. Credit consequential confirmed findings, informative clean checks,
  reproducible evidence, clear explanations, and actionable qualifications. Explain the
  cost of misleading allegations or distracting detail.

Base comparisons on **adjudicated evidence**, citing cross-walk items, claim verdicts,
and exact outputs where relevant. Weight scientific consequence rather than raw finding
counts, length, confidence of tone, or agreement with the majority. Compare scope and access
to materials before treating an omission as a weakness; silence alone is not an error.
Do not treat unresolved findings as refuted or an untested clean check as established.

Give a relative judgment for accuracy and another for usefulness. Name the leading run
or tied group and explain material differences among the remaining runs. Use a **tie**
whenever the evidence does not clearly distinguish runs; distinguish demonstrated similar
performance from insufficient evidence to separate them. Accuracy and usefulness may have
different leaders. Do not force an overall winner or numerical score.

State the limits: this comparison concerns these audits of this paper, not a general model
ranking. Consensus adjudication is itself fallible, and accepted converged findings are
not necessarily independently verified. Explain when these limits prevent a distinction.

## Stage 7 — Report

Write for a reader who has not read the audits. Use plain language and short, connected
paragraphs. Be concise, but give enough context to understand each claim, its evidence,
and what the audit changes. There is no fixed word cap; length should follow the number
and complexity of the main claims. Start from the template and use this order:

1. **Paper summary**: briefly explain the research question, study design, sample or
   data, main analyses, and the paper's central conclusions.
2. **Paper-wide problems**: if confirmed or partly confirmed major problems affect the
   paper as a whole, briefly explain each problem, the evidence establishing it, and
   its impact on the paper's conclusions. Distinguish demonstrated impact from uncertainty
   that prevents verification. If there are no such problems, omit this section.
   A serious problem confined to one claim belongs with that claim.
3. **Main claims**: give each main claim a descriptive subsection heading and these
   four elements, in order:
   - **Claim**: explain what the authors assert, including the population or scope.
   - **Evidence**: describe the evidence the paper offers: relevant design, measurements,
     comparisons, and results. Include deciding estimates and uncertainty when available,
     the effective sample size, and whether the design could detect the claimed effect,
     with a figure, table, or analysis reference. Do this for claims that survive too.
   - **Holds after audit**: explicitly say whether it holds, holds with qualifications,
     is weakened, is unsupported, or cannot be determined with the available evidence.
     Give the consensus strength verdict and explain its meaning for this claim.
     An untested claim is not a verified claim.
   - **If weakened**: explain the audit evidence that weakens it and the scientific
     impact: what narrower conclusion remains defensible, what no longer follows, and
     what is still unresolved. Distinguish reanalysis results from plausible concerns.
     If it is unchanged, say so briefly; do not invent a weakness.

Describe a paper-wide problem once in full, then explain its specific impact on each
affected claim. Readers must not need to decode an issue ID or consult the appendix to
understand a verdict. Brief repetition that makes a claim understandable is appropriate.

Appendix:

- **Relative audit accuracy and usefulness**: one row per included run, with separate
  accuracy and usefulness assessments and deciding evidence; then separate comparative
  judgments, including ties and their reasons, and the limits of the comparison.
- **Published critiques**: the search record and one row per critique point, with its
  assessment, the finding or check that tested it, and its disposition.
- **Minor issues**: changes to a reported number, test, or result label that do not
  change a claim verdict; give the location and effect.
- **Noticed during adjudication**: unadjudicated concerns that could change a verdict,
  clearly labelled as unconfirmed; otherwise "None."
- **Claim verdicts by audit**: per-run claim IDs and verdicts, with reasons for differences
  from the consensus.
- **Finding cross-walk**: every finding ID of every run, consensus item, status, outcome,
  classification and rationale, severity, affected claims, and whether it changes a major conclusion.
- **Adjudication notes**: evidence settling disputed or refuted findings, revised
  magnitudes or severities, and why audits differed. Include the family rationale for
  multiplicity findings, even when converged.
- **Cosmetic issues**: typos, labels, rounding or missing details that change no result
  or conclusion, with locations.
- **Adjudication analyses**, provenance, and session info.

Run IDs and finding IDs stay in the appendix. Consensus IDs may be used as optional
links to detailed evidence, but never as substitutes for explanations in the main body.

## Stage 8 — Render and verify

Render `consensus.qmd` to `consensus.html` with embedded resources. Check every number in
the prose against the rendered output. Read the main body as a standalone explanation:
does it summarize the paper, explain any paper-wide problems and their impact, and give
each main claim its evidence, audit verdict, and the evidence and impact of any weakening?
Remove unexplained shorthand and unnecessary detail without losing those elements.
Confirm the cross-walk lists every finding ID of every run in `inputs.json`, and that
multiplicity criticisms justify the family and account for the authors' correction.
Confirm every published critique point has a disposition and every main claim has a
power and precision row.
Confirm every included run has an evidence-based accuracy and usefulness assessment,
and that both relative judgments state any ties and limitations.
Give the operator the file path. Do not publish it.
Pliki w paczce (8)
  • README.md 1 KB
  • SKILL.md 19 KB
  • scripts/consensus.py 6 KB
  • scripts/consensus.sh 6 KB
  • scripts/discover.py 2 KB
  • scripts/redact.py 4 KB
  • scripts/unblind.py 1 KB
  • templates/consensus_template.qmd 7 KB

Wersje

W bazie nie ma innych skilli o nazwie audit-consensus.

Migawka bazy z 13.11.2026 · wersja paczki 987c95c3ae02 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 90audit-summary
    Podsumowuje zakończone audyty naukowe wielu prac, z liczbami przejrzystości oraz potwierdzonymi ustaleniami technicznymi i interpretacyjnymi.
  • 100audit-prep
    Przygotowuje foldery artykułów do audytów powtarzalności i trafności badań neuroobrazowych z izolowanymi przebiegami.
  • 100paper-verification
    Weryfikuje twierdzenia artykułu względem kodu i danych, sprawdza zgodność liczb, wzorów z kodem oraz poprawność cytowań.
  • 100reproducibility-audit
    Prowadzi audyt odtwarzalności publikacji: pobiera dane i kod autorów, odtwarza wyniki w R i Quarto oraz ocenia siłę dowodów.
  • 95bio-consensus-sequences
    Generuje sekwencje konsensusowe FASTA przez nakładanie wariantów z VCF na referencję (bcftools consensus) lub budowę konsensusu wirusowego i amplikonowego przez iVar.
  • 100paper-finder
    Znajduje zweryfikowane DOI prac naukowych potrzebnych badaczowi, bez wymyślania publikacji, i sprawdza je w Crossref.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.