rigor
/rigor
Pilnuje rzetelności wyniku badawczego: sprawdza pomiar, ten sam benchmark, pełne mianowniki i brak przecieku danych.
- Autor
- Steven51516 ↗
- Repozytorium
- sliu-skill ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 13.11.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 1
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 13.11.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Pilnuje rzetelności wyniku badawczego: sprawdza pomiar, ten sam benchmark, pełne mianowniki i brak przecieku danych.
Oryginalny opis (EN)
Rules for a trustworthy research answer, followed while you work: verify the measurement, same benchmark, numbers from files with the full denominator, no leakage, every difference recorded. Use when setting up, running or writing up a result.
Jak działa
- What each ticket needs
- The rules
- Rules of the result
Podgląd SKILL.md
--- name: rigor description: "Rules for a trustworthy research answer, followed while you work: verify the measurement, same benchmark, numbers from files with the full denominator, no leakage, every difference recorded. Use when setting up, running or writing up a result." --- # Rigor An answer counts when it can be trusted. This skill is the reference for producing one: what makes a result trustworthy, and the rules to follow while you work, not after. ## What each ticket needs | ticket | apply | |---|---| | `experiment` (compares runs, or computes numbers from our own data or results) | all of the rules below | | `task` that builds something later tickets use (a data version, an encoded store, a tokenizer, a launcher) | the measurement, numbers from files | | `research` (reading only), `grilling`, `prototype`, any other `task` (getting access, setting up an environment) | none | ## The rules 1. **Verify the measurement first.** A score depends on the input, the reference, the prediction and the scorer. Check all four before judging the model: labels present and aligned with the inputs, units and scale, the scorer on a known case, and the reference itself: ground truth, or named a proxy when a model made it (another model's predictions, a model's own confidence). A score is never normalised by the evaluated model's own outputs (its best run, its maximum). A new benchmark or metric, or a new or changed pipeline (any code on the path that produces the number), needs a positive control (a known answer it must reproduce) and a null (what an information-free input scores). Code that has already passed does not: on a repeat, check only what varies between repeats and would fail silently. Add a control when a result looks too good, sits at the floor or ceiling, or cannot be explained. A speed or cost number gets the same care: name what bounds it (GPU, data loading, I/O, the scheduler) and rule out failed, skipped or cached work and code that never ran. 2. **Same benchmark.** The runs compared are scored on the same benchmark: eval set and N, metric, scorer, decoding (the map's Contract names it; with `Benchmark: none`, the eval the ticket's pre-registration fixes, the same for every arm). A number from another benchmark is not comparable until re-scored on this one. When the benchmark itself changes, re-score the reference too. 3. **Numbers from files, with the full denominator.** Every number is read from a saved result file and written with its source: `0.712 (mean TM, test-v2 N=338, greedy; runs/arm-b/eval/scores.json)`. Failed, invalid and unscored items stay in the count; a score on the valid subset never stands in for the full one. Missing is not zero. Validity (parsed, complete, well-formed) is reported apart from accuracy. Never average different metrics or scales. 4. **No leakage.** The eval items, or near-duplicates of them, are not in the training data, and no setting was chosen on the eval set. A set you choose settings on or iterate against becomes a selection set and stops being independent evidence: final claims use a held-out set. 5. **Differences recorded.** A new kind of experiment records its full setup (data, code, config, launch). Otherwise record the reference run and every difference from it; a reference other tickets compare against has its full setup recorded (in a run card, where the project uses them). When several things changed together (data, budget, model, metric), the result belongs to all of them: credit one only if it was varied alone. ## Rules of the result - **Screen, then confirm.** A single run or a fixed cheap budget is a **screen**: its answer is `[tentative]`. A claim the map stands on is **confirmed**: before results, the pre-registration fixes the repeats (seeds or draws; 3 by default), the comparison (paired where possible), the smallest gap that matters, and how the spread is reported. It is `[established]` when the gap clears both that threshold and the spread between repeats. - **Criteria before results.** The pre-registration's success and stop rules are fixed before anything runs; amendments are dated and come before the numbers they touch. - **Classify a failure before calling it negative.** Execution, measurement, data, method or hypothesis: only the last two are negative; the rest are inconclusive, with the fix.
Wersje
W bazie nie ma innych skilli o nazwie rigor.
Migawka bazy z 13.11.2026 · wersja paczki 11cec3c2acb6 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 90evaluation-leakage-audit
Sprawdza ewaluację, benchmark albo harness oceniania pod kątem wycieku danych i wyników, które potwierdzają same siebie. - 90benchmark-research-skill
Przegląda istniejące benchmarki dla pytania badawczego i wyciąga zbiory danych, metryki oraz bazowe wyniki z artykułów. - 100briefbound-research-rigor-review
Ocenia rygor dowodowy wyniku badania AI/ML, awansu baseline, wniosku z ablacji, twierdzenia z benchmarku lub kosztownego kolejnego eksperymentu, zanim zostanie zaakceptowany, uogólniony lub opublikowany.