Przejdź do treści
Biblioteka skilli
Dane i analityka · Analiza

controls-samples-diagnostics

/controls-samples-diagnostics

Sprawdza kontrole, próby i mianowniki definiujące estymand w analizie badawczej, w tym zmienne pośredniczące, porównania modeli i kowariaty.

Licencja
MIT
Ostatnia zmiana repozytorium
29.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
1
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 29.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Sprawdza kontrole, próby i mianowniki definiujące estymand w analizie badawczej, w tym zmienne pośredniczące, porównania modeli i kowariaty.

Oryginalny opis (EN)

Diagnostics for controls, samples, and denominators that define the estimand: mediator controls, control horse races, same-source covariates, covariate-by-period under fixed effects, treated shares from the regression sample, and denominator transportability. Invoke from `code-and-data` whenever a horse race, mediator, attribution claim, subset restriction, or same-source covariate touches a target result.

Jak działa

  1. Core checks
  2. Conditional checks
  3. Report back
Sekcje z nagłówków SKILL.md, w oryginale · 155 linii, 1 949 słów
Podgląd SKILL.md
---
name: controls-samples-diagnostics
description: >-
  Diagnostics for controls, samples, and denominators that define the
  estimand: mediator controls, control horse races, same-source covariates,
  covariate-by-period under fixed effects, treated shares from the
  regression sample, and denominator transportability. Invoke from
  `code-and-data` whenever a horse race, mediator, attribution claim,
  subset restriction, or same-source covariate touches a target result.
---

# Controls, samples, and denominators diagnostics

Establish what each control, sample restriction, and denominator does to
the estimand before reading any coefficient movement. Controls are treated
as free insurance, but a control can redefine the question *or inject bias
outright*: control for the mechanism and the effect "disappears"; control
for a variable built from the same source as the regressor and the two
fight over shared measurement error; restrict the sample and the claim
quietly becomes a claim about the subset; or add a control that is itself a
collider or a near-instrument and *manufacture* the bias it appears to
prevent (Cinelli, Forney & Pearl 2024). A coefficient drop can mean
confounding exposed — or the estimand changed mid-argument, or a bad
control was added; the checks below tell them apart.

Trigger: a control horse race, a mediator control, an aggregate attribution
claim, a subset restriction, covariates built from the same source as
the regressor, **a control the authors added to demonstrate robustness (a
kitchen-sink covariate set), a pre-treatment control claimed as safe, a
control or sample plausibly affected by the treatment or outcome,** or
**any discretionary specification choice not forced by
theory — a single-unit/small-group dummy, an ad-hoc interaction, a bespoke
sample carve-out, a hand-picked control, an unruled tuning constant — in
any target spec**. `NOT_APPLICABLE` when none of these touches a target result.

Method crib (interim — WP-audited):
`../../references/methods/goldsmithpinkham_hull_kolesar_2024_contamination.md` —
contamination bias under multiple treatments + controls; published AER
version pending.

## Core checks

- **Coefficient movement versus standard-error inflation.** Print before/after estimates, intervals, SEs, sample, and identifying variation; report changes in units (percentage changes only with a nonzero, meaningful denominator). A stable coefficient with a wider interval indicates reduced precision; a coefficient moving toward zero indicates attenuation in that specification. Neither pattern establishes whether the added control is valid. Check temporal and causal ordering, collider/mediator risks, changed estimands, and loss of identifying variation. VIF diagnoses linear dependence among regressors, not causal validity: low VIF cannot prove confounding or a zero causal effect. Interpret the result against the paper's actual claim and a substantively justified specification. Two naming requirements make the check answerable: for every contested control, name which catalog model the paper's causal structure implies (the Cinelli–Forney–Pearl crib's model number) — if the paper has no causal model, that absence is itself the finding — and name the reference specification against which movement is measured, with the reason; movement relative to an unnamed baseline supports opposite stories. When the paper's defense is coefficient stability across control sets, route to the OVB-sensitivity record (Cinelli–Hazlett robustness value benchmarked to named covariates, or Oster δ/Rmax) — "the coefficient was stable" alone is not a sensitivity analysis. And test whether the stability check could have detected anything: when the added controls are noisy proxies for the true confounder, attenuation makes coefficient invariance expected even under confounding — the more powerful test is the balancing regression (proxy on the left, treatment on the right), per `../../references/methods/pei_pischke_schwandt_2019_left_right.md`. A paper resting identification on RHS stability with poorly measured proxies has weak evidence, whatever the stability.

- **Mediator controls identified and removed.** A control that sits on the
  causal path from treatment to outcome answers a different question;
  re-estimate without it and report both numbers with the distinction
  stated. Before prescribing removal, ask whether the mediator is itself
  confounded (a treatment→mediator confounder or a mediator↔outcome
  confounder): then both the with- and without-control estimates are biased
  and the finding is "unidentified without stronger assumptions," not "the
  mediator inflated it."
- **Bad controls that inject bias — test by removal, watch the direction.**
  Beyond mediators, a control can *create* the bias it appears to remove.
  Re-estimate without each suspect control and report not just whether
  significance moves but *which way the point estimate moves*: movement toward the null or other anchors is sensitivity evidence, not proof the control was harmful. Establish control validity from the causal structure and provenance; the null and other estimates are not known truth.
  - **M-bias (pre-treatment collider).** Do not exempt a covariate because it
    is "pre-treatment." If it is a common effect of unobserved causes of both
    treatment and outcome, conditioning on it opens a back-door; test removal and interpret movement conditional on the proposed causal structure.
  - **Bias amplification (near-instrument).** Flag any covariate that predicts
    the treatment strongly but has no independent path to the outcome. Under
    residual confounding it amplifies existing bias; remove it and report the
    move.
  - **Selection / case-control (collider or descendant of the outcome).**
    Treat a post-outcome control, or a sample/frame selected on a variable
    the outcome affects, as a collider until shown otherwise — it induces a
    spurious association that no coefficient stability can dispel.
    Exception (crib Model 18 only — Z caused by Y alone): under a true null,
    the adjusted test of zero remains valid — the bias afflicts the point
    estimate and any non-null claim, not the null test. Not when the
    treatment also causes Z (Model 17, X → Z ← Y): there the null test is
    biased too.

  The 18-model classification behind these rows — including the
  counterexamples to "pre-treatment = safe" (M-bias, bias amplification)
  and "post-treatment = bad" (neutral Models 14–15) — is the verified crib
  `../../references/methods/cinelli_forney_pearl_2024_bad_controls.md`.
- **OLS estimand under effect heterogeneity — the coefficient is not the
  ATE.** When an OLS coefficient with controls is read as a causal *average*
  effect, check what OLS actually recovers once effects vary across units.
  Even with a correctly specified linear model and valid controls, OLS
  returns a *conditional-variance-weighted* average of the underlying
  effects (Angrist 1998; Aronow & Samii 2016 — weights (D − E[D|X])²,
  non-negative), and with a binary treatment its weights on the ATT and
  ATU lie in [0, 1] but can sit far from the population shares
  (Słoczyński 2022) — so it need not equal the ATE the prose claims. The
  weights go **negative** when E[D|X] is nonlinear in the included
  controls, or with multiple treatments (Goldsmith-Pinkham, Hull & Kolesár
  2024, below); then the coefficient can fall outside the range of every
  unit-level effect.
  This is an estimand problem, not a functional-form problem: it survives
  passing every `binstest`/`nprobust` linearity check.
  - **Action.** Recover the target average directly and report the gap:
    reweight to the covariate/treatment distribution (regression-adjusted
    `margins`, an explicitly weighted estimator, or an AIPW/doubly-robust
    estimator, which also survives if *either* the outcome or the
    propensity model is right) and compare it to the unweighted OLS
    coefficient. A coefficient that diverges materially from the reweighted
    ATE — or that rests on strata carrying negative weights — is answering a
    different question than the paper states; record it as an
    estimand-redefinition finding for `claims`.
  - For a two-way fixed-effects / staggered-DiD coefficient the same
    pathology is the negative-weighting (forbidden-comparison) problem —
    hand to `did-event-study-diagnostics` rather than re-deriving it here.
  - **Multi-treatment and interacted specifications:** contamination bias
    (Goldsmith-Pinkham, Hull & Kolesár 2024) — each coefficient picks up the
    other treatments' heterogeneous effects even with correct controls. In
    a horse race between treatments in one regression, re-estimate each arm
    in a separately-interacted specification that blocks cross-treatment
    leakage and compare.
- **Same-source covariates.** Check controls built from the same underlying
  source as the regressor for near-duplication and shared no-data
  artifacts; a horse race between two versions of the same measurement is
  not a horse race. Report the artifact: the (residualised) correlation
  between the two versions and a construction-provenance diff —
  near-duplication is shown, not asserted. The `code-and-data` data audit's hidden-input
  check is the same hazard upstream; cross-reference it.
- **Covariate-by-period under fixed effects** — where the claim requires a
  time-invariant covariate's effect to vary: entered flat under unit FE it
  is absorbed, and its "insignificance" is mechanical.
- **Treated shares computed from the regression sample**, not taken from
  the text — the paper's descriptive shares and the estimation sample's
  shares routinely differ, and magnitude calculations built on the wrong
  one inherit the error.
- **Denominator transportability** — for aggregate attribution claims:
  a share of a net quantity is defined only at the level whose net the claim
  names. Where unit-level signs differ, the share can exceed 100% and hides
  offsetting movements: report the per-unit sign distribution and the gross
  positive and negative contributions beside it, and read the share as a
  statement about the net, never about the units. "Transportable" means the share describes the same denominator
  population the claim is about; when composition differs, report gross
  positive and negative contributions separately rather than a share of a
  net quantity.

## Conditional checks

- **Author-included discretionary specification choices — interrogate the choice, do not reproduce it.** Any term or tuning the authors added that theory did not force — a single-unit/small-group dummy, an ad-hoc interaction, a bespoke sample carve-out, a hand-picked control, a tuning constant chosen without a stated rule — is a researcher degree of freedom, not standard furniture. Treat each as an object of suspicion and establish:
  - **What it is for.** State the mechanism it nominally serves, and whether the paper justifies it anywhere in prose; a stated justification is weighed here, not an exemption from the checks below. "Disclosed in a table note" is not a justification.
  - **Selection on the outcome.** Check whether the choice tracks *how the data behave* — how badly a unit fits, which window is significant — rather than a principle stated in advance. The general tell: an equally-defensible alternative choice was available and would not have helped the result.
  - **Symmetric application.** Is the same treatment applied to comparable cases on the other side, or only where it rescues the result? Swap the choice onto an equally-qualified alternative — move the dummy, shift the window, drop or re-center the interaction — and report whether significance moves. If inference depends on the choice, report specification sensitivity. Infer outcome-driven selection only with documentary or code evidence of the selection process; crossing a significance threshold alone does not establish it.
  - **Every column, especially the weakest.** Re-run without the choice, or under the alternative, in every spec it enters (OLS/GAM, IV/2SLS, mechanism, placebo) — not only the headline. Report where the choice matters, even if it is inert in the headline.
  - **Single-unit dummy.** In an otherwise unrestricted OLS regression, a dummy for one observation removes that observation's contribution to the slope normal equations. Examine both residual size and leverage, the stated reason, and comparable units; low leverage does not prove the dummy serves no purpose. Re-run the relevant specifications without it and report estimate and uncertainty changes. Check IV effects separately; do not transfer the OLS equivalence mechanically to every estimator or attribute motive from coefficient movement.

- **Saturated group-by-period fixed effects or an explicit sample
  restriction** in place of single-group dummies — when *you* need to ask
  whether one group drives the result, answer it as a design choice, not a
  dummy coefficient.
- **Efficiency ordering of valid controls** — when the authors lean on a
  precision claim ("adding controls tightens the estimate"). Among controls
  that are all valid (block back-doors, touch no causal path), those closer
  to the *outcome* improve precision while those closer to the *treatment*
  hurt it and risk amplification. Check whether their precision-buying
  control set is outcome-side (legitimate) or treatment-side (fragile).

## Report back

Lead with what each check found and what it costs the claim, not a row for
every check. Still account for coverage: say what you ran, what was
`NOT_APPLICABLE` and why, and what was blocked (exact file, command, error). Record
estimand-redefinition findings in the module report's `## Leads for later stages` (folded into the `code-and-data` report)
for `claims` explicitly: "controlling
for X changes the question" is a story-level finding, and the supportable
claim must be written for the question the surviving specification actually
answers.

Wersje

W bazie nie ma innych skilli o nazwie controls-samples-diagnostics.

Migawka bazy z 13.11.2026 · wersja paczki 8f119d23ef37 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100rdd-diagnostics
    Diagnozuje projekty regresji nieciągłej i kinkowej: gęstość, wrażliwość na szerokość pasma, testy manipulacji i ciągłość kowariat.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.