data-cleaning
/data-cleaning
Zamienia wyniki kontroli jakości danych w kontrakt i potok, który głośno zawodzi, gdy dane przestają spełniać reguły.
- Autor
- angshenting ↗
- Repozytorium
- ds-spdd ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 10.11.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 1
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 10.11.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Zamienia wyniki kontroli jakości danych w kontrakt i potok, który głośno zawodzi, gdy dane przestają spełniać reguły.
Oryginalny opis (EN)
Turn data-quality findings (duplicates, invalid values, type drift, missingness beyond tolerance, category typos) into a validated canvas contract and a pipeline that fails loudly when the data stops meeting it. Use when EDA or the E section's quality rules fail, when a raw extract is known to be dirty, or when preprocessing needs new cleaning ops. Requires an existing valid canvas (`ds-canvas validate` exit 0); for greenfield canvases use spec-driven-ds first.
Jak działa
- Canvas surfaces
- Procedure
- Safeguards checklist (non-negotiable)
- Exit conditions
Podgląd SKILL.md
---
name: data-cleaning
description: Turn data-quality findings (duplicates, invalid values, type
drift, missingness beyond tolerance, category typos) into a validated
canvas contract and a pipeline that fails loudly when the data stops
meeting it. Use when EDA or the E section's quality rules fail, when a
raw extract is known to be dirty, or when preprocessing needs new
cleaning ops. Requires an existing valid canvas (`ds-canvas validate`
exit 0); for greenfield canvases use spec-driven-ds first.
---
# data-cleaning
Cleaning is a contract change, not a patch. Every fix is a decision
recorded in the canvas; the pipeline enforces it and fails loudly when the
data regresses.
## Canvas surfaces
- **E (entities)** — `quality_rules` under each dataset, machine-checkable
by `ds-canvas validate`:
- `unique` (fields are the uniqueness key)
- `non_negative`
- `missing_at_most` (requires `max`, a ratio between 0 and 1)
- `allowed_values` (requires non-empty `values` list)
- `range` (requires `min` and/or `max`, numbers)
Rule fields must be declared in the same dataset; a `field_groups` name
may stand in for its member fields.
- **S (structure)** — two op blocks with a strict placement law:
- `cleaning:` ops run **before the split**, must be stateless, and must
not declare `fit_on`: `deduplicate` (fields = dedupe key),
`coerce_type`, `clip` (at least one of `min`/`max`, numbers).
- `preprocessing:` ops run per split; anything fitted (imputers,
encoders, scalers, `winsorize`) lives here and must declare
`fit_on: train`.
- The validator rejects: fitted ops in `cleaning`, `fit_on` on cleaning
ops, `deduplicate` in `preprocessing` (row removal after the split
breaks the ratios), the target under any cleaning op, `clip` with
`min > max`, and any preprocessing op name it does not know (a
misspelled fitted op would otherwise skip the `fit_on: train` check).
- **O (operations)** — must show the order: clean → split → quality gate on
train → preprocess (fit on train) → train/select → evaluate once.
## Procedure
1. **Inventory findings.** Read the canvas and the latest EDA report
(`reports/eda.md`). List every quality problem with its evidence
(rule, field, observed value/rate).
2. **Decide per finding** — one of, and only one of:
- fix with a stateless op → S `cleaning` (pre-split)
- fix with a fitted op → S `preprocessing` with `fit_on: train`
- drop rows → only via `deduplicate` (pre-split, on the declared key)
- drop the field → remove from E and every op that referenced it
- document only → note it in E prose; no op, no rule
A finding with no decision is a decision left to the code — not
allowed.
3. **Record rules in E.** Add a `quality_rules` entry for every
invariant the cleaned data must hold (e.g. `non_negative` after a
`clip`, `unique` after `deduplicate`, `missing_at_most` before
imputation).
4. **Update the canvas**: E (types, notes, `quality_rules`), S
(`cleaning`/`preprocessing` ops), O (step order above). Then:
```
ds-canvas validate <canvas.md>
```
Exit condition: exit 0.
5. **Implement exactly what the canvas says** — no op exists in code that
is not declared, no declared op missing from code.
6. **Fail loudly.** The pipeline step that checks quality rules must
verify every E `quality_rule` on the train split and exit non-zero on
the first failure, writing what failed to the EDA report.
7. **Honor staleness.** E and/or S changed, so the canvas Results are
invalid: remove the Results section entirely (an empty Results section
is a validation error), run one full selection round (train → select on
validation → evaluate test once), then:
- if the round meets the declared targets, re-add Results from the new
test run and `ds-canvas validate` (exit 0);
- if it does not, **Results stay out of the canvas** — the validator
rejects achieved values below target, and that is correct. Do not
lower targets or re-tune on test to manufacture a pass. Document the
miss in the report and surface the decision point: iterate at the
model layer (new candidates/hyperparameters, selected on validation)
or renegotiate the requirement in R with the stakeholder.
8. **Update the report**: what was found, what was decided, why, and the
new Results — every claim traced to the canvas.
## Safeguards checklist (non-negotiable)
- Nothing is fitted on anything but the train split (`split_before_fit`).
- Row removal happens only before the split; ratios in A stay intact.
- The target never appears under any cleaning op (leakage).
- PII handling in Safeguards is unchanged by cleaning.
- The test split is touched exactly once per selection round.
## Exit conditions
- [ ] Every finding has a decision recorded in the canvas
- [ ] `ds-canvas validate` exits 0
- [ ] Quality gate passes on train and fails non-zero when a rule breaks
- [ ] One fresh selection round run; Results re-added if targets met, the
miss documented in the report if not
- [ ] Report updated, claims traceable
Wersje
| Repozytorium | Wynik | Licencja |
|---|---|---|
| Ta strona · angshenting/ds-spdd | 100 | MIT |
| ericrisco/rsc-harness | 100 | MIT |
| magnus919/agent-skills | 100 | MIT |
| SSameer20/data-analyst-toolkit | 100 | MIT |
| seb1n/awesome-ai-agent-skills | 100 | MIT |
| param087/agent-ml-skills | 100 | MIT |
| Amey-Thakur/AI-SKILLS | 90 | MIT |
| lisonevf/finance-skills | 90 | MIT |
Migawka bazy z 13.11.2026 · wersja paczki 16e0c4b23a1a (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100feature-engineering
Dodaje wyprowadzone cechy do projektu data science i zostawia tylko te, które poprawiają wynik walidacji. - 100ds-canvas-sync
Przenosi zmiany kodu pipeline'u z powrotem do pliku canvas.md, gdy kod odszedł od opisu po refaktoryzacjach, poprawkach lub nowych metrykach. - 91data-governance-and-quality
Ustala standardy jakości danych, reguły walidacji, słownik danych i mapowania dla platformy ubezpieczeniowej. - 87data-profiling-and-cleaning
Profiluje zbiór danych przed analizą i czyści go kodem, który da się uruchomić ponownie, z raportem jakości i dziennikiem zmian. - 90clean-recon
Audytuje istniejący kod czyszczenia danych, wskazując brakującą walidację, ciche gubienie danych i luki jakościowe. - 95invalid-data-cleaning
Przygotowuje duże dane z Excela: zależnie od liczby wierszy konwertuje je do Parquet, czyści wskazane kolumny tekstowe wyrażeniami regularnymi (na przykład zostawia chińskie znaki) i eksportuje wynik z linkiem.