data-cleaning
/data-cleaning
Turn data-quality findings (duplicates, invalid values, type drift, missingness beyond tolerance, category typos) into a validated canvas contract and a pipeline that fails loudly when the data stops meeting it. Use when EDA or the E section's quality rules fail, when a raw extract is known to be dirty, or when preprocessing needs new cleaning ops. Requires an existing valid canvas (`ds-canvas validate` exit 0); for greenfield canvases use spec-driven-ds first.
- Author
- angshenting ↗
- Repository
- ds-spdd ↗
- Licence
- MIT
- Last repository change
- 10.11.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 10.11.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Turn data-quality findings (duplicates, invalid values, type drift, missingness beyond tolerance, category typos) into a validated canvas contract and a pipeline that fails loudly when the data stops meeting it. Use when EDA or the E section's quality rules fail, when a raw extract is known to be dirty, or when preprocessing needs new cleaning ops. Requires an existing valid canvas (`ds-canvas validate` exit 0); for greenfield canvases use spec-driven-ds first.
Original author description
Turn data-quality findings (duplicates, invalid values, type drift, missingness beyond tolerance, category typos) into a validated canvas contract and a pipeline that fails loudly when the data stops meeting it. Use when EDA or the E section's quality rules fail, when a raw extract is known to be dirty, or when preprocessing needs new cleaning ops. Requires an existing valid canvas (`ds-canvas validate` exit 0); for greenfield canvases use spec-driven-ds first.
How it works
- Canvas surfaces
- Procedure
- Safeguards checklist (non-negotiable)
- Exit conditions
SKILL.md preview
---
name: data-cleaning
description: Turn data-quality findings (duplicates, invalid values, type
drift, missingness beyond tolerance, category typos) into a validated
canvas contract and a pipeline that fails loudly when the data stops
meeting it. Use when EDA or the E section's quality rules fail, when a
raw extract is known to be dirty, or when preprocessing needs new
cleaning ops. Requires an existing valid canvas (`ds-canvas validate`
exit 0); for greenfield canvases use spec-driven-ds first.
---
# data-cleaning
Cleaning is a contract change, not a patch. Every fix is a decision
recorded in the canvas; the pipeline enforces it and fails loudly when the
data regresses.
## Canvas surfaces
- **E (entities)** — `quality_rules` under each dataset, machine-checkable
by `ds-canvas validate`:
- `unique` (fields are the uniqueness key)
- `non_negative`
- `missing_at_most` (requires `max`, a ratio between 0 and 1)
- `allowed_values` (requires non-empty `values` list)
- `range` (requires `min` and/or `max`, numbers)
Rule fields must be declared in the same dataset; a `field_groups` name
may stand in for its member fields.
- **S (structure)** — two op blocks with a strict placement law:
- `cleaning:` ops run **before the split**, must be stateless, and must
not declare `fit_on`: `deduplicate` (fields = dedupe key),
`coerce_type`, `clip` (at least one of `min`/`max`, numbers).
- `preprocessing:` ops run per split; anything fitted (imputers,
encoders, scalers, `winsorize`) lives here and must declare
`fit_on: train`.
- The validator rejects: fitted ops in `cleaning`, `fit_on` on cleaning
ops, `deduplicate` in `preprocessing` (row removal after the split
breaks the ratios), the target under any cleaning op, `clip` with
`min > max`, and any preprocessing op name it does not know (a
misspelled fitted op would otherwise skip the `fit_on: train` check).
- **O (operations)** — must show the order: clean → split → quality gate on
train → preprocess (fit on train) → train/select → evaluate once.
## Procedure
1. **Inventory findings.** Read the canvas and the latest EDA report
(`reports/eda.md`). List every quality problem with its evidence
(rule, field, observed value/rate).
2. **Decide per finding** — one of, and only one of:
- fix with a stateless op → S `cleaning` (pre-split)
- fix with a fitted op → S `preprocessing` with `fit_on: train`
- drop rows → only via `deduplicate` (pre-split, on the declared key)
- drop the field → remove from E and every op that referenced it
- document only → note it in E prose; no op, no rule
A finding with no decision is a decision left to the code — not
allowed.
3. **Record rules in E.** Add a `quality_rules` entry for every
invariant the cleaned data must hold (e.g. `non_negative` after a
`clip`, `unique` after `deduplicate`, `missing_at_most` before
imputation).
4. **Update the canvas**: E (types, notes, `quality_rules`), S
(`cleaning`/`preprocessing` ops), O (step order above). Then:
```
ds-canvas validate <canvas.md>
```
Exit condition: exit 0.
5. **Implement exactly what the canvas says** — no op exists in code that
is not declared, no declared op missing from code.
6. **Fail loudly.** The pipeline step that checks quality rules must
verify every E `quality_rule` on the train split and exit non-zero on
the first failure, writing what failed to the EDA report.
7. **Honor staleness.** E and/or S changed, so the canvas Results are
invalid: remove the Results section entirely (an empty Results section
is a validation error), run one full selection round (train → select on
validation → evaluate test once), then:
- if the round meets the declared targets, re-add Results from the new
test run and `ds-canvas validate` (exit 0);
- if it does not, **Results stay out of the canvas** — the validator
rejects achieved values below target, and that is correct. Do not
lower targets or re-tune on test to manufacture a pass. Document the
miss in the report and surface the decision point: iterate at the
model layer (new candidates/hyperparameters, selected on validation)
or renegotiate the requirement in R with the stakeholder.
8. **Update the report**: what was found, what was decided, why, and the
new Results — every claim traced to the canvas.
## Safeguards checklist (non-negotiable)
- Nothing is fitted on anything but the train split (`split_before_fit`).
- Row removal happens only before the split; ratios in A stay intact.
- The target never appears under any cleaning op (leakage).
- PII handling in Safeguards is unchanged by cleaning.
- The test split is touched exactly once per selection round.
## Exit conditions
- [ ] Every finding has a decision recorded in the canvas
- [ ] `ds-canvas validate` exits 0
- [ ] Quality gate passes on train and fails non-zero when a rule breaks
- [ ] One fresh selection round run; Results re-added if targets met, the
miss documented in the report if not
- [ ] Report updated, claims traceable
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · angshenting/ds-spdd | 100 | MIT |
| ericrisco/rsc-harness | 100 | MIT |
| magnus919/agent-skills | 100 | MIT |
| SSameer20/data-analyst-toolkit | 100 | MIT |
| seb1n/awesome-ai-agent-skills | 100 | MIT |
| param087/agent-ml-skills | 100 | MIT |
| Amey-Thakur/AI-SKILLS | 90 | MIT |
| lisonevf/finance-skills | 90 | MIT |
Database snapshot: 13.11.2026 · package version 16e0c4b23a1a (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100feature-engineering
Add derived features (interactions, transforms, group aggregates) to an existing DS project as validated canvas contract — hypothesis first, provenance declared in E, constructed by S ops, won or lost in selection on validation, losers removed from the contract. Use when the… - 100ds-canvas-sync
Reverse-syncs pipeline code changes back into ds/canvas.md (the SPDD code-to-canvas direction) in a project set up with spec-driven-ds. Use when pipeline code has drifted from the canvas after refactors, bug fixes, or new preprocessing/metrics/model code; before reporting… - 91data-governance-and-quality
Data governance strategy, quality validation rules, and data dictionary management for vehicle insurance platform. Use when defining data quality standards, implementing validation rules, managing field mappings, resolving data conflicts, or establishing data governance… - 87data-profiling-and-cleaning
Profiles a dataset before anyone estimates anything on it, then cleans it through code that can be rerun, producing a data quality report, a cleaning log, a sample construction table and an analysis-ready file. Enforces that raw files are never edited, that every variable used… - 90clean-recon
Audit existing data cleaning code — find missing validation, silent data loss, and quality gaps. Use when asked to "audit our data cleaning", "are we losing data silently", or "find data quality gaps". - 95invalid-data-cleaning
用于大规模Excel数据的预处理,通过统计总行数判断是否转换为Parquet格式以提升读写效率,并使用正则表达式清洗指定文本列(如仅保留中文字符),最后导出清洗后的文件并提供下载链接。