Skip to content
Skills Library
Data & analytics · Building & coding

data-cleaning

/data-cleaning

Turn data-quality findings (duplicates, invalid values, type drift, missingness beyond tolerance, category typos) into a validated canvas contract and a pipeline that fails loudly when the data stops meeting it. Use when EDA or the E section's quality rules fail, when a raw extract is known to be dirty, or when preprocessing needs new cleaning ops. Requires an existing valid canvas (`ds-canvas validate` exit 0); for greenfield canvases use spec-driven-ds first.

Repository
ds-spdd ↗
Licence
MIT
Last repository change
10.11.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 10.11.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Turn data-quality findings (duplicates, invalid values, type drift, missingness beyond tolerance, category typos) into a validated canvas contract and a pipeline that fails loudly when the data stops meeting it. Use when EDA or the E section's quality rules fail, when a raw extract is known to be dirty, or when preprocessing needs new cleaning ops. Requires an existing valid canvas (`ds-canvas validate` exit 0); for greenfield canvases use spec-driven-ds first.

Original author description

Turn data-quality findings (duplicates, invalid values, type drift, missingness beyond tolerance, category typos) into a validated canvas contract and a pipeline that fails loudly when the data stops meeting it. Use when EDA or the E section's quality rules fail, when a raw extract is known to be dirty, or when preprocessing needs new cleaning ops. Requires an existing valid canvas (`ds-canvas validate` exit 0); for greenfield canvases use spec-driven-ds first.

How it works

  1. Canvas surfaces
  2. Procedure
  3. Safeguards checklist (non-negotiable)
  4. Exit conditions
Original SKILL.md headings · 93 lines, 727 words
SKILL.md preview
---
name: data-cleaning
description: Turn data-quality findings (duplicates, invalid values, type
  drift, missingness beyond tolerance, category typos) into a validated
  canvas contract and a pipeline that fails loudly when the data stops
  meeting it. Use when EDA or the E section's quality rules fail, when a
  raw extract is known to be dirty, or when preprocessing needs new
  cleaning ops. Requires an existing valid canvas (`ds-canvas validate`
  exit 0); for greenfield canvases use spec-driven-ds first.
---

# data-cleaning

Cleaning is a contract change, not a patch. Every fix is a decision
recorded in the canvas; the pipeline enforces it and fails loudly when the
data regresses.

## Canvas surfaces

- **E (entities)** — `quality_rules` under each dataset, machine-checkable
  by `ds-canvas validate`:
  - `unique` (fields are the uniqueness key)
  - `non_negative`
  - `missing_at_most` (requires `max`, a ratio between 0 and 1)
  - `allowed_values` (requires non-empty `values` list)
  - `range` (requires `min` and/or `max`, numbers)
  Rule fields must be declared in the same dataset; a `field_groups` name
  may stand in for its member fields.
- **S (structure)** — two op blocks with a strict placement law:
  - `cleaning:` ops run **before the split**, must be stateless, and must
    not declare `fit_on`: `deduplicate` (fields = dedupe key),
    `coerce_type`, `clip` (at least one of `min`/`max`, numbers).
  - `preprocessing:` ops run per split; anything fitted (imputers,
    encoders, scalers, `winsorize`) lives here and must declare
    `fit_on: train`.
  - The validator rejects: fitted ops in `cleaning`, `fit_on` on cleaning
    ops, `deduplicate` in `preprocessing` (row removal after the split
    breaks the ratios), the target under any cleaning op, `clip` with
    `min > max`, and any preprocessing op name it does not know (a
    misspelled fitted op would otherwise skip the `fit_on: train` check).
- **O (operations)** — must show the order: clean → split → quality gate on
  train → preprocess (fit on train) → train/select → evaluate once.

## Procedure

1. **Inventory findings.** Read the canvas and the latest EDA report
   (`reports/eda.md`). List every quality problem with its evidence
   (rule, field, observed value/rate).
2. **Decide per finding** — one of, and only one of:
   - fix with a stateless op → S `cleaning` (pre-split)
   - fix with a fitted op → S `preprocessing` with `fit_on: train`
   - drop rows → only via `deduplicate` (pre-split, on the declared key)
   - drop the field → remove from E and every op that referenced it
   - document only → note it in E prose; no op, no rule
   A finding with no decision is a decision left to the code — not
   allowed.
3. **Record rules in E.** Add a `quality_rules` entry for every
   invariant the cleaned data must hold (e.g. `non_negative` after a
   `clip`, `unique` after `deduplicate`, `missing_at_most` before
   imputation).
4. **Update the canvas**: E (types, notes, `quality_rules`), S
   (`cleaning`/`preprocessing` ops), O (step order above). Then:
   ```
   ds-canvas validate <canvas.md>
   ```
   Exit condition: exit 0.
5. **Implement exactly what the canvas says** — no op exists in code that
   is not declared, no declared op missing from code.
6. **Fail loudly.** The pipeline step that checks quality rules must
   verify every E `quality_rule` on the train split and exit non-zero on
   the first failure, writing what failed to the EDA report.
7. **Honor staleness.** E and/or S changed, so the canvas Results are
   invalid: remove the Results section entirely (an empty Results section
   is a validation error), run one full selection round (train → select on
   validation → evaluate test once), then:
   - if the round meets the declared targets, re-add Results from the new
     test run and `ds-canvas validate` (exit 0);
   - if it does not, **Results stay out of the canvas** — the validator
     rejects achieved values below target, and that is correct. Do not
     lower targets or re-tune on test to manufacture a pass. Document the
     miss in the report and surface the decision point: iterate at the
     model layer (new candidates/hyperparameters, selected on validation)
     or renegotiate the requirement in R with the stakeholder.
8. **Update the report**: what was found, what was decided, why, and the
   new Results — every claim traced to the canvas.

## Safeguards checklist (non-negotiable)

- Nothing is fitted on anything but the train split (`split_before_fit`).
- Row removal happens only before the split; ratios in A stay intact.
- The target never appears under any cleaning op (leakage).
- PII handling in Safeguards is unchanged by cleaning.
- The test split is touched exactly once per selection round.

## Exit conditions

- [ ] Every finding has a decision recorded in the canvas
- [ ] `ds-canvas validate` exits 0
- [ ] Quality gate passes on train and fails non-zero when a rule breaks
- [ ] One fresh selection round run; Results re-added if targets met, the
      miss documented in the report if not
- [ ] Report updated, claims traceable

Versions

RepositoryScoreLicence
This page · angshenting/ds-spdd100MIT
ericrisco/rsc-harness100MIT
magnus919/agent-skills100MIT
SSameer20/data-analyst-toolkit100MIT
seb1n/awesome-ai-agent-skills100MIT
param087/agent-ml-skills100MIT
Amey-Thakur/AI-SKILLS90MIT
lisonevf/finance-skills90MIT

Database snapshot: 13.11.2026 · package version 16e0c4b23a1a (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100feature-engineering
    Add derived features (interactions, transforms, group aggregates) to an existing DS project as validated canvas contract — hypothesis first, provenance declared in E, constructed by S ops, won or lost in selection on validation, losers removed from the contract. Use when the…
  • 100ds-canvas-sync
    Reverse-syncs pipeline code changes back into ds/canvas.md (the SPDD code-to-canvas direction) in a project set up with spec-driven-ds. Use when pipeline code has drifted from the canvas after refactors, bug fixes, or new preprocessing/metrics/model code; before reporting…
  • 91data-governance-and-quality
    Data governance strategy, quality validation rules, and data dictionary management for vehicle insurance platform. Use when defining data quality standards, implementing validation rules, managing field mappings, resolving data conflicts, or establishing data governance…
  • 87data-profiling-and-cleaning
    Profiles a dataset before anyone estimates anything on it, then cleans it through code that can be rerun, producing a data quality report, a cleaning log, a sample construction table and an analysis-ready file. Enforces that raw files are never edited, that every variable used…
  • 90clean-recon
    Audit existing data cleaning code — find missing validation, silent data loss, and quality gaps. Use when asked to "audit our data cleaning", "are we losing data silently", or "find data quality gaps".
  • 95invalid-data-cleaning
    用于大规模Excel数据的预处理,通过统计总行数判断是否转换为Parquet格式以提升读写效率,并使用正则表达式清洗指定文本列(如仅保留中文字符),最后导出清洗后的文件并提供下载链接。

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.