Skip to content
Skills Library
Data & analytics · Analysis

data-cleaning

/data-cleaning

Clean and prepare messy data (missing values, outliers, types, duplicates) with decisions that preserve signal and avoid leakage. Use when raw data needs to be made analysis-ready without corrupting it.

Repository
AI-SKILLS ↗
Licence
MIT
Last repository change
05.09.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 05.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Clean and prepare messy data (missing values, outliers, types, duplicates) with decisions that preserve signal and avoid leakage. Use when raw data needs to be made analysis-ready without corrupting it.

Original author description

Clean and prepare messy data (missing values, outliers, types, duplicates) with decisions that preserve signal and avoid leakage. Use when raw data needs to be made analysis-ready without corrupting it.

How it works

  1. Method
  2. Boundaries
Original SKILL.md headings · 54 lines, 412 words
SKILL.md preview
---
name: data-cleaning
description: Clean and prepare messy data (missing values, outliers, types, duplicates) with decisions that preserve signal and avoid leakage. Use when raw data needs to be made analysis-ready without corrupting it.
---

# Data cleaning

Real data is dirty: missing, duplicated, mistyped, inconsistent. Cleaning
is not mechanical; every choice (drop this, impute that, cap the outlier)
is a decision that can preserve or destroy signal, and some cleaning steps
leak the future into the past. Clean deliberately, documenting why.

## Method

1. **Understand before you clean.** Do EDA first (see
   exploratory-data-analysis) so you know what "dirty" means here and why:
   cleaning blind (drop all rows with any null, cap everything at the 99th
   percentile) destroys signal you did not know was there.
2. **Handle missing values by their meaning.** Missing-at-random can be
   imputed (median for skewed numerics, a constant plus a "was-missing"
   indicator flag often beats fancy imputation); missing-with-meaning (no
   subscription = null plan) is its own category, not a value to fill.
   Dropping rows is fine when few and random, dangerous when missingness
   correlates with the target (see the missingness discussion in
   feature-engineering-tabular).
3. **Treat outliers as errors or extremes, deliberately.** Confirmed data
   errors (impossible values, typos) get fixed or removed; genuine extremes
   get kept, capped, or transformed depending on the model's sensitivity
   (tree models tolerate outliers; linear and distance-based do not).
   Never silently delete extremes that are real data (see
   exploratory-data-analysis step 3).
4. **Fix types and standardize.** Parse dates as dates, numbers as numbers
   (strip currency symbols, thousands separators), and standardize
   categorical values (unify "USA"/"US"/"United States", trim whitespace,
   normalize case). Inconsistent categories fragment signal across
   duplicate levels.
5. **Deduplicate carefully.** Find true duplicates (same entity recorded
   twice) versus legitimate repeats (same value, different events). Dedupe
   on the keys that define uniqueness, not blindly on all columns, and know
   which record to keep (see idempotency thinking in data-pipeline-design).
6. **Fit cleaning on train only, apply to all.** Imputation values, scaling
   parameters, and category mappings are learned from the training data
   and applied to validation/test, never fit on the full dataset. Fitting a
   median-imputer on all data before splitting leaks test information into
   training (see train-test-discipline, feature-engineering-tabular).

## Boundaries

- Cleaning transforms data; log what you changed and why, because the
  cleaning decisions are part of the analysis and affect the result. "We
  dropped the outliers" is a choice a reviewer must be able to see and
  question.
- Over-cleaning (aggressive outlier removal, heavy smoothing) can erase the
  very signal or rare-but-important cases you care about (fraud, failures).
- Cleaning does not fix a biased or unrepresentative sample; a spotless
  dataset drawn wrong is still wrong (see the sampling caution in
  statistical-inference, correlation-causation).

Versions

RepositoryScoreLicence
This page · Amey-Thakur/AI-SKILLS90MIT
ericrisco/rsc-harness100MIT
magnus919/agent-skills100MIT
SSameer20/data-analyst-toolkit100MIT
angshenting/ds-spdd100MIT
seb1n/awesome-ai-agent-skills100MIT
param087/agent-ml-skills100MIT
lisonevf/finance-skills90MIT

Database snapshot: 13.11.2026 · package version ca225c74ccb7 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 90clean-transform
    Design a data cleaning and transformation pipeline — missing values, outliers, and deduplication. Use when asked to "clean this dataset", "handle missing values", or "deduplicate this data".
  • 100clean-data
    Use when a clinical CSV/Excel dataset needs profiling and cleaning before analysis (missing values, outliers, duplicates, type mismatches). Profiles, flags and generates cleaning code in three stages, each gated on the researcher's approval. Never auto-cleans.
  • 100data-cleaning-standards
    Clean and standardize vehicle insurance CSV/Excel data. Use when handling missing values, fixing data formats, removing duplicates, or standardizing fields. Mentions "clean data", "handle nulls", "standardize", "duplicates", or "normalize".
  • 90data-scrubbing
    Guides cleaning and standardizing tabular datasets before analysis, modeling, or reporting—profiling, quality rules, missing values, duplicates, outliers, type coercion, encoding fixes, record linkage, deduplication, high-level PII handling (not legal advice),…
  • 100dataset-quality-audit
    Run comprehensive quality checks on tabular data (CSV/Excel/TSV/JSON), detecting missing values, duplicates, outliers, format issues, and type inconsistencies to produce an overall score, grade, and actionable suggestions. Triggered when users ask to check data quality, find…
  • 100data-validator
    Validate data quality in CSV, JSON, and database exports by checking for missing values, type mismatches, duplicates, outliers, and schema violations. Use when building ETL pipelines, auditing data imports, checking data freshness, or ensuring data contracts between teams.…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.