data-cleaning
/data-cleaning
Clean and prepare messy data (missing values, outliers, types, duplicates) with decisions that preserve signal and avoid leakage. Use when raw data needs to be made analysis-ready without corrupting it.
- Author
- Amey-Thakur ↗
- Repository
- AI-SKILLS ↗
- Licence
- MIT
- Last repository change
- 05.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 05.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Clean and prepare messy data (missing values, outliers, types, duplicates) with decisions that preserve signal and avoid leakage. Use when raw data needs to be made analysis-ready without corrupting it.
Original author description
Clean and prepare messy data (missing values, outliers, types, duplicates) with decisions that preserve signal and avoid leakage. Use when raw data needs to be made analysis-ready without corrupting it.
How it works
- Method
- Boundaries
SKILL.md preview
--- name: data-cleaning description: Clean and prepare messy data (missing values, outliers, types, duplicates) with decisions that preserve signal and avoid leakage. Use when raw data needs to be made analysis-ready without corrupting it. --- # Data cleaning Real data is dirty: missing, duplicated, mistyped, inconsistent. Cleaning is not mechanical; every choice (drop this, impute that, cap the outlier) is a decision that can preserve or destroy signal, and some cleaning steps leak the future into the past. Clean deliberately, documenting why. ## Method 1. **Understand before you clean.** Do EDA first (see exploratory-data-analysis) so you know what "dirty" means here and why: cleaning blind (drop all rows with any null, cap everything at the 99th percentile) destroys signal you did not know was there. 2. **Handle missing values by their meaning.** Missing-at-random can be imputed (median for skewed numerics, a constant plus a "was-missing" indicator flag often beats fancy imputation); missing-with-meaning (no subscription = null plan) is its own category, not a value to fill. Dropping rows is fine when few and random, dangerous when missingness correlates with the target (see the missingness discussion in feature-engineering-tabular). 3. **Treat outliers as errors or extremes, deliberately.** Confirmed data errors (impossible values, typos) get fixed or removed; genuine extremes get kept, capped, or transformed depending on the model's sensitivity (tree models tolerate outliers; linear and distance-based do not). Never silently delete extremes that are real data (see exploratory-data-analysis step 3). 4. **Fix types and standardize.** Parse dates as dates, numbers as numbers (strip currency symbols, thousands separators), and standardize categorical values (unify "USA"/"US"/"United States", trim whitespace, normalize case). Inconsistent categories fragment signal across duplicate levels. 5. **Deduplicate carefully.** Find true duplicates (same entity recorded twice) versus legitimate repeats (same value, different events). Dedupe on the keys that define uniqueness, not blindly on all columns, and know which record to keep (see idempotency thinking in data-pipeline-design). 6. **Fit cleaning on train only, apply to all.** Imputation values, scaling parameters, and category mappings are learned from the training data and applied to validation/test, never fit on the full dataset. Fitting a median-imputer on all data before splitting leaks test information into training (see train-test-discipline, feature-engineering-tabular). ## Boundaries - Cleaning transforms data; log what you changed and why, because the cleaning decisions are part of the analysis and affect the result. "We dropped the outliers" is a choice a reviewer must be able to see and question. - Over-cleaning (aggressive outlier removal, heavy smoothing) can erase the very signal or rare-but-important cases you care about (fraud, failures). - Cleaning does not fix a biased or unrepresentative sample; a spotless dataset drawn wrong is still wrong (see the sampling caution in statistical-inference, correlation-causation).
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · Amey-Thakur/AI-SKILLS | 90 | MIT |
| ericrisco/rsc-harness | 100 | MIT |
| magnus919/agent-skills | 100 | MIT |
| SSameer20/data-analyst-toolkit | 100 | MIT |
| angshenting/ds-spdd | 100 | MIT |
| seb1n/awesome-ai-agent-skills | 100 | MIT |
| param087/agent-ml-skills | 100 | MIT |
| lisonevf/finance-skills | 90 | MIT |
Database snapshot: 13.11.2026 · package version ca225c74ccb7 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90clean-transform
Design a data cleaning and transformation pipeline — missing values, outliers, and deduplication. Use when asked to "clean this dataset", "handle missing values", or "deduplicate this data". - 100clean-data
Use when a clinical CSV/Excel dataset needs profiling and cleaning before analysis (missing values, outliers, duplicates, type mismatches). Profiles, flags and generates cleaning code in three stages, each gated on the researcher's approval. Never auto-cleans. - 100data-cleaning-standards
Clean and standardize vehicle insurance CSV/Excel data. Use when handling missing values, fixing data formats, removing duplicates, or standardizing fields. Mentions "clean data", "handle nulls", "standardize", "duplicates", or "normalize". - 90data-scrubbing
Guides cleaning and standardizing tabular datasets before analysis, modeling, or reporting—profiling, quality rules, missing values, duplicates, outliers, type coercion, encoding fixes, record linkage, deduplication, high-level PII handling (not legal advice),… - 100dataset-quality-audit
Run comprehensive quality checks on tabular data (CSV/Excel/TSV/JSON), detecting missing values, duplicates, outliers, format issues, and type inconsistencies to produce an overall score, grade, and actionable suggestions. Triggered when users ask to check data quality, find… - 100data-validator
Validate data quality in CSV, JSON, and database exports by checking for missing values, type mismatches, duplicates, outliers, and schema violations. Use when building ETL pipelines, auditing data imports, checking data freshness, or ensuring data contracts between teams.…