Przejdź do treści
Biblioteka skilli
Dane i analityka · Analiza

data-cleaning

/data-cleaning

Czyści, profiluje, waliduje, przekształca i dokumentuje nieuporządkowane dane tabelaryczne, tekstowe, JSON i relacyjne w odtwarzalnym przepływie opartym na dowodach.

Repozytorium
agent-skills ↗
Licencja
MIT
Ostatnia zmiana repozytorium
25.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
23
Skrypty
4
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 25.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co sprawdzić przed instalacją
  • 4 skrypty w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.

Skrypty: scripts/profile_dataset.py, scripts/reconcile_dataset.py, scripts/test_profile_dataset.py, scripts/test_reconcile_dataset.py

Co robi i kiedy

Czyści, profiluje, waliduje, przekształca i dokumentuje nieuporządkowane dane tabelaryczne, tekstowe, JSON i relacyjne w odtwarzalnym przepływie opartym na dowodach.

Oryginalny opis (EN)

Clean, profile, validate, reshape, and document messy tabular, text, JSON, and relational data through an evidence-first, reproducible workflow. Use when preparing data for analysis, reporting, modeling, ingestion, migration, or matching, including AI-suggested repair, entity-match review, score calibration limits, and reversible repair ledgers. Do not use for statistical modeling, dashboard design, or operating a named data platform; route those tasks to data-scientist, data-engineering, or the relevant tool skill.

Jak działa

  1. Route by task
  2. Available Scripts
  3. Default workflow
  4. Non-negotiable controls
  5. Completion gate
  6. When not to use
  7. Prerequisites
  8. Limitations
Sekcje z nagłówków SKILL.md, w oryginale · 73 linii, 975 słów
Podgląd SKILL.md
---
name: data-cleaning
description: >-
  Clean, profile, validate, reshape, and document messy tabular, text, JSON, and
  relational data through an evidence-first, reproducible workflow. Use when
  preparing data for analysis, reporting, modeling, ingestion, migration, or
  matching, including AI-suggested repair, entity-match review, score calibration
  limits, and reversible repair ledgers. Do not use for statistical modeling, dashboard design, or operating
  a named data platform; route those tasks to data-scientist, data-engineering,
  or the relevant tool skill.
license: MIT
compatibility: Works with any Agent Skills client. The bundled profiler requires Python 3.9+ and the standard library; ecosystem tools are optional.
metadata:
  domain: data-quality-and-cleaning
  source: primary-docs-plus-orientation-article
---

# Data cleaning

Treat cleaning as a controlled transformation of an observed dataset, not cosmetic editing. Preserve raw input, state the target use and grain, make every lossy decision explicit, and prove that the cleaned output satisfies a contract.

When a model proposes data transformations, load [the AI boundary workflow](references/ai-repair-review.md)
and use [the companion record](templates/ai-repair-ledger.csv).

## Route by task

| Need | Read next |
|---|---|
| End-to-end method, scope, and stopping rules | `references/methodology.md` |
| Choose a library or platform | `references/tool-selection.md` |
| Missingness, duplicates, types, ranges, categories, dates, joins | `references/operations.md` |
| Text, identifiers, Unicode, and entity resolution | `references/text-and-entity.md` |
| Schemas, contracts, validation, drift, scale | `references/validation-and-scale.md` |
| CLI, OpenRefine, monitoring, and interactive remediation | `references/cli-and-interactive-tools.md` |
| Source claims and version-sensitive caveats | `references/sources.md` |
| Plan, logs, exceptions, contracts, or reports | `templates/cleaning-plan.md`, `templates/transformation-log.jsonl`, `templates/exception-register.csv`, `templates/schema-contract.yml`, `templates/quality-report.md` |
| Lightweight profile or reconciliation | Run `python3 scripts/profile_dataset.py --help` or `python3 scripts/reconcile_dataset.py --help` |

## Available Scripts

| Script | Purpose | Invocation |
|---|---|---|
| `scripts/profile_dataset.py` | Dependency-free first-pass profiling of a CSV, TSV, or JSONL input without modifying it: missingness, cardinality, type candidates, duplicates, ranges, and value anomalies. Run it at workflow step 3 (Profile before changing) as the evidence-gathering pass before designing any cleaning decision. | `python3 scripts/profile_dataset.py data.csv --output profile.json` |
| `scripts/reconcile_dataset.py` | Reconciliation between a before and after delimited dataset: row counts, key uniqueness/overlap, and per-column sums (`--sum`), keyed by `--key`, writing a machine-readable report. Run it during Validate twice / Review to prove grain preservation and quantify exactly what a transformation changed. | `python3 scripts/reconcile_dataset.py raw.csv cleaned.csv --key id --sum amount --output reconciliation.json` |
| `scripts/test_profile_dataset.py` | Pytest suite covering the profiler's behavior on representative inputs. Run it after modifying the profiler or when auditing its output; CI discovers it automatically. | `python3 -m pytest scripts/test_profile_dataset.py` |
| `scripts/test_reconcile_dataset.py` | Pytest suite covering the reconciler's keying, summing, and reporting behavior. Run it after modifying the reconciler or when auditing its output; CI discovers it automatically. | `python3 -m pytest scripts/test_reconcile_dataset.py` |

## Default workflow

1. **Frame:** identify the decision, owner, source, privacy constraints, unit of observation, keys, expected grain, time window, and acceptance threshold. Do not silently infer a business rule from a suspicious value.
2. **Freeze evidence:** record source path/URI, retrieval time, file size/hash where feasible, encoding, delimiter, schema, row/column counts, and software versions. Keep raw data read-only and write to a new output.
3. **Profile before changing:** inspect missingness, sentinel values, duplicates, cardinality, type candidates, ranges, invalid dates, whitespace/Unicode anomalies, cross-field relationships, and drift. Use the bundled profiler for a dependency-free first pass.
4. **Design decisions:** classify each finding as preserve, standardize, repair, impute, quarantine, reject, or escalate. Record rationale, rule, affected rows, confidence, reversibility, and owner.
5. **Transform in layers:** prefer deterministic named steps: parse → canonicalize → type/coerce → validate → deduplicate → resolve entities → impute/quarantine → reshape. Keep raw, staged, rejected, and final datasets distinct.
6. **Validate twice:** run structural checks before and after transformation. Validate row/grain preservation, key uniqueness, referential integrity, allowed values, units, bounds, null policy, and expected distributions. Tests should identify failing records.
7. **Review and release:** compare before/after metrics, inspect samples of every changed class, obtain domain approval for semantic or lossy changes, publish the report and provenance, and make the run reproducible.

## Non-negotiable controls

- Never overwrite raw data or silently drop rows, columns, categories, outliers, or unmatched entities.
- Separate invalid, missing, not applicable, not collected, and withheld when the domain distinguishes them.
- Parse dates and numbers with an explicit locale, timezone, unit, and error policy. Count parse failures; do not silently turn them into nulls.
- Normalize text conservatively. Retain original and normalized values plus confidence when matching or repairing.
- Fit imputers, encoders, normalization parameters, and deduplication rules only on the permitted training/reference partition. Avoid leakage across time or evaluation boundaries.
- Treat profiling as evidence for investigation, not permission to auto-fix. An anomaly can be a real event.
- Use quarantine for records that cannot be repaired safely. “Clean” means accepted by a stated contract, not “no rows remain.”

## Completion gate

A cleaning task is complete only when the output, transformation/decision log, validation evidence, provenance, and unresolved issues exist; raw data remains intact; acceptance checks pass; and a reviewer can reproduce or audit the result. If semantic ambiguity remains, stop at quarantine or escalation rather than inventing a value.

## When not to use

Do not use this skill for inferential statistics or model selection, which belong to `data-scientist`; for ETL orchestration, storage, or production data-quality operations, route to `data-engineering`; or for operating a named validation or database platform, route to that tool's skill. This skill supplies cleaning judgment and artifacts those workflows consume.

## Prerequisites

- Python 3.9+ with the standard library only for both bundled scripts (per `compatibility`); ecosystem tools (OpenRefine, pandas-backed tooling) are optional accelerators covered in `references/cli-and-interactive-tools.md`.
- A raw input you can keep read-only plus write access to a separate output location — every script reads without modifying its input.
- The templates above when the task warrants formal artifacts: a cleaning plan, transformation log, exception register, schema contract, or quality report.
- `pytest` only when running the bundled test suites.

## Limitations

- The bundled profiler and reconciler are first-pass evidence tools: they surface anomalies and quantify deltas but do not decide preserve/repair/impute/quarantine — those classifications stay with the workflow's decision step.
- Both scripts handle delimited text and JSONL; binary formats, relational databases, and nested document stores need other tooling.
- Profiling output is evidence for investigation, never permission to auto-fix; an anomaly can be a real event.
- A passing reconciliation proves structural preservation on the checked keys and sums only — semantic correctness of values still requires the review and release gate.
Pliki w paczce (23)
  • README.md 2 KB
  • SKILL.md 8 KB
  • evals/evals.json 7 KB
  • evals/trigger-cases.json 1 KB
  • references/ai-repair-review.md 3 KB
  • references/cli-and-interactive-tools.md 1 KB
  • references/methodology.md 2 KB
  • references/operations.md 2 KB
  • references/sources.md 3 KB
  • references/text-and-entity.md 2 KB
  • references/tool-selection.md 1 KB
  • references/validation-and-scale.md 1 KB
  • scripts/profile_dataset.py 5 KB
  • scripts/reconcile_dataset.py 4 KB
  • scripts/test_profile_dataset.py 2 KB
  • scripts/test_reconcile_dataset.py 2 KB
  • templates/ai-repair-ledger.csv 1 KB
  • templates/cleaning-plan.md 1 KB
  • templates/decision-log.csv 1 KB
  • templates/exception-register.csv 1 KB
  • templates/quality-report.md 1 KB
  • templates/schema-contract.yml 1 KB
  • templates/transformation-log.jsonl 1 KB

Wersje

RepozytoriumWynikLicencja
Ta strona · magnus919/agent-skills100MIT
ericrisco/rsc-harness100MIT
SSameer20/data-analyst-toolkit100MIT
angshenting/ds-spdd100MIT
seb1n/awesome-ai-agent-skills100MIT
param087/agent-ml-skills100MIT
Amey-Thakur/AI-SKILLS90MIT
lisonevf/finance-skills90MIT

Migawka bazy z 13.11.2026 · wersja paczki 824b5b9d9eff (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100data-scientist
    Prowadzi analizę danych, modelowanie predykcyjne i statystyczne oraz wyciąga z danych wnioski biznesowe.
  • 90data-file-ops
    Sprawdza, czyści, przekształca i porównuje lokalne pliki JSON, CSV, TSV i tekstowe, dając powtarzalne wyniki.
  • 90data-scrubbing
    Czyści i standaryzuje zbiory tabelaryczne: braki, duplikaty, wartości odstające, łączenie rekordów i kontrole przed raportem.
  • 90data-scientist
    Bada, czyści i modeluje dane w CSV, JSON, Parquet i SQL, pisząc powtarzalne skrypty oraz uczciwe statystyki.
  • 100data-architect
    Ocenia, projektuje i rozwija architekturę danych: platformy, produkty danych, data mesh, przepływy zdarzeń, governance, modelowanie i decyzje migracyjne.
  • 90sweet
    Pracuje z danymi tabelarycznymi, np. CSV, Parquet, JSON czy Excel, przez narzędzie Sweet: czyści, profiluje, przekształca i porównuje.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.