exploring-data
/exploring-data
Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. Also covers near-duplicate row detection, cross-file key overlap ("can these join?"), dataset drift vs a stored baseline, and time-series profiling.
- Author
- oaustegard ↗
- Repository
- claude-skills ↗
- Licence
- MIT
- Last repository change
- 25.10.2026
Assessment and security
- Files in this package
- 13
- Scripts
- 9
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- 9 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: scripts/analyze.sh, scripts/check_install.sh, scripts/install_large.sh, scripts/install_ydata.sh, scripts/profile_large.py, scripts/sketch_ops.py, scripts/summarize_insights.py, scripts/visualize_findings.py
What it does and when to use it
Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. Also covers near-duplicate row detection, cross-file key overlap ("can these join?"), dataset drift vs a stored baseline, and time-series profiling.
Triggers from the description
Original author description
Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. Also covers near-duplicate row detection, cross-file key overlap ("can these join?"), dataset drift vs a stored baseline, and time-series profiling.
How it works
- 0. Route by size FIRST
- A. Standard path (ydata-profiling)
- B. Large-file path (DuckDB, fixed memory)
- C. Sketch ops (any file size, fixed memory)
SKILL.md preview
---
name: exploring-data
description: Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. Also covers near-duplicate row detection, cross-file key overlap ("can these join?"), dataset drift vs a stored baseline, and time-series profiling.
metadata:
version: 0.2.1
---
# Exploring Data
## 0. Route by size FIRST
```bash
ls -la <filepath> # or: wc -l for row estimate
```
- **< 200MB and < ~5M rows** → ydata-profiling path (section A). Exact stats, interactive HTML.
- **Larger** → large-file path (section B). ydata-profiling loads everything into pandas and will crawl or OOM; the DuckDB/sketch path runs in fixed memory at any size.
- **Task-specific ops** (any size): duplicates, join feasibility, drift → section C.
## A. Standard path (ydata-profiling)
### 1. Check if installed (instant)
```bash
bash /mnt/skills/user/exploring-data/scripts/check_install.sh
```
Returns: `installed` or `not_installed`
### 2. Install if needed (one-time, ~19s)
```bash
if [ "$(bash /mnt/skills/user/exploring-data/scripts/check_install.sh)" = "not_installed" ]; then
bash /mnt/skills/user/exploring-data/scripts/install_ydata.sh
fi
```
### 3. Run analysis (always generates JSON + HTML by default)
```bash
bash /mnt/skills/user/exploring-data/scripts/analyze.sh <filepath> [minimal|full] [html|json]
```
**Defaults:** minimal + html (also generates JSON)
**Output:**
- `eda_report.html` - Interactive report for user
- `eda_report.json` - Machine-readable for Claude analysis
### 4. If Claude needs to analyze (user asks "what do you think?" etc.)
```bash
python /mnt/skills/user/exploring-data/scripts/summarize_insights.py /mnt/user-data/outputs/eda_report.json
```
Claude should read the stdout markdown summary, NOT the full JSON report.
### 5. Present findings visually (don't just hand over the ydata HTML)
The ydata report is exhaustive but dense; a link to it is a weak deliverable.
Turn the JSON into a compact dashboard of the findings that matter:
```bash
python3 /mnt/skills/user/exploring-data/scripts/visualize_findings.py \
/mnt/user-data/outputs/eda_report.json
# → /mnt/user-data/outputs/eda_findings.html
```
Emits a single self-contained HTML file (Chart.js from cdnjs, dark-mode aware):
missingness by column (tiered good/bad), the most skewed or zero-inflated
numeric distributions as small-multiple histograms, and the largest categorical
breakdowns. `--top N` caps charts per category (default 6). Also reads
`profile_large.py --json` output, so the large-file path gets the same treatment.
Present BOTH files: `eda_findings.html` for the headline read, `eda_report.html`
for the full drill-down. In a chat surface that renders inline visuals, prefer
rendering the two or three findings that actually answer the user's question as
inline charts over linking a file — a link the user has to open is the weakest
form of "showing" data.
### Modes
**Minimal (default, 5-10s):** overview, variable analysis, correlations, missing values, alerts
**Full (10-20s):** minimal + scatter matrices, sample data, character analysis
Full-mode triggers: "comprehensive analysis", "detailed EDA", "full profiling", "deep analysis". Otherwise minimal.
### Time series
If the data has a datetime index/column and the user cares about temporal behavior
(gaps, trends, seasonality, autocorrelation), pass `tsmode=True` to ProfileReport —
run the venv python directly instead of analyze.sh:
```python
ProfileReport(df, tsmode=True, sortby="<datetime_col>", title=...)
```
This adds gap detection, stationarity and seasonality checks that the default
report omits.
### Small-file drift
Comparing two versions of a dataset that BOTH fit in memory: use ydata's native
compare — `ProfileReport(df_a).compare(ProfileReport(df_b)).to_file(...)`.
For files too big to load, or comparing against a months-old file you no longer
have, use the sketch snapshot/drift ops in section C.
## B. Large-file path (DuckDB, fixed memory)
### 1. Install deps (idempotent, ~10s first time)
```bash
bash /mnt/skills/user/exploring-data/scripts/install_large.sh
```
### 2. Profile
```bash
python3 /mnt/skills/user/exploring-data/scripts/profile_large.py <file> [--json out.json]
```
Streams the file through DuckDB: per-column null%, approximate distinct counts
(HLL), min/max/mean, approximate quantiles (t-digest) for numerics, top-5
values for strings, plus quality flags (mostly-null, constant, id-like
columns). Markdown lands on stdout — read it directly, no summarize step
needed. Handles csv/tsv/parquet/json/ndjson. 1M rows profiles in seconds;
memory is flat regardless of file size.
For ad-hoc follow-up queries on the same large file, use DuckDB SQL directly
(`duckdb.connect().execute("SELECT ... FROM read_csv_auto('...')")`) rather
than loading pandas.
## C. Sketch ops (any file size, fixed memory)
All via `scripts/sketch_ops.py` (deps from install_large.sh). These answer
questions profilers don't:
### Near-duplicate rows
```bash
python3 sketch_ops.py dups <file> [--threshold 0.9] [--cols a,b,c] [--unweighted]
```
Exact duplicates counted by hash; near-duplicates via MinHash LSH over row
tokens. `--threshold` is a **weighted** Jaccard cutoff: a token occurring c
times in a row counts c times, so `new york new york` and `york new` score 0.5
rather than 1.0. Pass `--unweighted` for set semantics, where repeats are
discarded. Use `--cols` to restrict to the columns that define identity.
### Key overlap / join feasibility
```bash
python3 sketch_ops.py overlap <fileA> <fileB> --key <col> [--key-b <col>]
```
Theta sketches per key column → estimated intersection, Jaccard, and "% of A's
keys in B" both ways — answers "will this join hold?" without loading either
file.
### Drift vs stored baseline
```bash
python3 sketch_ops.py snapshot <file> --out baseline.sketch.json # ~20KB
python3 sketch_ops.py drift <newfile> --baseline baseline.sketch.json
```
Snapshot serializes HLL (all columns) + KLL quantile sketches (numeric
columns) to a small JSON. Drift reports schema changes, >10% shifts in
distinct counts, and IQR-relative quantile movement. The snapshot is a few KB
— store it (repo, memory) and diff next month's delivery against it without
keeping the original file.
Note: snapshot/dups stream rows through Python (~1M rows in a few seconds);
profile_large is pure DuckDB and faster. For a quick look at a big file,
profile first, sketch ops only when the question calls for them.
Files in this package (13)
- CHANGELOG.md 3 KB
- README.md 1 KB
- SKILL.md 7 KB
- references/USAGE.md 2 KB
- scripts/analyze.sh 2 KB
- scripts/check_install.sh 1 KB
- scripts/install_large.sh 1 KB
- scripts/install_ydata.sh 1 KB
- scripts/profile_large.py 4 KB
- scripts/sketch_ops.py 8 KB
- scripts/summarize_insights.py 12 KB
- scripts/visualize_findings.py 12 KB
- tests/test_sketch_ops.py 3 KB
Versions
The database has no other skills named exploring-data.
Identical content appears in 1 other GitHub folders. We show one copy with the highest score.
Database snapshot: 13.11.2026 · package version e60626973f18 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100duckdb-data-profiling
Profile approved CSV or Parquet files with DuckDB and produce reproducible quality and join checks. Use for local exploratory analysis, schema drift, identifier preservation or source reconciliation. Does not authorize sensitive-data downloads or arbitrary SQL execution. - 95duckdb-en
DuckDB CLI specialist for SQL analysis, data processing and file conversion. Use for SQL queries, CSV/Parquet/JSON analysis, database queries, or data conversion. Triggers on "duckdb", "sql", "query", "data analysis", "parquet", "convert data". - 100matlab-use-duckdb
Use DuckDB from MATLAB via Database Toolbox (R2026a+) as a non-math operations engine on large tabular files (CSV/Parquet/JSON) and as a zero-config embedded database. Use when connecting to DuckDB, querying CSV, Parquet, and JSON files directly with SQL, reducing or profiling… - 100duckdb
DuckDB as an in-process data transformation and ETL engine. Covers file ingestion (Parquet/CSV/JSON/Excel), SQL-based transformations, Spark replacement for small-to-medium pipelines, dbt-duckdb integration, data lake querying, and pipeline orchestration patterns. WHEN: \"DuckDB… - 100duckdb-in-process-analytical-queries
Use this skill when running vectorized in-process SQL queries directly against Parquet, Arrow, and CSV files using DuckDB. - 100data-profiling
Data Profiling. Explore and profile a dataset before modeling - EDA (exploratory data analysis), distributions, skew, outliers, missingness (MCAR, MAR, MNAR), duplicates, cardinality, class balance, label quality, leakage risk, bias and representativeness, drift between periods.…