Skip to content
Skills Library
Data & analytics · Analysis

exploring-data

/exploring-data

Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. Also covers near-duplicate row detection, cross-file key overlap ("can these join?"), dataset drift vs a stored baseline, and time-series profiling.

Licence
MIT
Last repository change
25.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
13
Scripts
9
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 9 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/analyze.sh, scripts/check_install.sh, scripts/install_large.sh, scripts/install_ydata.sh, scripts/profile_large.py, scripts/sketch_ops.py, scripts/summarize_insights.py, scripts/visualize_findings.py

What it does and when to use it

Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. Also covers near-duplicate row detection, cross-file key overlap ("can these join?"), dataset drift vs a stored baseline, and time-series profiling.

Triggers from the description

explore dataanalyze datasetEDAprofile datacan these join?
Original author description

Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. Also covers near-duplicate row detection, cross-file key overlap ("can these join?"), dataset drift vs a stored baseline, and time-series profiling.

How it works

  1. 0. Route by size FIRST
  2. A. Standard path (ydata-profiling)
  3. B. Large-file path (DuckDB, fixed memory)
  4. C. Sketch ops (any file size, fixed memory)
Original SKILL.md headings · 151 lines, 826 words
SKILL.md preview
---
name: exploring-data
description: Exploratory data analysis. Use when users upload .csv/.xlsx/.json/.parquet files or request "explore data", "analyze dataset", "EDA", "profile data". Small files get ydata-profiling HTML/JSON reports; large files (over 200MB or 5M rows) get fixed-memory DuckDB/sketch profiling. Also covers near-duplicate row detection, cross-file key overlap ("can these join?"), dataset drift vs a stored baseline, and time-series profiling.
metadata:
  version: 0.2.1
---

# Exploring Data

## 0. Route by size FIRST

```bash
ls -la <filepath>   # or: wc -l for row estimate
```

- **< 200MB and < ~5M rows** → ydata-profiling path (section A). Exact stats, interactive HTML.
- **Larger** → large-file path (section B). ydata-profiling loads everything into pandas and will crawl or OOM; the DuckDB/sketch path runs in fixed memory at any size.
- **Task-specific ops** (any size): duplicates, join feasibility, drift → section C.

## A. Standard path (ydata-profiling)

### 1. Check if installed (instant)
```bash
bash /mnt/skills/user/exploring-data/scripts/check_install.sh
```
Returns: `installed` or `not_installed`

### 2. Install if needed (one-time, ~19s)
```bash
if [ "$(bash /mnt/skills/user/exploring-data/scripts/check_install.sh)" = "not_installed" ]; then
    bash /mnt/skills/user/exploring-data/scripts/install_ydata.sh
fi
```

### 3. Run analysis (always generates JSON + HTML by default)
```bash
bash /mnt/skills/user/exploring-data/scripts/analyze.sh <filepath> [minimal|full] [html|json]
```

**Defaults:** minimal + html (also generates JSON)

**Output:**
- `eda_report.html` - Interactive report for user
- `eda_report.json` - Machine-readable for Claude analysis

### 4. If Claude needs to analyze (user asks "what do you think?" etc.)
```bash
python /mnt/skills/user/exploring-data/scripts/summarize_insights.py /mnt/user-data/outputs/eda_report.json
```

Claude should read the stdout markdown summary, NOT the full JSON report.

### 5. Present findings visually (don't just hand over the ydata HTML)

The ydata report is exhaustive but dense; a link to it is a weak deliverable.
Turn the JSON into a compact dashboard of the findings that matter:
```bash
python3 /mnt/skills/user/exploring-data/scripts/visualize_findings.py \
    /mnt/user-data/outputs/eda_report.json
# → /mnt/user-data/outputs/eda_findings.html
```
Emits a single self-contained HTML file (Chart.js from cdnjs, dark-mode aware):
missingness by column (tiered good/bad), the most skewed or zero-inflated
numeric distributions as small-multiple histograms, and the largest categorical
breakdowns. `--top N` caps charts per category (default 6). Also reads
`profile_large.py --json` output, so the large-file path gets the same treatment.

Present BOTH files: `eda_findings.html` for the headline read, `eda_report.html`
for the full drill-down. In a chat surface that renders inline visuals, prefer
rendering the two or three findings that actually answer the user's question as
inline charts over linking a file — a link the user has to open is the weakest
form of "showing" data.

### Modes

**Minimal (default, 5-10s):** overview, variable analysis, correlations, missing values, alerts
**Full (10-20s):** minimal + scatter matrices, sample data, character analysis

Full-mode triggers: "comprehensive analysis", "detailed EDA", "full profiling", "deep analysis". Otherwise minimal.

### Time series
If the data has a datetime index/column and the user cares about temporal behavior
(gaps, trends, seasonality, autocorrelation), pass `tsmode=True` to ProfileReport —
run the venv python directly instead of analyze.sh:
```python
ProfileReport(df, tsmode=True, sortby="<datetime_col>", title=...)
```
This adds gap detection, stationarity and seasonality checks that the default
report omits.

### Small-file drift
Comparing two versions of a dataset that BOTH fit in memory: use ydata's native
compare — `ProfileReport(df_a).compare(ProfileReport(df_b)).to_file(...)`.
For files too big to load, or comparing against a months-old file you no longer
have, use the sketch snapshot/drift ops in section C.

## B. Large-file path (DuckDB, fixed memory)

### 1. Install deps (idempotent, ~10s first time)
```bash
bash /mnt/skills/user/exploring-data/scripts/install_large.sh
```

### 2. Profile
```bash
python3 /mnt/skills/user/exploring-data/scripts/profile_large.py <file> [--json out.json]
```

Streams the file through DuckDB: per-column null%, approximate distinct counts
(HLL), min/max/mean, approximate quantiles (t-digest) for numerics, top-5
values for strings, plus quality flags (mostly-null, constant, id-like
columns). Markdown lands on stdout — read it directly, no summarize step
needed. Handles csv/tsv/parquet/json/ndjson. 1M rows profiles in seconds;
memory is flat regardless of file size.

For ad-hoc follow-up queries on the same large file, use DuckDB SQL directly
(`duckdb.connect().execute("SELECT ... FROM read_csv_auto('...')")`) rather
than loading pandas.

## C. Sketch ops (any file size, fixed memory)

All via `scripts/sketch_ops.py` (deps from install_large.sh). These answer
questions profilers don't:

### Near-duplicate rows
```bash
python3 sketch_ops.py dups <file> [--threshold 0.9] [--cols a,b,c] [--unweighted]
```
Exact duplicates counted by hash; near-duplicates via MinHash LSH over row
tokens. `--threshold` is a **weighted** Jaccard cutoff: a token occurring c
times in a row counts c times, so `new york new york` and `york new` score 0.5
rather than 1.0. Pass `--unweighted` for set semantics, where repeats are
discarded. Use `--cols` to restrict to the columns that define identity.

### Key overlap / join feasibility
```bash
python3 sketch_ops.py overlap <fileA> <fileB> --key <col> [--key-b <col>]
```
Theta sketches per key column → estimated intersection, Jaccard, and "% of A's
keys in B" both ways — answers "will this join hold?" without loading either
file.

### Drift vs stored baseline
```bash
python3 sketch_ops.py snapshot <file> --out baseline.sketch.json   # ~20KB
python3 sketch_ops.py drift <newfile> --baseline baseline.sketch.json
```
Snapshot serializes HLL (all columns) + KLL quantile sketches (numeric
columns) to a small JSON. Drift reports schema changes, >10% shifts in
distinct counts, and IQR-relative quantile movement. The snapshot is a few KB
— store it (repo, memory) and diff next month's delivery against it without
keeping the original file.

Note: snapshot/dups stream rows through Python (~1M rows in a few seconds);
profile_large is pure DuckDB and faster. For a quick look at a big file,
profile first, sketch ops only when the question calls for them.
Files in this package (13)
  • CHANGELOG.md 3 KB
  • README.md 1 KB
  • SKILL.md 7 KB
  • references/USAGE.md 2 KB
  • scripts/analyze.sh 2 KB
  • scripts/check_install.sh 1 KB
  • scripts/install_large.sh 1 KB
  • scripts/install_ydata.sh 1 KB
  • scripts/profile_large.py 4 KB
  • scripts/sketch_ops.py 8 KB
  • scripts/summarize_insights.py 12 KB
  • scripts/visualize_findings.py 12 KB
  • tests/test_sketch_ops.py 3 KB

Versions

The database has no other skills named exploring-data.

Identical content appears in 1 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version e60626973f18 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100duckdb-data-profiling
    Profile approved CSV or Parquet files with DuckDB and produce reproducible quality and join checks. Use for local exploratory analysis, schema drift, identifier preservation or source reconciliation. Does not authorize sensitive-data downloads or arbitrary SQL execution.
  • 95duckdb-en
    DuckDB CLI specialist for SQL analysis, data processing and file conversion. Use for SQL queries, CSV/Parquet/JSON analysis, database queries, or data conversion. Triggers on "duckdb", "sql", "query", "data analysis", "parquet", "convert data".
  • 100matlab-use-duckdb
    Use DuckDB from MATLAB via Database Toolbox (R2026a+) as a non-math operations engine on large tabular files (CSV/Parquet/JSON) and as a zero-config embedded database. Use when connecting to DuckDB, querying CSV, Parquet, and JSON files directly with SQL, reducing or profiling…
  • 100duckdb
    DuckDB as an in-process data transformation and ETL engine. Covers file ingestion (Parquet/CSV/JSON/Excel), SQL-based transformations, Spark replacement for small-to-medium pipelines, dbt-duckdb integration, data lake querying, and pipeline orchestration patterns. WHEN: \"DuckDB…
  • 100duckdb-in-process-analytical-queries
    Use this skill when running vectorized in-process SQL queries directly against Parquet, Arrow, and CSV files using DuckDB.
  • 100data-profiling
    Data Profiling. Explore and profile a dataset before modeling - EDA (exploratory data analysis), distributions, skew, outliers, missingness (MCAR, MAR, MNAR), duplicates, cardinality, class balance, label quality, leakage risk, bias and representativeness, drift between periods.…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.