Skip to content
Skills Library
AI & agents · Analysis

dataset-curation

/dataset-curation

Use when the user wants to analyze dataset bias, create stratified samples, evaluate fairness, or plan dataset collection. Triggers on phrases like "dataset bias", "stratified sample", "class imbalance", "data distribution", "fairness analysis", or "ethical review".

Repository
phd-skills ↗
Licence
MIT
Last repository change
11.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 11.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Use when the user wants to analyze dataset bias, create stratified samples, evaluate fairness, or plan dataset collection. Triggers on phrases like "dataset bias", "stratified sample", "class imbalance", "data distribution", "fairness analysis", or "ethical review".

Triggers from the description

dataset biasstratified sampleclass imbalancedata distributionfairness analysisethical review
Original author description

Use when the user wants to analyze dataset bias, create stratified samples, evaluate fairness, or plan dataset collection. Triggers on phrases like "dataset bias", "stratified sample", "class imbalance", "data distribution", "fairness analysis", or "ethical review".

How it works

  1. Step 1: Distribution Analysis
  2. Step 2: Bias Assessment
  3. Step 3: Stratified Sampling
  4. Step 4: Quality Assessment
  5. Step 5: Expansion Recommendations
  6. Step 6: Ethical Review Checklist
  7. Output Format
Original SKILL.md headings · 99 lines, 528 words
SKILL.md preview
---
name: dataset-curation
description: >
  Use when the user wants to analyze dataset bias, create stratified
  samples, evaluate fairness, or plan dataset collection. Triggers on
  phrases like "dataset bias", "stratified sample", "class imbalance",
  "data distribution", "fairness analysis", or "ethical review".
---

# Dataset Curation Methodology

You are helping a researcher curate, analyze, or expand a dataset with attention to bias, fairness, and quality.

## Step 1: Distribution Analysis

Before any curation action, understand the current state:

### Per-Class Distribution
- Count instances per class/label/tag
- Compute imbalance ratio (max_count / min_count)
- Identify severely underrepresented classes (< 5% of max class)
- Visualize: bar chart of class frequencies sorted by count

### Co-occurrence Analysis
- Build co-occurrence matrix: which labels appear together
- Identify spurious correlations (e.g., "violence" always co-occurs with "male")
- Check for label leakage between splits

### Metadata Distribution
- Source diversity: how many sources/movies/documents contribute
- Temporal distribution: are all time periods represented?
- Content diversity: genre, style, domain coverage

## Step 2: Bias Assessment

For each identified imbalance or correlation:

1. **Is it real-world reflective?** Some imbalances reflect genuine phenomena
2. **Is it harmful?** Would a model trained on this data make unfair predictions?
3. **Is it fixable?** Can we collect more data, resample, or reweight?

### Fairness Dimensions
Check for bias along relevant protected attributes:
- Gender representation (if applicable)
- Racial/ethnic representation (if applicable)
- Age distribution (if applicable)
- Geographic/cultural diversity (if applicable)

### Bias Metrics
- Demographic parity: equal positive rates across groups
- Equalized odds: equal TPR and FPR across groups
- Representation ratio: group proportion in data vs population

## Step 3: Stratified Sampling

When creating splits (train/val/test):

1. **Primary stratification**: by label/class distribution
2. **Secondary stratification**: by source (prevent source leakage across splits)
3. **Validation**:
   - Chi-squared test for label distribution similarity across splits
   - No source overlap between splits
   - Rare classes have minimum representation in each split

Split ratios depend on dataset size:
- Large (>50k): 80/10/10 or 90/5/5
- Medium (5k-50k): 70/15/15 or 80/10/10
- Small (<5k): k-fold cross-validation preferred

## Step 4: Quality Assessment

For labeled datasets, assess annotation quality:

- **Inter-annotator agreement**: Cohen's kappa, Fleiss' kappa, or Krippendorff's alpha
- **Label noise estimation**: sample and manually verify N labels
- **Edge cases**: identify ambiguous examples that annotators might disagree on
- **Consistency checks**: automated rules for label validity

## Step 5: Expansion Recommendations

If the dataset needs more data:

1. **Priority classes**: which classes benefit most from more data
2. **Source suggestions**: where to find more data for underrepresented classes
3. **Collection strategy**: active learning, targeted scraping, synthetic augmentation
4. **Cost estimation**: time and resources for each approach

## Step 6: Ethical Review Checklist

Before using or publishing any dataset:

- [ ] Content sensitivity: does the data contain sensitive material?
- [ ] Consent: was data collected with appropriate consent?
- [ ] Privacy: are individuals identifiable? Is anonymization needed?
- [ ] Licensing: are data sources used within their license terms?
- [ ] Potential harms: could the dataset be misused?
- [ ] Documentation: is the dataset documented with a datasheet/data card?

## Output Format

Produce:
1. **Distribution report**: per-class counts, imbalance ratios, co-occurrence matrix
2. **Bias findings**: identified biases with severity and actionability
3. **Split recommendation**: stratification strategy with validation results
4. **Expansion plan**: prioritized suggestions for addressing gaps
5. **Ethics checklist**: completed checklist with notes per item

Versions

RepositoryScoreLicence
This page · fcakyon/phd-skills100MIT
wshobson/agents100MIT

Database snapshot: 13.11.2026 · package version dcee46fd004c (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100validating-ai-ethics-and-fairness
    Validate AI/ML models and datasets for bias, fairness, and ethical concerns. Use when auditing AI systems for ethical compliance, fairness assessment, or bias detection. Trigger with phrases like "evaluate model fairness", "check for bias", or "validate AI ethics".
  • 90bias-fairness-assessment
    Assess an AI model, feature, or dataset for bias and fairness across groups — representational and allocative harms, disparate performance, and skewed refusals — using appropriate fairness metrics, and recommend mitigations. Use when evaluating whether an AI system treats people equitably.
  • 100validating-ai-ethics-and-fairness
    Validate AI/ML models and datasets for bias, fairness, and ethical concerns. Use when auditing AI systems for ethical compliance, fairness assessment, or bias detection. Trigger with phrases like "evaluate model fairness", "check for bias", or "validate AI ethics".
  • 100dataset-loader-creator
    Create dataset loader creator operations. Auto-activating skill for ML Training. Triggers on: dataset loader creator, dataset loader creator Part of the ML Training skill category. Use when working with dataset loader creator functionality. Trigger with phrases like "dataset…
  • 100golden-dataset
    Golden dataset lifecycle patterns for curation, versioning, quality validation, and CI integration. Use when building evaluation datasets, managing dataset versions, validating quality scores, or integrating golden tests into pipelines.
  • 100grad-ai-ethics
    Apply AI ethics frameworks (fairness, accountability, transparency, privacy) to evaluate AI systems for algorithmic bias, explainability gaps, and value alignment failures. Use this skill when the user needs to audit an AI system for ethical risks, design fairness constraints,…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.