Skip to content
Skills Library
AI & agents · Planning

training-data-annotation

/training-data-annotation

Design and improve human annotation programs for machine-learning training, validation, and evaluation data, including task and label schemas, active acquisition, annotator guidance, calibration, disagreement, adjudication, provenance, and cost-quality stopping. Do not use for general statistical inference, data storage pipelines, model training, or annotation-interface implementation; route those concerns to data-scientist, data-engineering, ml-engineering, or product-design-and-ux.

Licence
MIT
Last repository change
25.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
11
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Design and improve human annotation programs for machine-learning training, validation, and evaluation data, including task and label schemas, active acquisition, annotator guidance, calibration, disagreement, adjudication, provenance, and cost-quality stopping. Do not use for general statistical inference, data storage pipelines, model training, or annotation-interface implementation; route those concerns to data-scientist, data-engineering, ml-engineering, or product-design-and-ux.

Original author description

Design and improve human annotation programs for machine-learning training, validation, and evaluation data, including task and label schemas, active acquisition, annotator guidance, calibration, disagreement, adjudication, provenance, and cost-quality stopping. Do not use for general statistical inference, data storage pipelines, model training, or annotation-interface implementation; route those concerns to data-scientist, data-engineering, ml-engineering, or product-design-and-ux.

How it works

  1. Workflow
  2. Acquisition guardrails
  3. Human and quality guardrails
  4. Routing and exit
  5. When not to use
  6. Templates
Original SKILL.md headings · 83 lines, 587 words
SKILL.md preview
---
name: training-data-annotation
description: >-
  Design and improve human annotation programs for machine-learning training,
  validation, and evaluation data, including task and label schemas, active
  acquisition, annotator guidance, calibration, disagreement, adjudication,
  provenance, and cost-quality stopping. Do not use for general statistical
  inference, data storage pipelines, model training, or annotation-interface
  implementation; route those concerns to data-scientist, data-engineering,
  ml-engineering, or product-design-and-ux.
license: MIT
compatibility: Reference-only methodology; no runtime dependency.
metadata:
  spec-version: '1.0'
  tags: annotation, active-learning, labeling, human-review, training-data
---

# Training Data Annotation

Treat annotation as a measurement system and a human workflow. The objective is
useful, auditable labels for a defined downstream decision, not the largest
possible label count.

## Workflow

1. Define the downstream task, population, label unit, label set, edge cases,
   abstention label, and what constitutes an unusable item. Separate training,
   validation, held-out evaluation, and challenge data before sampling.
2. Write an annotation guide with positive and negative examples, boundary
   rules, source requirements, escalation, and a change log. Have domain owners
   approve consequential definitions.
3. Establish provenance and rights: source, collection time, license/consent,
   transformations, sensitive fields, access, retention/deletion, and dataset
   version. Do not move data into a labeling system until this record is clear.
4. Acquire items using a declared mix of representative sampling and targeted
   acquisition. Uncertainty finds confusing items; diversity and novelty find
   gaps; stratification protects rare or high-risk slices. Keep acquisition
   batches identifiable and retain a representative held-out audit sample.
5. Qualify annotators for the task, train them on calibration items, and measure
   quality and workload. Record role and relevant expertise in restricted
   metadata. Do not use agreement as a universal truth score.
6. Run blinded calibration and periodic blind rechecks. Compare labels with
   adjudicated references where available; preserve meaningful disagreement,
   ambiguity, and abstention instead of forcing consensus.
7. Review disagreement by item, label, slice, annotator, and guide version.
   Adjudicate only with a stated rule and qualified authority. Feed confirmed
   guide defects back into training and re-review affected labels.
8. Stop or continue based on marginal downstream value, coverage, quality,
   cost, and time. An item quota may stop spending but cannot by itself establish sufficient quality. Freeze the
   manifest and hand off to data-engineering for storage/orchestration and
   ml-engineering for training use.

## Acquisition guardrails

Never let model confidence define the whole dataset. Sample a representative
held-out audit independently of active-learning selection, and compare active
batch outcomes against it. Track uncertainty method, model/version, seed,
selection scores, slice quotas, and excluded items. If confidence is uncalibrated
or the model is out of distribution, treat its ranking as a search heuristic and
increase diversity, novelty, and human review.

## Human and quality guardrails

Design pay, volume, time limits, breaks, and escalation so workers can perform
the task reliably; route employment, compensation, and legal questions to the
appropriate human owner. Blind annotators to model predictions when measuring
independent labels. If preannotations are shown, measure anchoring by a blinded
control or separate review pass. Report agreement by label type and slice with
the uncertainty and prevalence needed to interpret it. A minority label can be
correct when the item is ambiguous; adjudication must explain the rule used.

## Routing and exit

Load [references/task-and-label-design.md](references/task-and-label-design.md)
for schemas and guide design, [references/acquisition-and-stopping.md](references/acquisition-and-stopping.md)
for sampling and stopping, [references/annotator-quality.md](references/annotator-quality.md)
for people and disagreement, and [references/provenance-and-rights.md](references/provenance-and-rights.md)
for dataset authority. Use the templates when a durable record is needed.
For primary methodology sources and transfer limits, read
[references/source-index.md](references/source-index.md).

Route statistical estimators and power to [data-scientist](../data-scientist/SKILL.md),
storage and pipelines to [data-engineering](../data-engineering/SKILL.md), model
training and leakage checks to [ml-engineering](../ml-engineering/SKILL.md), and
labeling-interface interaction design to [product-design-and-ux](../product-design-and-ux/SKILL.md).

## When not to use

Do not use this skill as the primary owner for statistical inference or
experimental design, data platforms, model training, or UI implementation.
Those neighboring skills consume this skill's annotation manifest and quality
evidence.

## Templates

- [annotation guide](templates/annotation-guide.md) — use when recording this stage of the annotation workflow.
- [annotation plan](templates/annotation-plan.md) — use when recording this stage of the annotation workflow.
- [quality report](templates/quality-report.md) — use when recording this stage of the annotation workflow.
Files in this package (11)
  • README.md 2 KB
  • SKILL.md 5 KB
  • evals/evals.json 4 KB
  • references/acquisition-and-stopping.md 1 KB
  • references/annotator-quality.md 1 KB
  • references/provenance-and-rights.md 1 KB
  • references/source-index.md 1 KB
  • references/task-and-label-design.md 1 KB
  • templates/annotation-guide.md 1 KB
  • templates/annotation-plan.md 1 KB
  • templates/quality-report.md 1 KB

Versions

The database has no other skills named training-data-annotation.

Identical content appears in 1 other GitHub folders. We show one copy with the highest score.

Database snapshot: 13.11.2026 · package version ad3221e736ab (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100ml-engineer
    Provides machine learning engineering patterns for model training pipelines, feature engineering, experiment tracking, and model serving. Use when working with ML code or when the user mentions ML engineering, model training, or feature pipelines.
  • 100product-management-human-data-platform
    Guides product management for human data platforms—annotation and labeling products, workforce workflows, task design, quality systems (gold sets, adjudication, inter-annotator agreement), customer ML-team project delivery, contributor experience, and privacy-safe handling of…
  • 100model-training
    Train machine learning models end-to-end, covering data loading, preprocessing, architecture selection, training loops, validation, and checkpointing. Use when the user requests model training or provides relevant inputs for this workflow.
  • 100model-evaluation-metrics
    Build model evaluation metrics operations. Auto-activating skill for ML Training. Triggers on: model evaluation metrics, model evaluation metrics Part of the ML Training skill category. Use when working with model evaluation metrics functionality. Trigger with phrases like…
  • 100ml-pipeline-creation
    Design, implement, and validate reproducible machine-learning pipelines spanning data preparation, training, evaluation, registry, and deployment gates. Use when the user requests an ML pipeline, needs to turn model scripts into an orchestrated workflow, or provides pipeline…
  • 100data-scientist
    Execute data science workflows from exploration to production. Apply machine learning modeling, statistical analysis, A/B testing, causal inference, feature engineering, model evaluation, and MLOps patterns. Triggers on "build predictive model", "design A/B test", "feature…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.