training-data-annotation
/training-data-annotation
Design and improve human annotation programs for machine-learning training, validation, and evaluation data, including task and label schemas, active acquisition, annotator guidance, calibration, disagreement, adjudication, provenance, and cost-quality stopping. Do not use for general statistical inference, data storage pipelines, model training, or annotation-interface implementation; route those concerns to data-scientist, data-engineering, ml-engineering, or product-design-and-ux.
- Author
- magnus919 ↗
- Repository
- agent-skills ↗
- Licence
- MIT
- Last repository change
- 25.10.2026
Assessment and security
- Files in this package
- 11
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Design and improve human annotation programs for machine-learning training, validation, and evaluation data, including task and label schemas, active acquisition, annotator guidance, calibration, disagreement, adjudication, provenance, and cost-quality stopping. Do not use for general statistical inference, data storage pipelines, model training, or annotation-interface implementation; route those concerns to data-scientist, data-engineering, ml-engineering, or product-design-and-ux.
Original author description
Design and improve human annotation programs for machine-learning training, validation, and evaluation data, including task and label schemas, active acquisition, annotator guidance, calibration, disagreement, adjudication, provenance, and cost-quality stopping. Do not use for general statistical inference, data storage pipelines, model training, or annotation-interface implementation; route those concerns to data-scientist, data-engineering, ml-engineering, or product-design-and-ux.
How it works
- Workflow
- Acquisition guardrails
- Human and quality guardrails
- Routing and exit
- When not to use
- Templates
SKILL.md preview
--- name: training-data-annotation description: >- Design and improve human annotation programs for machine-learning training, validation, and evaluation data, including task and label schemas, active acquisition, annotator guidance, calibration, disagreement, adjudication, provenance, and cost-quality stopping. Do not use for general statistical inference, data storage pipelines, model training, or annotation-interface implementation; route those concerns to data-scientist, data-engineering, ml-engineering, or product-design-and-ux. license: MIT compatibility: Reference-only methodology; no runtime dependency. metadata: spec-version: '1.0' tags: annotation, active-learning, labeling, human-review, training-data --- # Training Data Annotation Treat annotation as a measurement system and a human workflow. The objective is useful, auditable labels for a defined downstream decision, not the largest possible label count. ## Workflow 1. Define the downstream task, population, label unit, label set, edge cases, abstention label, and what constitutes an unusable item. Separate training, validation, held-out evaluation, and challenge data before sampling. 2. Write an annotation guide with positive and negative examples, boundary rules, source requirements, escalation, and a change log. Have domain owners approve consequential definitions. 3. Establish provenance and rights: source, collection time, license/consent, transformations, sensitive fields, access, retention/deletion, and dataset version. Do not move data into a labeling system until this record is clear. 4. Acquire items using a declared mix of representative sampling and targeted acquisition. Uncertainty finds confusing items; diversity and novelty find gaps; stratification protects rare or high-risk slices. Keep acquisition batches identifiable and retain a representative held-out audit sample. 5. Qualify annotators for the task, train them on calibration items, and measure quality and workload. Record role and relevant expertise in restricted metadata. Do not use agreement as a universal truth score. 6. Run blinded calibration and periodic blind rechecks. Compare labels with adjudicated references where available; preserve meaningful disagreement, ambiguity, and abstention instead of forcing consensus. 7. Review disagreement by item, label, slice, annotator, and guide version. Adjudicate only with a stated rule and qualified authority. Feed confirmed guide defects back into training and re-review affected labels. 8. Stop or continue based on marginal downstream value, coverage, quality, cost, and time. An item quota may stop spending but cannot by itself establish sufficient quality. Freeze the manifest and hand off to data-engineering for storage/orchestration and ml-engineering for training use. ## Acquisition guardrails Never let model confidence define the whole dataset. Sample a representative held-out audit independently of active-learning selection, and compare active batch outcomes against it. Track uncertainty method, model/version, seed, selection scores, slice quotas, and excluded items. If confidence is uncalibrated or the model is out of distribution, treat its ranking as a search heuristic and increase diversity, novelty, and human review. ## Human and quality guardrails Design pay, volume, time limits, breaks, and escalation so workers can perform the task reliably; route employment, compensation, and legal questions to the appropriate human owner. Blind annotators to model predictions when measuring independent labels. If preannotations are shown, measure anchoring by a blinded control or separate review pass. Report agreement by label type and slice with the uncertainty and prevalence needed to interpret it. A minority label can be correct when the item is ambiguous; adjudication must explain the rule used. ## Routing and exit Load [references/task-and-label-design.md](references/task-and-label-design.md) for schemas and guide design, [references/acquisition-and-stopping.md](references/acquisition-and-stopping.md) for sampling and stopping, [references/annotator-quality.md](references/annotator-quality.md) for people and disagreement, and [references/provenance-and-rights.md](references/provenance-and-rights.md) for dataset authority. Use the templates when a durable record is needed. For primary methodology sources and transfer limits, read [references/source-index.md](references/source-index.md). Route statistical estimators and power to [data-scientist](../data-scientist/SKILL.md), storage and pipelines to [data-engineering](../data-engineering/SKILL.md), model training and leakage checks to [ml-engineering](../ml-engineering/SKILL.md), and labeling-interface interaction design to [product-design-and-ux](../product-design-and-ux/SKILL.md). ## When not to use Do not use this skill as the primary owner for statistical inference or experimental design, data platforms, model training, or UI implementation. Those neighboring skills consume this skill's annotation manifest and quality evidence. ## Templates - [annotation guide](templates/annotation-guide.md) — use when recording this stage of the annotation workflow. - [annotation plan](templates/annotation-plan.md) — use when recording this stage of the annotation workflow. - [quality report](templates/quality-report.md) — use when recording this stage of the annotation workflow.
Files in this package (11)
- README.md 2 KB
- SKILL.md 5 KB
- evals/evals.json 4 KB
- references/acquisition-and-stopping.md 1 KB
- references/annotator-quality.md 1 KB
- references/provenance-and-rights.md 1 KB
- references/source-index.md 1 KB
- references/task-and-label-design.md 1 KB
- templates/annotation-guide.md 1 KB
- templates/annotation-plan.md 1 KB
- templates/quality-report.md 1 KB
Versions
The database has no other skills named training-data-annotation.
Identical content appears in 1 other GitHub folders. We show one copy with the highest score.
Database snapshot: 13.11.2026 · package version ad3221e736ab (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100ml-engineer
Provides machine learning engineering patterns for model training pipelines, feature engineering, experiment tracking, and model serving. Use when working with ML code or when the user mentions ML engineering, model training, or feature pipelines. - 100product-management-human-data-platform
Guides product management for human data platforms—annotation and labeling products, workforce workflows, task design, quality systems (gold sets, adjudication, inter-annotator agreement), customer ML-team project delivery, contributor experience, and privacy-safe handling of… - 100model-training
Train machine learning models end-to-end, covering data loading, preprocessing, architecture selection, training loops, validation, and checkpointing. Use when the user requests model training or provides relevant inputs for this workflow. - 100model-evaluation-metrics
Build model evaluation metrics operations. Auto-activating skill for ML Training. Triggers on: model evaluation metrics, model evaluation metrics Part of the ML Training skill category. Use when working with model evaluation metrics functionality. Trigger with phrases like… - 100ml-pipeline-creation
Design, implement, and validate reproducible machine-learning pipelines spanning data preparation, training, evaluation, registry, and deployment gates. Use when the user requests an ML pipeline, needs to turn model scripts into an orchestrated workflow, or provides pipeline… - 100data-scientist
Execute data science workflows from exploration to production. Apply machine learning modeling, statistical analysis, A/B testing, causal inference, feature engineering, model evaluation, and MLOps patterns. Triggers on "build predictive model", "design A/B test", "feature…