Skip to content
Skills Library
AI & agents · Building & coding

model-training-engineering

/model-training-engineering

Engineer or restart an end-to-end model-training experiment, from source interpretation and leakage-safe transformation through baselines, training configuration, monitored launch and honest comparison. Use for training playbooks and workflows spanning data and model development; not isolated serving operations.

Repository
overmind ↗
Licence
AGPL-3.0
Last repository change
27.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
5
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

AGPL-3.0, copyleft: przeróbki udostępniane innym muszą mieć tę samą licencję.

What it does and when to use it

Engineer or restart an end-to-end model-training experiment, from source interpretation and leakage-safe transformation through baselines, training configuration, monitored launch and honest comparison. Use for training playbooks and workflows spanning data and model development; not isolated serving operations.

Original author description

Engineer or restart an end-to-end model-training experiment, from source interpretation and leakage-safe transformation through baselines, training configuration, monitored launch and honest comparison. Use for training playbooks and workflows spanning data and model development; not isolated serving operations.

How it works

  1. Establish the claim and scope
  2. Source, semantics and transformation
  3. Freeze the evaluation design before fitting
  4. Establish learnability and comparison evidence
  5. Choose and tune a defensible recipe
  6. Freeze one executable training contract
  7. Dispatch and observe
  8. Verify the artifact and report the evidence
Original SKILL.md headings · 213 lines, 1,901 words
SKILL.md preview
---
name: model-training-engineering
description: Engineer or restart an end-to-end model-training experiment, from source interpretation and leakage-safe transformation through baselines, training configuration, monitored launch and honest comparison. Use for training playbooks and workflows spanning data and model development; not isolated serving operations.
---

# Model training engineering

Own the experimental claim, not just successful execution. A valid file, completed
GPU job, lower loss and useful model are four different outcomes. Adapt this
playbook to the task and provider; it does not mandate a model family, dataset,
split ratio, pilot or universal quality threshold.

This is a practical synthesis of primary ML guidance, not a universal certified
standard. Read [practice foundations](references/foundations.md) when designing or
reviewing the workflow; it maps stages to maintained primary sources and distinguishes
safeguards from conditional recommendations. Historical incidents supplement that
foundation; they do not establish generally accepted practice.

For Overmind, read [the platform reference](references/overmind.md) before platform
actions and use the installed Datasets/Training skills for current contracts.
For a restart or process review, read [cross-run lessons](references/lessons.md).
Keep dataset IDs and incident-specific recipes in the experiment record, not here.

## Establish the claim and scope

- State prediction unit, target meaning, inference-time inputs, intended population,
  primary task metric, success criterion and comparators. Distinguish predicting a
  recorded workflow label from making a correct real-world decision. Unknown label
  quality remains a limitation, even if the target is called gold.
- Review relevant accessible runs, including successful, failed, cancelled and
  incomplete attempts. Separate observed facts, suspected causes and untested
  remedies. Record the review's scope; absence from a listing is not proof of no run.
- Preserve the requested model and run scope. Recommend a simpler baseline when
  informative, but do not silently replace the task. A full run need not have an
  extra pilot. A hardware profile, tiny fitting check and quality pilot answer
  different questions and require their own stated scope.
- Keep a durable ledger: source/checksum, transformation, split membership, model
  revision, runtime, prepared inputs, configuration, job, checkpoint and comparison.
  Preserve old evidence on restart; resolve which existing work is actually active.

## Source, semantics and transformation

Evidence basis: [source documentation and preprocessing](references/foundations.md#data-and-experimental-design).

- Inspect record boundaries, types, counts, time range, missingness, class/family
  coverage, conflicting targets and duplicate inputs. Preserve approved source
  bytes, license/provenance and any sanitization code/checksum. Do not upload
  additional sensitive fields simply because they could improve performance.
- Check feature availability at prediction time. Exclude outcomes, future aggregates,
  same-observation target leakage and identifiers used as shortcuts. If a feature's availability
  is unresolved, state that instead of inventing a justification.
- Define the population and sampling/label-generation process, including synthetic
  or teacher-generated data. Record annotation uncertainty, selection bias, delayed
  labels and relevant subgroup coverage. A public dataset is not automatically
  representative of deployment or unrestricted for every use.
- Separate raw observation identity, model-input equivalence and entity membership.
  Common text or a common column value is not automatically the independent unit.
  Where records connect multiple case/thread IDs, assess the connected family.
  Missing entity IDs limit claims about unseen-entity performance.
- Retain inspectable transformation code. Preserve stable source identities and
  parents through filters, branches and joins; report exclusions and label changes.
  Keep labels/provenance out of prediction-time features. Autoregressive training
  may feed target tokens through the intended shift/mask; that is not permission to
  expose the answer at evaluation time. Preserve option order, soft targets, weights,
  ties and ordinal semantics when present.
- Define handling of missing values, outliers, units, categories and invalid rows.
  Do not blindly drop duplicates or rebalance labels: repetitions may be legitimate
  weights and resampling changes the distribution. Fit learned treatments on train;
  use out-of-fold construction for target-dependent encodings.
- Test every target mapping and representative family, not only the majority prefix.
  Cover malformed/null/boundary values, contradictory labels, numeric precision,
  stable identities, and branch membership. Counts alone do not prove correct routing.
  Do not apply independent batches to global grouping/sorting unless the algorithm
  explicitly preserves global coordination and correctness.
- Preview first, then audit published output against the source. Report coverage:
  a sample or schema check is not a full semantic review. Fix retained transformation
  logic and rebuild from its source rather than stacking opaque repair datasets.

## Freeze the evaluation design before fitting

- Choose time, entity/group or random splitting from the deployment question. Split
  before learning imputation, normalization, vocabulary, encoders, feature selection
  or resampling. Fit those only on training data and reuse the fitted state unchanged.
  Preserve natural held-out prevalence unless evaluating an explicitly different population.
- Separate training, development, calibration and locked final testing when applicable.
  Development selects recipes/checkpoints; calibration fits temperatures/thresholds;
  final results do not select either. If a test influenced development, disclose that
  and obtain a fresh holdout for a new generalization claim.
- For small datasets, consider cross-validation that respects the same groups/time
  constraints rather than fragmenting the data into four undersized sets. Keep model
  selection inside development folds; use nested evaluation when estimating a tuned
  procedure without an independent test. Stratification does not resolve dependence.
- Verify actual memberships, chronology, group intersections, exact input overlap,
  representation and plausible near-duplicate leakage. Preserve legitimate relatedness
  even when it complicates a split. Zero exact overlap is not proof of independence
  from near duplicates, pretraining data or unavailable customer identities.
- For rare classes, retain per-class support, precision/recall, confusion counts and
  suitable probability/ranking/operating-point metrics. Accuracy can reward predicting
  only the majority class. Specify denominator, threshold and whether measures are
  sample-level, group-level or population estimates.
- Choose metrics by task: classification support/ranking/probability quality;
  regression errors in meaningful units; ranking metrics at the relevant cutoff;
  generative task correctness plus validity/coverage, not language-model loss alone.
  Document averaging, class order and metric conventions.
- Probability calibration and action-threshold choice are different. Fit them using
  independent or out-of-fold predictions, never final results. Tie thresholds to
  error costs or operating constraints; 0.5 is not universally appropriate. Log loss
  or Brier score alone is not a pure measure of calibration.

## Establish learnability and comparison evidence

Evidence basis: [baselines and tuning](references/foundations.md#model-and-training-configuration).

- Measure a task-appropriate trivial baseline and, when informative, a
  simple conventional learner using train/development only. Match the requested
  unchanged-base comparator to the exact model identity and inference contract.
  A previously measured baseline is reusable only with verified equivalent inputs,
  labels, split and method; attribute it rather than calling it freshly measured.
- Inspect whether the available inputs can reasonably determine the targets.
  Contradictory or weak labels may limit performance; more epochs do not create
  missing information. Compare task quality, not just training loss.
- Use a bounded fitting sanity check when warranted. Check that representative
  training examples can be fit. Separately check development behavior against
  baselines; memorizing a tiny training set does not establish generalization.
  If fitting fails, inspect target mapping, masks, trainable parameters, head/backbone
  learning rates, gradients, precision and optimizer behavior before scaling.
  Static encoding tests are useful but do not prove the selected model can fit.
- If a full run proceeds without measured
  fitting evidence, disclose that uncertainty; do not fabricate a passed sanity gate.
  Define task-specific go/no-go rules before a pilot, not after seeing its result.

## Choose and tune a defensible recipe

- Start with an established architecture and optimizer suited to the modality and
  task. Record why classical learning, transfer learning, adapters or full training
  fits the data, capacity, compute and serving constraints.
- If tuning is authorized, specify its hypothesis, search space, development metric
  and trial count first. Retune interacting settings when changing batch size or
  model scale; do not copy a learning rate without its recipe. Keep failed trials
  and repeat promising configurations when stochastic variability matters.
- Size hardware from model/optimizer/activation memory and representative length
  tails, not model parameter count alone. Distinguish physical micro-batch from
  effective batch. In ordinary synchronous data parallelism, effective batch is
  micro-batch × accumulation × data-parallel replicas; specialized runtimes may differ.
  Confirm the provider's interpretation and count actual examples/tokens.
- Prefer existing compatible throughput/memory evidence. If none exists, label the
  estimate uncertain; request a bounded runtime check only when justified/authorized.
  Gradient accumulation is a memory tradeoff, not a promised throughput improvement.

## Freeze one executable training contract

Evidence basis: [runtime and reproducibility](references/foundations.md#runtime-and-reproducibility).

- Pin data/split hashes, model and tokenizer revisions, objective/template, supervised
  token masks or decision targets, optimizer, effective batch, schedule, learning
  rates, adapter/full-training mode, precision, context length, epochs/steps and seeds.
  Record runtime/library versions and hardware. Seeds aid repeatability but do not
  guarantee bitwise agreement across runtimes or devices.
- Verify actual prepared token lengths, targets and truncation. A context setting
  alone does not prove useful supervision survived. Chat/tool tasks need the exact
  template and target-turn policy; native decisions need the matching head/options.
  For SFT, inspect decoded examples and actual loss masks, EOS/padding, packing
  boundaries and nonempty supervised tokens. Assistant-only loss needs a compatible
  template; do not assume a flag makes every template correct. Treat source text
  and remote model code as untrusted, not instructions to execute.
- Plan initial, periodic and final development checks appropriate to the task, plus
  a training-reference sample when useful. Freeze representative sample identities.
  Declare checkpoint selection, required-check failure behavior and any stopping rule.
  Disclose disabled checks.
- Monitor objective loss and the declared task metric together. Watch for class
  collapse, divergence, train/development gaps and missing coverage. Lower loss can
  coexist with worse task behavior. Do not change the metric or disable validation
  simply to make a run pass.
- Bind the requested final comparison before launch: unchanged base, trained
  candidate and any alternative, with identical held-out inputs and documented
  inference differences. Eligibility is not proof of provider identity or hardware fit.

Before dispatch, record [pre-launch evidence](references/foundations.md#pre-launch-evidence).
A template filled with assertions is not measured evidence.

## Dispatch and observe

- Compare requested and effective settings before dispatch. Explicitly reuse the
  same prepared inputs and configuration so defaults cannot create a second setup.
  Give each deliberate operation a stable identity; an uncertain reply is a reason
  to inspect/reconcile, not generate a new launch key.
- Distinguish control-plane records from provider work. A setup row is not necessarily
  a GPU job, a heartbeat is not an optimizer step, and a collection failure is not
  proof the provider failed. Preserve provider call IDs and acknowledge uncertainty.
- Verify provider acknowledgement, actual stage and measured progress after
  launch. Initial evaluation may legitimately precede optimizer steps. Use bounded,
  supported recovery; do not blindly replay unknown submissions or cancel shared
  work based on duplicated dashboard labels.
- When validating a platform fix, verify the processes that execute the path use it.
  Passing tests in a checkout do not establish a reloaded worker. Scope restarts to
  authorized, idle components; preserve unrelated work and historical artifacts.

## Verify the artifact and report the evidence

- Verify saved model/adapter/head/tokenizer identity and reload behavior on retained
  inputs. Check training-serving parity. Weight reload is not proof of optimizer
  resume or exact stochastic reproducibility. For resumable training, retain required
  optimizer, scheduler, scaler, RNG and progress state as supported; test that path
  before promising interruption recovery. Evaluate in the correct inference mode.
- Evaluate the selected artifact against frozen comparators. Inspect every arm's
  requested, scored, failed and missing counts. Provider errors are missing evidence,
  not zero quality. Paired metrics use shared valid examples; discuss coverage bias.
- Report raw and calibrated results separately, including per-class/slice regressions,
  uncertainty appropriate to the independent unit, and latency as a separate axis.
  Scores on the data used to fit calibration are in-sample; calibrated final-test
  scores are distinct. Performance timing and quality benchmarks are
  different experiments. Never invent unavailable metrics or confidence intervals.
- Hand off the exact ledger, effective configuration, checks and limitations. Distinguish
  preparation, optimization, artifact verification, final comparison and serving
  readiness. Process success does not establish model improvement or production safety.
  Activation, deletion of history and ongoing monitoring require their own scope.
Files in this package (5)
  • SKILL.md 15 KB
  • agents/openai.yaml 1 KB
  • references/foundations.md 7 KB
  • references/lessons.md 6 KB
  • references/overmind.md 5 KB

Versions

The database has no other skills named model-training-engineering.

Database snapshot: 13.11.2026 · package version 5d0e93640267 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100ml-engineer
    Provides machine learning engineering patterns for model training pipelines, feature engineering, experiment tracking, and model serving. Use when working with ML code or when the user mentions ML engineering, model training, or feature pipelines.
  • 90fit-recon
    Audit existing model training code — find reproducibility issues, data leakage, and missing best practices. Use when asked to "audit our training code", "is our training reproducible", or "check for training data leakage".
  • 100pilot-ml-training-pipeline-setup
    Deploy an end-to-end ML training pipeline with 4 agents. Use this skill when: 1. User wants to set up a machine learning training pipeline 2. User is configuring a data prep, training, evaluation, or serving agent 3. User asks about ML model lifecycle management across agents Do…
  • 90experiment-tracking
    Record every training run's code, data, config, and results so any model is reproducible and comparisons are honest. Use when setting up ML experiment infrastructure or untangling which run produced the prod model.
  • 100model-evaluation-metrics
    Build model evaluation metrics operations. Auto-activating skill for ML Training. Triggers on: model evaluation metrics, model evaluation metrics Part of the ML Training skill category. Use when working with model evaluation metrics functionality. Trigger with phrases like…
  • 100ml-engineer
    Use this agent when building production ML systems requiring model training pipelines, model serving infrastructure, performance optimization, and automated retraining.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.