Skip to content
Skills Library
AI & agents · Planning

ml-engineering

/ml-engineering

Plan and execute production ML engineering work — classifier versus generative model selection, zero-shot baselines, coding specialization, model training and fine-tuning (LoRA/QLoRA), evaluation and eval-set design, quantization decisions, inference deployment, lineage, feature parity, drift response, and regression triage, grounded in practical engineering patterns for production ML systems. Do not use for statistical modeling and experimental design (that's the data scientist) or for operating a specific inference engine (that's a tool skill such as llama-cpp or vllm).

Licence
MIT
Last repository change
25.10.2026

Assessment and security

95/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content30 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
46
Scripts
10
HTML comments
0
URLs in SKILL.md
1
Deprecation

No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 10 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/audit_sequence_windows.py, scripts/audit_text_outcomes.py, scripts/check-eval-overlap.py, scripts/compare_task_runs.py, scripts/nli_score_contract.py, scripts/test_audit_sequence_windows.py, scripts/test_audit_text_outcomes.py, scripts/test_check_eval_overlap.py

What it does and when to use it

Plan and execute production ML engineering work — classifier versus generative model selection, zero-shot baselines, coding specialization, model training and fine-tuning (LoRA/QLoRA), evaluation and eval-set design, quantization decisions, inference deployment, lineage, feature parity, drift response, and regression triage, grounded in practical engineering patterns for production ML systems. Do not use for statistical modeling and experimental design (that's the data scientist) or for operating a specific inference engine (that's a tool skill such as llama-cpp or vllm).

Original author description

Plan and execute production ML engineering work — classifier versus generative model selection, zero-shot baselines, coding specialization, model training and fine-tuning (LoRA/QLoRA), evaluation and eval-set design, quantization decisions, inference deployment, lineage, feature parity, drift response, and regression triage, grounded in practical engineering patterns for production ML systems. Do not use for statistical modeling and experimental design (that's the data scientist) or for operating a specific inference engine (that's a tool skill such as llama-cpp or vllm).

How it works

  1. The ML Engineer's Domain
  2. Reference Files
  3. Templates
  4. Scripts
  5. Evals
  6. Specialization decision boundary
  7. Core Principles
  8. When not to use
Original SKILL.md headings · 103 lines, 1,046 words
SKILL.md preview
---
name: ml-engineering
description: >-
  Plan and execute production ML engineering work — classifier versus generative
  model selection, zero-shot baselines, coding specialization, model training and
  fine-tuning (LoRA/QLoRA), evaluation and eval-set design, quantization
  decisions, inference deployment, lineage, feature parity, drift response, and
  regression triage, grounded in practical engineering patterns for production ML
  systems. Do not use for statistical
  modeling and experimental design (that's the data scientist) or for operating
  a specific inference engine (that's a tool skill such as llama-cpp or vllm).
license: MIT
metadata:
  tags: ml, machine-learning, fine-tuning, training, evaluation, quantization, mlops,
    inference, vllm, gguf
  source_repo: https://github.com/magnus919/hermes-profiles
---

# ML Engineering Methodology

Machine learning engineering is the bridge between model research and production systems. This methodology covers the engineering disciplines needed to train, evaluate, deploy, and maintain ML models reliably.

## The ML Engineer's Domain

| You own | You don't own |
|---------|--------------|
| Model training — LoRA/QLoRA fine-tuning, full fine-tuning, distributed training | Statistical modeling and experimental design — that's the data scientist |
| Model evaluation — benchmark suites, custom eval sets, regression testing | Causal inference and hypothesis testing — that's the data scientist |
| Quantization — GGUF, GPTQ, AWQ, bitsandbytes | Training data collection and labeling — that's the data/ML ops team |
| Inference serving — [vLLM](../vllm/SKILL.md), [llama.cpp](../llama-cpp/SKILL.md), TGI, Triton | Business metrics and KPI definition — that's the product manager |
| Evaluation harness — lm-eval-harness, custom pipelines | Data pipeline architecture — that's the data engineer |
| Model deployment — containerization, versioning, A/B testing | Infrastructure provisioning — that's the platform engineer |

## Reference Files

| Reference | When to load |
|-----------|-------------|
| `references/nli-classification.md` | NLI premise/hypothesis contracts, binary/ternary heads, score modes and truncation |
| `references/nli-evaluation.md` | Domain/language validation, calibration, abstention, cost and adaptation experiments |
| `references/nli-sources.md` | Primary NLI evidence and limitations |
| `references/task-specific-extraction.md` | Span/encoder/generative extraction, offsets, source grounding, critical recall and receipts |
| `references/task-specific-summarization.md` | Extractive/abstractive model choice, source/target retention, factuality and downstream compaction |
| `references/text-model-sources.md` | Primary extraction/summarization evidence and uncertainty |
| `references/tabular-representations.md` | Self-supervised tabular encoders, corruption/contrastive objectives, row-context and fair comparisons |
| `references/sequence-sources.md` | Primary-source findings and uncertainty for tabular and sequence representations |
| `references/transaction-sequences.md` | Ordered-event representation, windows, label maturity, forward/entity validation and stateful serving |
| `references/specialization-decisions.md` | Choosing rules/classifiers/zero-shot/prompt/SFT/RL, designing fixed-budget comparisons and interpreting verifier or case-study claims |
| `references/specialization-evidence.md` | Primary-source findings, contradictory results and local validation limits |
| `references/fine-tuning.md` | Setting up a LoRA/QLoRA/ full fine-tuning run — data prep, hyperparameters, validation strategy |
| `references/evaluation.md` | Evaluating a model — benchmark selection, custom eval sets, regression tracking, comparison methodology |
| `references/evaluation-and-lineage.md` | Metric/configuration decisions, repeated stochastic comparisons, end-to-end lineage, temporal feature parity, drift response, and adaptation/serving tradeoffs |
| `references/quantization-inference.md` | Quantizing a model and serving it — GGUF/GPTQ/AWQ/bitsandbytes comparison, calibration data strategies, KV cache quantization, vLLM/llama.cpp/TGI/Triton architecture, production considerations |
| `references/training-infrastructure.md` | Selecting and provisioning training infrastructure — GPU selection, VRAM budgeting, multi-GPU strategies (DDP/FSDP/DeepSpeed), cloud vs on-prem, storage, monitoring |

## Templates

| Template | When to Use |
|-----------|-------------|
| `templates/nli-experiment.md`, `templates/nli-logits.json` | NLI study worksheet and synthetic saved-logit contract |
| `templates/text-model-record.md`, `templates/extraction-receipt.json`, `templates/summary-receipt.json` | Task/lineage worksheet and synthetic extraction/summary contracts |
| `templates/sequence-model-record.md` | Representation, pretraining exposure, time/window/state and evaluation record |
| `templates/sequence-windows.json` | Synthetic time-aware window audit contract |
| `templates/specialization-study.md` | Preregistering task-matched alternatives, data freeze, quality/resource guardrails and stopping rules |
| `templates/task-run-baseline.json`, `templates/task-run-candidate.json` | Synthetic saved-receipt contract examples; not model results |
| `templates/training-run-record.md` | Recording a training or fine-tuning run — model and data versions, full config, environment, eval results — so it can be reproduced |
| `templates/eval-regression-table.md` | Tracking model quality across runs and triaging a regression — one row per eval case or capability subset |
| `templates/quantization-decision-record.md` | Recording a quantization decision — baseline, candidates compared, quality threshold, and rollback path |
| `templates/model-lineage-record.md` | Linking data/features, code/configuration, runs, artifacts, evaluations, registry state, and deployed serving versions |
| `templates/drift-response-record.md` | Recording drift signals, thresholds, diagnosis, retrain/rollback decisions, and post-action evidence |

## Scripts

| Script | When to Use |
|-----------|-------------|
| `scripts/nli_score_contract.py` | Saved NLI score arithmetic and mapping/shape checks; no model execution or calibration |
| `scripts/audit_text_outcomes.py` | Exact source-span and summary-receipt consistency checks; no model or semantic judgment |
| `scripts/audit_sequence_windows.py` | Declared window availability, identity, outcome and fold-boundary screen; no model execution |
| `scripts/compare_task_runs.py` | Descriptive paired task comparison with matched pins/roster, incomplete-evidence rejection and group bootstrap; no release approval |
| `scripts/check-eval-overlap.py` | Checking a training corpus against an eval corpus for test-set leakage (shared n-grams); `--json` for CI, exit 1 when an eval file exceeds the overlap threshold |

## Evals

`evals/evals.json` — output-quality eval manifest for this skill: fine-tuning plan review, eval-set design, quantization decision, deployment plan, regression triage, and training-run reproducibility.

## Specialization decision boundary

Start from the required outcome and permitted prediction-time information.
Include simpler rules, supervised or zero-shot classifiers where appropriate;
classification scores are not automatically calibrated probabilities. A proprietary
LLM case study is orientation, not proof of architecture optimality. Compare
matched context/policy/budgets, preserve critical regressions and require every
registered guardrail before promotion. New helper exit zero is descriptive
comparison completion only; model execution and release remain separate evidence.

Route training operations to [unsloth](../unsloth/SKILL.md), typed decisions to
[system-one](../system-one/SKILL.md), labels/provenance to
[training-data-annotation](../training-data-annotation/SKILL.md), experimental
analysis/calibration to [data-scientist](../data-scientist/SKILL.md), and agent-level
evaluation to [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md).

For tabular foundation-model choice, read `references/tabular-foundations.md`;
for named TabPFN setup/context/inference/fine-tuning, use
[tabpfn](../tabpfn/SKILL.md).

## Core Principles

**Measure before you optimize** — Never quantize, prune, or distill a model without first measuring its baseline performance. Optimization without measurement is guessing.

**Reproducibility is non-negotiable** — Every training run needs a reproducible config: seed, data version, hyperparameters, and evaluation methodology. If you can't reproduce it, you can't ship it.

**Baseline first** — Before running an expensive fine-tuning run, establish a baseline with the base model. If the base model is already good enough, the fine-tuning budget is better spent elsewhere.

**Test at the boundary** — Model evaluation is most informative at the edges of the capability distribution, not at the center. Hard examples reveal more than easy ones.

**The evaluation set is a liability** — Every example in your eval set is a potential test-set leak. Use held-out sets, rotate examples, and periodically audit for contamination with the overlap checker.

## When not to use

Do not use this skill for statistical modeling, experimental design, or causal inference — that's the data scientist's discipline. Do not use it to operate a specific inference engine: for llama.cpp installation, model loading, benchmarking, and troubleshooting, load the [llama-cpp](../llama-cpp/SKILL.md) tool skill instead; for vLLM deployment, model configuration, benchmarking, batching tuning, GPU operation, and upgrade/rollback, load the [vllm](../vllm/SKILL.md) tool skill instead. This skill provides the methodology (eval-set design, quantization trade-offs, deployment plans, regression triage); the tool skills own the runbooks.
Files in this package (46)
  • README.md 3 KB
  • SKILL.md 10 KB
  • evals/evals.json 29 KB
  • evals/specialization-challenges.md 13 KB
  • references/evaluation-and-lineage.md 4 KB
  • references/evaluation.md 1 KB
  • references/fine-tuning.md 1 KB
  • references/nli-classification.md 5 KB
  • references/nli-evaluation.md 4 KB
  • references/nli-sources.md 5 KB
  • references/quantization-inference.md 41 KB
  • references/sequence-sources.md 2 KB
  • references/specialization-decisions.md 15 KB
  • references/specialization-evidence.md 6 KB
  • references/tabular-foundations.md 5 KB
  • references/tabular-representations.md 5 KB
  • references/task-specific-extraction.md 6 KB
  • references/task-specific-summarization.md 6 KB
  • references/text-model-sources.md 2 KB
  • references/training-infrastructure.md 4 KB
  • references/transaction-sequences.md 6 KB
  • scripts/audit_sequence_windows.py 6 KB
  • scripts/audit_text_outcomes.py 6 KB
  • scripts/check-eval-overlap.py 9 KB
  • scripts/compare_task_runs.py 7 KB
  • scripts/nli_score_contract.py 4 KB
  • scripts/test_audit_sequence_windows.py 4 KB
  • scripts/test_audit_text_outcomes.py 4 KB
  • scripts/test_check_eval_overlap.py 8 KB
  • scripts/test_compare_task_runs.py 5 KB
  • scripts/test_nli_score_contract.py 4 KB
  • templates/drift-response-record.md 1 KB
  • templates/eval-regression-table.md 2 KB
  • templates/extraction-receipt.json 1 KB
  • templates/model-lineage-record.md 1 KB
  • templates/nli-experiment.md 1 KB
  • templates/nli-logits.json 1 KB
  • templates/quantization-decision-record.md 2 KB
  • templates/sequence-model-record.md 2 KB
  • templates/sequence-windows.json 2 KB
  • templates/specialization-study.md 3 KB
  • templates/summary-receipt.json 1 KB
  • templates/task-run-baseline.json 1 KB
  • templates/task-run-candidate.json 1 KB
  • templates/text-model-record.md 1 KB
  • templates/training-run-record.md 3 KB

Versions

The database has no other skills named ml-engineering.

Database snapshot: 13.11.2026 · package version b60ac7ba18a6 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100llama
    Expert Meta Llama (Llama 3 / 3.2 / 3.3) assistance covering fine-tuning (LoRA/QLoRA), vLLM deployment, and prompt templates. Use when deploying open-weight foundation models locally or in the cloud.
  • 100llmops-fine-tuning-and-deployment
    Guides deciding whether fine-tuning is the right approach for an LLM use case (versus prompt engineering or RAG), and if so, executing it safely — covering LoRA/QLoRA/full fine-tuning configs, dataset curation, evaluation, and deployment. Use when the user asks whether to…
  • 100lora-qlora-recipes
    Configure LoRA and QLoRA supervised fine-tuning with current best-practice hyperparameters. Use when writing or reviewing a LoRA/QLoRA training configuration, choosing rank/alpha/target modules, or deciding between LoRA, QLoRA, and full fine-tuning.
  • 100fine-tuning-llms-lora-qlora
    Use this skill when fine-tuning open-weight foundation models on domain data efficiently using parameter-efficient LoRA and 4-bit QLoRA.
  • 90unsloth-fine-tuning
    Fast LLM fine-tuning with Unsloth - 2-5x faster training, 50-80% less VRAM. Use for single-GPU LoRA/QLoRA SFT, GRPO/RL reasoning training, vision/TTS fine-tuning, and GGUF export to Ollama/vLLM/llama.cpp. Supports 300+ models including Llama, Qwen, Gemma, DeepSeek, Mistral, Phi, and gpt-oss.
  • 100llm-finetuning
    Use this skill when the user says 'fine-tune an LLM', 'LoRA', 'QLoRA', 'PEFT', 'SFT', 'instruction tuning', 'TRL SFTTrainer', 'DPO', 'train on my data', 'merge adapter'. Decides whether fine-tuning is the right tool, prepares chat-format datasets, configures LoRA/QLoRA with PEFT…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.