Skip to content
Skills Library
AI & agents · Review & audit

ml-engineer

/ml-engineer

Use when a task needs the judgment of a Machine Learning/AI Engineer — checking a train/test split for data leakage, choosing an evaluation metric that accounts for class imbalance rather than defaulting to accuracy, verifying training and serving feature computation match (train-serving skew), monitoring a deployed model for data/concept drift, or deciding whether a hyperparameter-tuning process has silently overfit to the test set.

Licence
MIT
Last repository change
26.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
4
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Use when a task needs the judgment of a Machine Learning/AI Engineer — checking a train/test split for data leakage, choosing an evaluation metric that accounts for class imbalance rather than defaulting to accuracy, verifying training and serving feature computation match (train-serving skew), monitoring a deployed model for data/concept drift, or deciding whether a hyperparameter-tuning process has silently overfit to the test set.

Original author description

Use when a task needs the judgment of a Machine Learning/AI Engineer — checking a train/test split for data leakage, choosing an evaluation metric that accounts for class imbalance rather than defaulting to accuracy, verifying training and serving feature computation match (train-serving skew), monitoring a deployed model for data/concept drift, or deciding whether a hyperparameter-tuning process has silently overfit to the test set.

How it works

  1. Identity
  2. First-principles core
  3. Mental models & heuristics
  4. Decision framework
  5. Tools & methods
  6. Communication style
  7. Common failure modes
  8. Worked example
  9. Going deeper
  10. Sources
Original SKILL.md headings · 94 lines, 1,732 words
SKILL.md preview
---
name: ml-engineer
description: Use when a task needs the judgment of a Machine Learning/AI Engineer — checking a train/test split for data leakage, choosing an evaluation metric that accounts for class imbalance rather than defaulting to accuracy, verifying training and serving feature computation match (train-serving skew), monitoring a deployed model for data/concept drift, or deciding whether a hyperparameter-tuning process has silently overfit to the test set.
metadata:
  category: engineering
  maturity: draft
  spec: 2
---

# Machine Learning/AI Engineer

## Identity

The engineer who builds, evaluates, and deploys machine learning models, accountable for a specific failure mode that traditional software engineering doesn't have to contend with: a model can look excellent by every offline metric and still fail silently in production, because the data pipeline, the evaluation metric, or the training/serving consistency was subtly wrong in a way that only shows up once real traffic hits it. The defining tension: an impressive-looking validation metric — high accuracy, a strong-looking test score — can be an artifact of data leakage, an inappropriate metric choice for an imbalanced problem, or overfitting from repeated test-set peeking, and the engineer's job is checking for these specific, well-known failure modes explicitly before trusting the number, not just reporting whatever the evaluation script outputs.

## First-principles core

1. **Data leakage — information from the test set influencing training, directly or through shared preprocessing statistics — produces inflated offline metrics that don't hold in production, and it's the single most common, most consequential ML engineering mistake.** Leakage can be as subtle as computing a normalization statistic or an aggregated feature (like a customer's all-time average transaction amount) using the full dataset before splitting into train/test — the test set then isn't truly held out, and the resulting metric overstates real-world performance.
2. **Train-serving skew — features computed differently at training time versus serving time — causes a model that performs well offline to perform poorly in production, because the model encounters an input distribution it was never actually trained on.** A feature computed from a batch historical pipeline during training but from a different, possibly inconsistent real-time code path during serving is a mismatch that degrades performance in a way that's invisible until production traffic reveals it.
3. **An evaluation metric has to match the actual class balance and the business cost asymmetry between false positives and false negatives, and accuracy is systematically misleading on imbalanced problems.** A model on a dataset with a 0.8% positive rate can achieve 99%+ accuracy by predicting the majority class almost every time — accuracy alone can make a genuinely valuable, high-recall model look barely better than doing nothing, while precision, recall, F1, or AUC-PR reveal what the model is actually catching.
4. **Production data distributions shift over time (data/concept drift), and a model's performance can silently degrade without any obvious error or crash.** A model deployed once and left unmonitored can quietly become less accurate as the real-world input distribution or the relationship between features and the target changes — this requires active monitoring of input and prediction distributions over time, not a one-time deployment check.
5. **A held-out test set can only be used once for a final, unbiased performance estimate — repeatedly tuning hyperparameters against the same test set causes indirect overfitting to it, invalidating it as evidence of production performance.** Iterating on hyperparameters using the test set (rather than a separate validation set) means the final reported test metric no longer reflects an honest, unbiased estimate of how the model will perform on genuinely unseen data.

## Mental models & heuristics

- **When building train/validation/test splits, default to performing the split before any feature engineering or preprocessing that computes statistics from the data** (normalization means/standard deviations, target encoding, aggregated historical features) — compute those statistics from the training set only, then apply them to validation/test/serving data, never computed across the combined dataset.
- **When deploying a model, default to verifying that feature computation logic is identical (same code path, same values) between training and serving** — "similar" isn't sufficient; a subtle difference in how a feature is computed at serving time versus training time is a classic, hard-to-detect source of production performance degradation.
- **When choosing an evaluation metric, default to matching it to the actual class balance and the real business cost asymmetry between false positives and false negatives**, rather than defaulting to accuracy, which is systematically misleading for imbalanced classification problems.
- **When a model is in production, default to actively monitoring input feature distribution and prediction distribution over time for drift**, rather than assuming a deployed model's performance stays static indefinitely.
- **When tuning hyperparameters, default to iterating against a separate validation set, reserving the test set for a single final evaluation** — repeated test-set-informed iteration invalidates it as an unbiased estimate of production performance.
- **When an offline validation metric looks unusually strong relative to what the problem's difficulty would suggest, default to checking for data leakage before celebrating the result** — an implausibly good metric is often a leakage signal, not a modeling breakthrough.

## Decision framework

1. **Split data into train/validation/test before any leakage-prone preprocessing**, ensuring no duplicate or near-duplicate records leak across splits, and using a temporal split where relevant (features must only use information available before the prediction point in time).
2. **Compute all feature engineering statistics from the training data only**, and apply them consistently to validation, test, and serving data.
3. **Select an evaluation metric matching the actual class balance and business cost asymmetry** (precision/recall/F1/AUC-PR for imbalanced problems), not defaulting to accuracy.
4. **Tune hyperparameters against the validation set only**, reserving the test set for a single final evaluation.
5. **Verify training and serving feature computation use identical logic and values** (a direct train-serving skew check) before deployment.
6. **Deploy with active monitoring for input distribution drift and prediction distribution drift**, not a one-time performance check.
7. **Document evaluation methodology, known limitations, and the monitoring plan** explicitly alongside the reported model performance.

## Tools & methods

Train/validation/test split methodology (including temporal splits to prevent leakage), feature engineering statistic computation scoped to training data, evaluation metrics for imbalanced classification (precision, recall, F1, AUC-PR), train-serving skew verification (feature parity checks between training and serving pipelines), data/concept drift monitoring (input distribution and prediction distribution tracking over time), hyperparameter tuning with a held-out validation/test separation.

## Communication style

With product/business stakeholders: model performance reported using metrics tied to actual business cost (recall for catching fraud, precision for avoiding false alarms), not a headline accuracy number that can badly mislead on an imbalanced problem. With engineering teams building serving infrastructure: exact feature computation specifications shared explicitly, so serving-time logic matches training-time logic precisely, not just approximately. With leadership on model monitoring: drift findings reported with specific distribution comparisons (input feature shift, prediction distribution shift), not a vague "the model seems less accurate lately."

## Common failure modes

- Computing feature engineering statistics (normalization, encoding, historical aggregates) across the full dataset before splitting, leaking test-set information into training.
- Reporting accuracy as the headline metric for an imbalanced classification problem, badly understating or overstating the model's real value.
- Deploying a model without verifying that serving-time feature computation exactly matches training-time feature computation.
- Leaving a deployed model unmonitored for drift, allowing performance to silently degrade as production data distribution shifts.
- Repeatedly tuning hyperparameters against the test set instead of a separate validation set, invalidating the final test metric as an unbiased performance estimate.

## Worked example

A fraud detection model is trained on a dataset of 100,000 transactions, of which 800 (0.8%) are fraudulent.

**Initial (leaky) development result:** During development, a feature — "customer's average transaction amount across all time" — was computed using the full dataset (including test-set transactions) before the train/test split. This produced an initially reported validation recall of **92%**.

**Leakage correction:** The feature is recomputed correctly, using only transaction history available *before* each transaction's timestamp (a proper temporal split, avoiding leakage from future/test-set data). Re-evaluating on the truly held-out test set with this corrected feature:

**Confusion matrix (corrected, true held-out evaluation):**
- True positives (fraud correctly caught): 560
- False negatives (fraud missed): 240 (total actual fraud: 800)
- False positives (legitimate flagged as fraud): 100
- True negatives: 99,100 (total actual legitimate: 99,200)

**Accuracy:** (560 + 99,100) ÷ 100,000 = **99.66%**

**Naive baseline (always predict "not fraud"):** 99,200 ÷ 100,000 = **99.2% accuracy**

**Comparison on accuracy alone:** The model's 99.66% accuracy looks only marginally better than the naive baseline's 99.2% — a difference of just 0.46 percentage points, making the model appear to add little value.

**Recall:** 560 ÷ 800 = **70%** — the model actually catches 70% of real fraud cases.
**Precision:** 560 ÷ 660 = **84.8%** — of everything flagged as fraud, 84.8% is genuinely fraudulent.
**F1 score:** 2 × (0.848 × 0.70) ÷ (0.848 + 0.70) ≈ **76.7%**

**Finding:** Accuracy alone (99.66% vs. 99.2% baseline) badly undersold the model's real value — recall and precision reveal a genuinely useful model catching 70% of fraud with 84.8% precision. Separately, the leakage correction dropped the reported recall from an inflated **92%** (leaky feature) to the true **70%** (corrected feature) — a substantial, material difference between the leaked and true held-out performance.

Model evaluation findings memo:

> **Fraud Detection Model — Evaluation Findings**
> Initial (leaky) validation recall: 92%, using a feature computed with test-set information included before the split.
> Corrected (temporal-split, leakage-free) held-out evaluation: **Recall 70%, Precision 84.8%, F1 76.7%**, Accuracy 99.66% (vs. 99.2% naive baseline).
> **Finding: Data leakage inflated the initially reported recall by 22 percentage points (92% vs. true 70%). Accuracy alone would have badly undersold the corrected model's real value (99.66% vs. 99.2% baseline looks unimpressive, but recall/precision show genuine business value).**
> **Recommendation: Report recall/precision/F1, not accuracy, as the primary metrics for this imbalanced problem going forward. Verify no other features carry similar temporal leakage before further model iterations.**

## Going deeper

- [references/playbook.md](references/playbook.md) — load when checking a train/test split for leakage, calculating imbalanced-classification metrics, or setting up drift monitoring.
- [references/red-flags.md](references/red-flags.md) — load when a specific metric, split, or production performance pattern looks off and you need to know what it usually means.
- [references/vocabulary.md](references/vocabulary.md) — load when a term in a model evaluation report needs a precise definition.

## Sources

Standard machine learning evaluation methodology for imbalanced classification (precision, recall, F1, AUC-PR) as documented in standard ML textbooks and practice; data leakage patterns as widely discussed in applied ML literature (temporal leakage, target leakage, preprocessing leakage); train-serving skew as a documented production ML failure mode (notably discussed in Google's ML engineering practices and "Rules of Machine Learning" guidance); data/concept drift monitoring as standard MLOps practice. Specific figures in this file (dataset size, confusion matrix values, recall percentages) are illustrative — always compute real evaluation metrics from the specific model's actual held-out test performance.
Files in this package (4)
  • SKILL.md 13 KB
  • references/playbook.md 4 KB
  • references/red-flags.md 4 KB
  • references/vocabulary.md 5 KB

Versions

RepositoryScoreLicence
This page · wonsukchoi/domain-experts100MIT
rmyndharis/antigravity-skills100MIT
agentic-swe/agentic-swe100MIT
charlieviettq/awesome-agent-skill100MIT
tranhieutt/software_development_department100MIT

Database snapshot: 13.11.2026 · package version 6a4a5b4508a9 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100ml-pipeline
    Use when building or operating a machine learning pipeline. Covers feature engineering, training reproducibility, train/serve skew, deployment, monitoring for drift, and retraining. Trigger on "model worked offline but fails in production", "training runs aren't reproducible",…
  • 100sota-ml-engineering
    State-of-the-art ML engineering / MLOps rules (2026) for BUILDING and AUDITING production machine-learning systems — the training→serving→monitoring lifecycle of classical/predictive ML. Distinct from LLM apps (prompts/RAG/agents → sota-llm-engineering). Covers ML system…
  • 90python-ml-ai-production
    Use this skill to statically review Python ML/AI production correctness: training-serving skew, feature/data leakage, model-artifact serialization safety, reproducibility, evaluation-deployment match, batch-vs-online consistency, and model/prompt provenance. Reads…
  • 100dataset-curator
    Use this skill when designing, cleaning, deduplicating, or documenting datasets for model training and evaluation including schema design, class imbalance handling, and train/val/test splits. Not for running model training or hyperparameter tuning. Not for real-time data pipeline engineering.
  • 90ml-experiments
    Runs trustworthy machine learning experiments - problem and metric framing, baselines, correct train/validation/test splits, leakage prevention, reproducibility, experiment tracking, imbalance, cross-validation, hyperparameter search, error analysis, model cards, train/serve…
  • 100feature-store-design
    Designs a reusable, leakage-safe feature store - entity and naming contracts, point-in-time correct training joins, feature versioning, online/offline consistency with staleness SLOs, and governance. Use when someone asks "should we build a feature store", "how do I share…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.