ml-lifecycle
/ml-lifecycle
Takes a model to production without fooling anyone: measure a non-learned baseline first, split by time, hunt leakage, compute features once for training and serving, evaluate the worst slice, roll out behind a shadow or canary, monitor input and prediction drift, and pin a rollback to the previous version. Use when shipping, retraining, or debugging a model whose live quality has degraded.
- Author
- khuynh22 ↗
- Repository
- agent-dev-team ↗
- Licence
- MIT
- Last repository change
- 26.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 26.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Takes a model to production without fooling anyone: measure a non-learned baseline first, split by time, hunt leakage, compute features once for training and serving, evaluate the worst slice, roll out behind a shadow or canary, monitor input and prediction drift, and pin a rollback to the previous version. Use when shipping, retraining, or debugging a model whose live quality has degraded.
Original author description
Takes a model to production without fooling anyone: measure a non-learned baseline first, split by time, hunt leakage, compute features once for training and serving, evaluate the worst slice, roll out behind a shadow or canary, monitor input and prediction drift, and pin a rollback to the previous version. Use when shipping, retraining, or debugging a model whose live quality has degraded.
How it works
- Process
- Output
- Verification
- Red flags
SKILL.md preview
--- name: ml-lifecycle description: Takes a model to production without fooling anyone: measure a non-learned baseline first, split by time, hunt leakage, compute features once for training and serving, evaluate the worst slice, roll out behind a shadow or canary, monitor input and prediction drift, and pin a rollback to the previous version. Use when shipping, retraining, or debugging a model whose live quality has degraded. license: MIT metadata: phase: build owners: [ml-engineer] version: "0.1.0" --- # ML Lifecycle A model in a notebook is a claim. A model in production with a baseline, a monitor, and a rollback is a system. Offline accuracy is not evidence of online value, and the first suspiciously good result is usually leakage rather than skill. Standing reference: `references/data-quality-checklist.md`. ## Process 1. **Measure a non-learned baseline first.** Last observed value, majority class, a three-line heuristic. Every subsequent number is a delta against it. If the model is within noise of the baseline, the correct deliverable is the baseline. 2. **Split by time when the data has time in it,** and hold out a period the model never sees. A random split on temporal data trains on the future and produces a number that does not survive next week. 3. **Hunt leakage before celebrating.** Any feature computed after the label exists, or read from a table that is updated retroactively, is leakage. Audit each feature by asking what its value would have been at prediction time. A first result far above the baseline is an alarm, not a win. 4. **Compute features once.** Training and serving read the same code path. Otherwise you ship a model that scores differently in production than in the notebook, and the investigation costs a week. If the paths must differ, measure the skew as a number. 5. **Evaluate slices, not just the aggregate.** By segment, cohort, volume tier, and on the tail. Report the worst slice in the headline, because the worst slice is what generates complaints and it is invisible in the average. 6. **Tie the offline metric to a product metric.** State the link explicitly: this ranking metric maps to that conversion outcome because of this mechanism. An offline win with no stated link is a number, not a reason to ship. 7. **Roll out behind shadow, then canary.** Score live traffic without acting on it, compare against the incumbent on the same requests, then take a percentage with an abort condition written down in advance. 8. **Monitor inputs and outputs after launch.** Feature distribution drift, prediction distribution drift, and the product metric. A degrading model raises no exception; the only signal is a monitor somebody built. 9. **Write the rollback before the launch.** Which version, how it is pinned, how long it takes, and what triggers it. Retraining is not a rollback: it takes hours and might not help. ## Output ```markdown ## Model: <name> v<n> - **Baseline:** <what> — <metric> = <n> - **Model:** <metric> = <n> (delta <n>), worst slice <name> = <n> (n=<n>) - **Split:** temporal, train <range>, holdout <range> - **Leakage audit:** <features checked, what was removed and why> - **Train/serve parity:** <shared path | measured skew> - **Product link:** <offline metric> to <product metric> via <mechanism> - **Rollout:** shadow <duration>, canary <n>%, abort if <condition> - **Monitors:** input drift <how>, prediction drift <how>, product metric <how> - **Rollback:** to v<n-1> via <mechanism>, takes <time>, tested on <date> ``` ## Verification - Baseline and model numbers come from the same holdout; both quoted. - Every holdout row is later in time than every training row. - Shadow and production scores were compared on the same requests, and the skew is a number. - Drift monitors were run against deliberately shifted input and fired. - The rollback was executed once outside production and timed. ## Red flags | Thought | Reality | |---------|---------| | "0.97 on the first attempt" | Leakage. Find it before telling anyone the number. | | "The baseline is too trivial to measure" | Then it is cheap, and it is the bar your model has to clear. | | "Shuffle split is standard" | Not on temporal data. You trained on the future. | | "Serving recomputes features slightly differently" | That is skew, and it is the most common silent production failure. | | "Accuracy is 94%" | On which slice? The worst one is the one people complain about. | | "It is doing fine in production" | Measured how? Models degrade without raising an error. | | "We can retrain if it drifts" | Retraining is not a rollback. What serves traffic meanwhile? | | "The offline metric improved" | Linked to which product outcome, by what mechanism? |
Versions
The database has no other skills named ml-lifecycle.
Database snapshot: 13.11.2026 · package version 762e11c33509 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90snowflake-data-science-ml
Use this skill to review the ML lifecycle in Snowflake for reproducibility and governability: feature engineering and leakage, point-in-time correctness and training/serving skew, training reproducibility, the model registry and versioning, batch and continuous inference, drift… - 100deploy-canary
Plan and execute canary releases for model updates — traffic splitting, rollback triggers, success metrics. Use when asked to "canary the new model", "roll out this model safely", or "set up a traffic split for a model update". - 100sota-ml-engineering
State-of-the-art ML engineering / MLOps rules (2026) for BUILDING and AUDITING production machine-learning systems — the training→serving→monitoring lifecycle of classical/predictive ML. Distinct from LLM apps (prompts/RAG/agents → sota-llm-engineering). Covers ML system… - 90release-to-production
Plan the release of an AI change (prompt, model, KB) through offline eval, shadow, canary and A/B if the sample allows. Use for "can we ship this?" or a provider model update. - 100model-serving-and-scaling
Guides deploying ML/LLM models for online inference with autoscaling, latency budgets, batching, canary/shadow rollout, and cost control. Use when the user asks to "serve a model", "deploy a model for inference", set up autoscaling for a model endpoint, reduce inference latency…