debug-rollouts
/debug-rollouts
Diagnoses or fixes Osmosis rollout startup, trajectory, grading, and timeout failures. Use for broken rollouts or unexpected rewards; model-quality comparisons belong to run-evals.
- Author
- Osmosis-AI ↗
- Repository
- osmosis-plugin ↗
- Licence
- MIT
- Last repository change
- 11.11.2026
Assessment and security
- Files in this package
- 3
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 2
No signals detected. Repository changed on 11.11.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Diagnoses or fixes Osmosis rollout startup, trajectory, grading, and timeout failures. Use for broken rollouts or unexpected rewards; model-quality comparisons belong to run-evals.
Original author description
Diagnoses or fixes Osmosis rollout startup, trajectory, grading, and timeout failures. Use for broken rollouts or unexpected rewards; model-quality comparisons belong to run-evals.
How it works
SKILL.md has no level-two headings · 19 lines, 541 words
SKILL.md preview
--- name: debug-rollouts description: Diagnoses or fixes Osmosis rollout startup, trajectory, grading, and timeout failures. Use for broken rollouts or unexpected rewards; model-quality comparisons belong to run-evals. --- # Diagnose a rollout Use `osmosis --help` to find commands and the installed subcommand's `--help` to check its flags. Resolve uncertainty with `osmosis-docs` or the [docs index](https://docs.osmosis.ai/llms.txt), using public terms only. Prefer current docs to bundled references; verify managed limits separately. Report mismatches and check [PyPI](https://pypi.org/project/osmosis-ai/) before recommending an upgrade. Upgrade or send docs feedback only when requested. Start with the reported error and one affected row or run. Briefly inspect existing evidence to locate the failed phase. For reproducible rollout-code or runtime problems, prefer a bounded `osmosis eval run` on that input as the main diagnostic loop; it exercises the actual entrypoint and gathers server, trajectory, and grading evidence together. Follow [local reproduction and evidence](references/local-eval.md) when reproducing or reading local results. Continue directly when the requested debug scope already authorizes the model, backend, rows, and attempts; honor logs-only requests and missing execution prerequisites. Use `osmosis --json eval info <name>`, `eval logs <name>`, or `train logs <name>` for managed evidence, with an explicit workspace outside the clone. Hosted-only Git sync, secret injection, or scheduling failures need that environment's evidence; a successful local eval cannot validate those paths. When existing evidence already establishes a cause, avoid an unchanged rerun and verify the fix when authorized. | Symptom | Evidence that distinguishes causes | | --- | --- | | Layout, import or startup failure | Inspect the config's entrypoint under `rollouts/<name>/`, the rollout's own dependencies and server port. If an SDK import or call fails, check the installed module or signature in the rollout environment. Use `osmosis --json doctor` for workspace problems. For managed runs compare the requested Git revision and the synced revision; a local edit cannot repair a running remote server. | | Missing data or label | Inspect the failing local row directly, or use `dataset preview` for platform data. Check whether the workflow consumed `ground_truth` (alias `label`) or task metadata. A metadata-only row need not have a prompt or label; an upload validator's row-count error does not invalidate a one-row local smoke test. | | No trajectory or reward | For a Python workflow, inspect the returned history or registered sample source, policy calls through the active rollout context, and the async grader's `ctx.set_reward(...)`. For native Harbor, inspect the selected task, native agent, and verifier output. Its `tests/test.sh` takes precedence over a Python grader. An oracle produces no model trajectory. | | Graded zero vs. missing grade | Trace one input through the policy output and grader/verifier. Distinguish a legitimate task failure from malformed output, an intentional skip, timeout, or unavailable model/sandbox. For hangs, check blocking calls in async code and sandbox/model-bridge connectivity before raising timeouts. | | Local publish or managed preflight failure | Check terminal status, workspace identity, named secrets (never values), and the exact error. Local execution, result upload, and fetching managed source from Git are different operations. | For a diagnosis request, return the failing phase, affected input, command/run name and evidence paths, likely cause, and the smallest check that could disprove it. If a fix is requested, verify it on the same affected input and report the before/after evidence; a code edit alone is not a verified fix. `train submit` and `eval submit` can start paid runs; neither is a diagnostic dry run.
Files in this package (3)
- SKILL.md 4 KB
- agents/openai.yaml 1 KB
- references/local-eval.md 3 KB
Versions
The database has no other skills named debug-rollouts.
Database snapshot: 13.11.2026 · package version 247134baa960 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 95create-rollouts
Creates or adapts an Osmosis rollout from an agent or task, including its grader or Harbor verifier and authorized local validation. Use for new implementation or integration work. - 91plan-eval
Plans an Osmosis evaluation when the task, data, scoring, or comparison is undecided. Use for experiment design and budget; defined model comparisons belong to run-evals. - 90agent-evals-and-observability
Design, run, review, or release framework- and vendor-neutral evaluations and observability for AI agents. Use when defining agent evals, datasets, graders, trajectory review, regression analysis, release gates, production traces, or privacy-aware telemetry. Covers task and… - 100evals
Designing and maintaining evaluations for LLM prompts, agents, and Agent Skills: success criteria, test-set composition, code/LLM-judge/human graders, trajectory and tool-use grading, trigger evals, regression suites, eval saturation. WHEN: \"eval\", \"evaluation suite\",…