Przejdź do treści
Biblioteka skilli
AI i agenci · Analiza

ladder

/ladder

Daje deweloperowi szybkie porównanie modelu lokalnego z modelem frontier jeszcze przed własnymi śladami: co potrafi model lokalny i czy mały model wystarczy.

Licencja
MIT
Ostatnia zmiana repozytorium
27.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
12
Skrypty
2
Komentarze HTML
0
Adresy URL w SKILL.md
2
Przestarzałość

Brak sygnałów. Repozytorium zmienione 27.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co sprawdzić przed instalacją
  • 2 skrypty w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.

Skrypty: env/world.py, serve.py

Co robi i kiedy

Daje deweloperowi szybkie porównanie modelu lokalnego z modelem frontier jeszcze przed własnymi śladami: co potrafi model lokalny i czy mały model wystarczy.

Wyzwalacze z opisu

what can a local model dois a small model good enoughcompare a local model to a frontier modelthe onboarding climb
Oryginalny opis (EN)

Use to give a developer an immediate local-vs-frontier model comparison before they have their own traces — "what can a local model do", "is a small model good enough", "compare a local model to a frontier model", "the onboarding climb". For comparing many models on a user's own eval, use compare-model-sweep; to serve a local model, use run-local-model-lab.

Jak działa

  1. When to use
  2. Run it
  3. What you see
  4. What you can actually run ($0 vs billed)
  5. Drive it headlessly
  6. Safety Gates
  7. What's here
Sekcje z nagłówków SKILL.md, w oryginale · 121 linii, 940 słów
Podgląd SKILL.md
---
name: ladder
description: Use to give a developer an immediate local-vs-frontier model comparison before they have their own traces — "what can a local model do", "is a small model good enough", "compare a local model to a frontier model", "the onboarding climb". For comparing many models on a user's own eval, use compare-model-sweep; to serve a local model, use run-local-model-lab.
metadata:
  understudy:
    mode: interactive
    safety: local-first
    cli_required: false
---

# Ladder — the onboarding "climb"

A small, self-contained **local** web UI for watching and comparing LLMs on the
same task. It is the no-data front door: a developer launches it and watches a
small local model and a frontier model attempt the same task side by side —
live reasoning, real tool calls, strict scoring — without needing any of their
own traces first. As the task gets harder, the small local model keeps pace and
then drops off where the frontier model carries on. That contrast is the hook
into the rest of the improvement loop.

## When to use

- A new user asks "what can a local model actually do?" / "is a small model good
  enough?" / "show me local vs frontier."
- During or right after `onboard`, to make the local-vs-frontier difference
  concrete before steering them toward their own workload.
- Any time a visceral, zero-setup-data demo of agentic behavior + scoring helps.

For the user's **own** eval across **many** candidate models, hand off to
`compare-model-sweep`. To serve a local model against the user's **real**
workload, hand off to `run-local-model-lab`.

## Run it

**Prerequisites** (otherwise-silent assumptions):

- Apple Silicon (arm64) Mac with **`uv`** on `PATH` — the local lane needs a
  current mlx stack (system `mlx_lm` is often too old to load the model).
- The local model cached at `~/.understudy/models/gemma-4-e2b-it-qat-mlx-vlm-understudy` —
  the `gemma-4-e2b` id resolves to **that** directory, not a dir literally named
  `gemma-4-e2b`. If it's missing, pull it with the `manage-local-models` skill
  (`orchestra models pull gemma-4-e2b-it-qat-mlx-vlm-understudy`); the server
  prints an actionable error if it's absent.
- **Remote lanes only:** run `orchestra login` once so `orchestra run` can
  inject the gateway key and org id. Without it a remote lane returns the literal
  notice `[gateway not configured …]` rather than erroring — don't mistake that
  for a real run.

Run from the repo root (the path below is relative):

```sh
orchestra run -- uv run --with mlx-vlm --with mlx-lm python skills/ladder/serve.py
```

First invocation builds the mlx stack from PyPI (network, a few minutes) and the
model cold-loads (~60s) — that pause is normal, not a hang. Then open
<http://localhost:8011/ladder.climb.html>, or drive it headlessly (below).

This is a developer-run demo/onboarding tool, **not** a headless production
service — though the SSE endpoint is browserless-friendly for an agent.

## What you see

- **Easy / medium rungs** — single-shot classify tasks; watch the model think,
  answer, and get a Pass/Fail against the gold label.
- **Hard rung** — a real tool-calling agent loop against the synthetic Larkfield
  world (find the account, read the policy, apply the latest discount, update the
  plan, email the right teams, `finish`), scored by final state.
- **VS mode** — two models race the same task side by side. Local-vs-local
  serializes on the one GPU; local-vs-gateway runs concurrently.

Remote runs go through the Orchestra gateway and are **billed**. The picker
marks every gateway-backed lane and every such run is disclosed.

## What you can actually run ($0 vs billed)

- **$0, fully local — the "climb".** Run `gemma-4-e2b` across all three rungs and
  watch where it breaks: it passes the easy/medium classify tasks and typically
  fails the hard tool-calling task. That difficulty climb is the free demo and a
  real signal on its own, entirely on your machine.
- **A side-by-side model-vs-model comparison usually costs money.** The "compare
  two" (VS) button pairs the local model against a gateway-backed remote model
  by default — so triggering VS bills a gateway run. No *second* free model ships
  enabled (the local LFM lane is coded but commented out in `serve.py`). So "$0"
  and "two models side by side" are not both available out of the box: take the
  free climb, or accept the gateway cost for the race.

## Drive it headlessly

No browser required — the page is just a client of one SSE endpoint:

```sh
curl -N 'http://localhost:8011/run?task=sort-email&model=gemma-4-e2b'
```

The stream ends with a `done` event carrying the result (`correct`, `response`,
`tok_s` for the classify rungs; `strict` / `dense` / `passes` for the hard rung).
`GET /tasks` lists task ids (`sort-email`, `match-search`, `hard.sla_route`); `GET
/models` lists the lanes.

## Safety Gates

- **Local-first, $0 on the local lane.** The mlx model runs on the developer's
  machine, the server binds `127.0.0.1` only, and nothing is uploaded.
- **No cross-site access.** The server sends no CORS header and answers 403 to
  requests whose `Host` is not `127.0.0.1`/`localhost` or that a browser marks
  cross-site (`Sec-Fetch-Site`/`Origin`), so a web page open in your browser
  cannot start billed runs. The viewer and `curl` are unaffected.
- **Gateway runs are billed.** Remote lanes go through the Orchestra gateway and
  cost money. The picker marks them and every such run is disclosed in the UI —
  do not route to one without the developer understanding it bills.
- **Synthetic data only.** The "Larkfield" world is invented; no real customer or
  workload data is involved.
- **No silent downloads.** The server only *loads* a cached local model from
  `~/.understudy/models`; if it is missing it says so — pull it with
  `manage-local-models`, never auto-download weights.

## What's here

Four stdlib, self-contained files: `serve.py` (HTTP + SSE server + the agent
loop), `env/world.py` (the synthetic Larkfield world + scoring),
`fixtures/hard/tool_tasks.jsonl` (the hard tasks as data — add a row and it's
live and scored), and `viewer/ladder.climb.html` (the UI).

See [`reference.md`](reference.md) for the architecture, the model lanes +
tool-call dialects, the Larkfield world + scoring contract, **how to add your own
tasks** (a new hard tool task is a single JSONL row — schema, assertion types, and
tool list are documented there), and the running gotchas (uv-vs-system mlx, the
one-GPU thread funnel, `UNDERSTUDY_MODEL_HOME`).
Pliki w paczce (12)
  • .gitignore 1 KB
  • SKILL.md 6 KB
  • reference.md 16 KB
  • serve.py 48 KB
  • verifiers-export.md 6 KB
  • env/world.py 27 KB
  • fixtures/classify_tasks.jsonl 1 KB
  • fixtures/hard/tool_tasks.jsonl 11 KB
  • viewer/ladder.climb.html 42 KB
  • viewer/brand/OFL.txt 4 KB
  • viewer/brand/inter-latin-variable.woff2 47 KB
  • viewer/brand/orchestra-logo-forest.svg 16 KB

Wersje

W bazie nie ma innych skilli o nazwie ladder.

Migawka bazy z 13.11.2026 · wersja paczki c1fd8ce763d7 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100compare-model-sweep
    Porównuje modele lokalne, bramkowe i frontier na tym samym evalu, pokazując obok siebie jakość, opóźnienia, koszt i niezawodność.
  • 90run-local-model-lab
    Uruchamia lokalny model na Macu z Apple Silicon na prawdziwym obciążeniu, ocenia go i pomaga zdecydować, czy wystarczy zamiast płatnego modelu hostowanego.
  • 100agent-eval-harness
    Testuje, czy agent AI, prompt lub skill naprawdę działa: buduje mały zestaw ewaluacyjny, ocenia przebiegi i wyłapuje regresje po zmianie promptu lub modelu.
  • 100feature-eval-author
    Buduje ewaluację behawioralną, która uruchamia mały model przez pi SDK i mierzy, czy rozszerzenie pi faktycznie zmienia zachowanie modelu: czyta wstrzyknięty stan, działa według dyrektywy lub wybiera właściwą pamięć.
  • 100jev-frontier-work
    Wybiera płatny model graniczny do trudnego zadania i pilnuje przebiegu delegowanej pracy, przerywając tylko przy decyzji.
  • 100eval-compare
    Porównuje wyniki ewaluacji wielu modeli lub uruchomień i tworzy zakładkowy raport HTML z kartami modeli, tabelami jakości i kosztu, rozbiciem na przypadki i osadzonymi raportami źródłowymi.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.