Przejdź do treści
Biblioteka skilli
Programowanie · Budowanie i kod

specialized-inference-engine

/specialized-inference-engine

Buduje i optymalizuje natywny silnik wnioskowania dla jednego modelu, porównując wyniki z referencją i mierząc szybkość.

Repozytorium
enginesmith ↗
Licencja
Apache-2.0
Ostatnia zmiana repozytorium
09.11.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
57
Skrypty
9
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 09.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

Co sprawdzić przed instalacją
  • 9 skryptów w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.

Skrypty: backends/nki-cpp/example/build.sh, backends/nki-cpp/example/build_neff.py, backends/nki-cpp/example/kernels.py, backends/nki-cpp/example/kernels_main.py, scripts/ab.py, scripts/bytes_per_token.py, scripts/engine_client.py, scripts/gpu_lock.py, scripts/same.py

Co robi i kiedy

Buduje i optymalizuje natywny silnik wnioskowania dla jednego modelu, porównując wyniki z referencją i mierząc szybkość.

Oryginalny opis (EN)

Build and optimize a specialized native inference engine for one fixed model checkpoint, matched token for token against a reference implementation and faster than it. Covers onboarding a model (model card, bytes and FLOPs per token, speed floor), bringing up a correct baseline, bitwise-safe optimization, interleaved timing, profiling, kernel fusion and packaging. Backend modules hold the platform techniques: Metal + Rust for Apple Silicon (M5 family), HIP + Rust for AMD Instinct MI325X (gfx942), and NKI + C++ for AWS Trainium2. Use for any session that writes, verifies or speeds up engine cod

Jak działa

  1. The rules that matter most
  2. Workflow
  3. Where the time goes (the general shape)
  4. Backends
  5. Files
Sekcje z nagłówków SKILL.md, w oryginale · 94 linii, 1 105 słów
Podgląd SKILL.md
---
name: specialized-inference-engine
description: Build and optimize a specialized native inference engine for one fixed model checkpoint, matched token for token against a reference implementation and faster than it. Covers onboarding a model (model card, bytes and FLOPs per token, speed floor), bringing up a correct baseline, bitwise-safe optimization, interleaved timing, profiling, kernel fusion and packaging. Backend modules hold the platform techniques: Metal + Rust for Apple Silicon (M5 family), HIP + Rust for AMD Instinct MI325X (gfx942), and NKI + C++ for AWS Trainium2. Use for any session that writes, verifies or speeds up engine code.
---

# Specialized inference engine

You are building an inference engine for **one model checkpoint on one kind of machine**, not a framework. That
narrowness is the whole advantage: every shape is known at compile time, every kernel can be fused for this graph,
and the decode loop can be shaped around this model's dispatch chain. A general framework cannot do any of that.

The task contract you were given (engine protocol, correctness gates, budgets, sandbox, what you may run) is
provided separately by the framework. **It always wins over this skill.** This skill is about how to build and
speed up the engine; it never restates the contract.

## The rules that matter most

1. **Correct before fast.** No tuning until the whole model matches the reference (phase 1 below). A fast engine
   that fails the correctness gate is worth nothing, and debugging numerics inside an optimized engine is slow.
2. **Every optimization is bitwise identical to its parent.** Once a correct baseline exists, a change that alters
   any output token or logit is a correctness change, not an optimization. Keep the same arithmetic expressions,
   the same rounding points and the same reduction orders, and verify with a token-for-token comparison over many
   prompts and request shapes before timing anything. See `references/correctness.md`.
3. **Measure, never guess, and measure the right thing.** Time two builds interleaved (A, B, B, A) on fresh prompts,
   compare ratios rather than absolute numbers, and know your noise floor. Thermal throttling and other GPU users
   move absolute numbers by several percent. Run `enginesmith dev-eval` and every timing tool in the foreground and
   read its output when it returns: a backgrounded run polled with `pgrep -f` matches the polling shell's own command
   line and never ends. See `references/measurement.md`.
4. **Know the floor.** Decode time per token is bounded below by bytes read per token divided by the measured
   streaming bandwidth. Compute it on day one (`references/model-card.md`) and judge every kernel against it.
5. **One mechanism per change, with its measured effect.** Record what you changed, how you verified it and what
   it bought, including the attempts that were neutral or slower. Negative results with numbers are as valuable
   as wins: they stop the next session from repeating them.
6. **Label every number** as measured (with where and how) or estimated. A missing measurement is "not measured",
   never zero.

## Workflow

The six phases, their entry and exit criteria and the evidence each must leave behind are in
`references/workflow.md`. In short:

| phase | goal | done when |
|---|---|---|
| 0. Onboard the model | fill the model card: graph, tensors, formats, bytes and FLOPs per token, reference pinned | the decode floor and prefill floor are computed |
| 1. Correct baseline | port the nearest existing kernels, match the reference's partition and rounding | teacher-forced and free-running checks pass at every length |
| 2. Measurement harness | bitwise A/B tool, interleaved timing tool, GPU lock, noise floor | an A/A run shows the noise floor |
| 3. Find the time | dispatch map, per-kernel GPU time, compare with the floor | a ranked list of opportunities with estimates |
| 4. Optimize | one mechanism at a time, bitwise-checked and timed | measured speedup, tokens identical |
| 5. Package | self-contained build, reproducible, no host paths | a fresh clone builds and runs |

Most sessions resume in phase 4. Read the parent engine's notes and the last session's results first; do not
redo work that was measured and rejected.

## Where the time goes (the general shape)

For a batch-1 decoder on a GPU with unified memory:

- **Decode** is memory-bound: every token streams the active weights once. A good engine runs its weight-streaming
  kernels near the machine's streaming bandwidth and then spends the rest of its effort on **fixed per-dispatch
  costs**, because a decode step is hundreds of small, dependent dispatches. Removing dispatches by fusion is
  usually the largest remaining lever once the GEMVs are near the bandwidth limit.
- **Prefill** is compute-bound for long prompts: use the GPU's matrix units, tile for the shapes you have, and
  share K/V tiles between the query heads of a KV head.
- **Long-context decode** becomes bound by the KV-cache read, not by weights; attention needs a split (two-pass)
  form past a few thousand keys.
- **The host** must never sit in the token loop: keep tokens on the GPU between steps, encode steps ahead, and wait
  once per request.

The backend module gives the measured numbers for its platform.

## Backends

Platform techniques live in `backends/<name>/`. Read the backend's `README.md` first; it lists its files and the
platform's hard rules.

| backend | status | for |
|---|---|---|
| `backends/metal-rust/` | written, measured on an Apple M5 Max | Apple Silicon, Metal Shading Language kernels, Rust host code |
| `backends/hip-rust/` | written; platform micro-benchmarks and the SGLang reference measured on one AMD Instinct MI325X, no worked engine yet | AMD Instinct MI325X (gfx942, CDNA3), HIP C++ kernels, Rust host code, judged against SGLang on ROCm |
| `backends/nki-cpp/` | toolchain, runtime, decode loop and profiling written, measured on a trn2.48xlarge | AWS Trainium2, NKI kernels compiled to NEFF at build time, C++ host code on libnrt |
| others (e.g. CUDA) | not written | see `backends/README.md` for how to add one |

## Files

| file | read when |
|---|---|
| `references/workflow.md` | starting a task or a session; deciding what to do next |
| `references/model-card.md` | onboarding a model; computing floors; checking a semantic detail of the model |
| `references/correctness.md` | before any change to arithmetic; when a check fails; when choosing what "identical" means |
| `references/measurement.md` | before timing anything; when a result looks too good or too noisy |
| `references/optimization-method.md` | choosing the next optimization; writing up a result |
| `backends/metal-rust/README.md` | any Metal + Rust work (index of the backend files) |
| `backends/hip-rust/README.md` | any HIP + Rust work on an AMD Instinct GPU (index of the backend files) |
| `backends/nki-cpp/README.md` | any Trainium work (index of the backend files) |
| `examples/lfm2.5/` | a complete worked example: model card, decode dispatch map and the optimization log of an engine that reached 1.53x the reference's decode speed in a paired evaluation (300+ tokens/s on an M5 Max) |
| `examples/qwen3-0.6b-trn2/` | a worked Trainium2 example: Qwen3-0.6B at 2.21x NxD Inference (held-out), how a kernel lab plus harness sessions built it, and the optimization log |
| `scripts/README.md` | the A/B, timing and lock tools, and when the task contract lets you use them |
Pliki w paczce (57)
  • SKILL.md 7 KB
  • backends/README.md 7 KB
  • backends/hip-rust/README.md 14 KB
  • backends/hip-rust/attention.md 11 KB
  • backends/hip-rust/counterexamples.md 7 KB
  • backends/hip-rust/decode-loop.md 9 KB
  • backends/hip-rust/kernels-fusion.md 12 KB
  • backends/hip-rust/kernels-gemv-quant.md 10 KB
  • backends/hip-rust/prefill.md 9 KB
  • backends/hip-rust/profiling.md 9 KB
  • backends/hip-rust/runtime.md 17 KB
  • backends/hip-rust/serving.md 7 KB
  • backends/metal-rust/README.md 9 KB
  • backends/metal-rust/attention.md 12 KB
  • backends/metal-rust/counterexamples.md 9 KB
  • backends/metal-rust/decode-loop.md 11 KB
  • backends/metal-rust/kernels-fusion.md 14 KB
  • backends/metal-rust/kernels-gemv-quant.md 13 KB
  • backends/metal-rust/prefill.md 10 KB
  • backends/metal-rust/profiling.md 11 KB
  • backends/metal-rust/runtime.md 13 KB
  • backends/metal-rust/serving.md 10 KB
  • backends/nki-cpp/README.md 12 KB
  • backends/nki-cpp/attention.md 5 KB
  • backends/nki-cpp/counterexamples.md 2 KB
  • backends/nki-cpp/decode-loop.md 4 KB
  • backends/nki-cpp/kernels-fusion.md 5 KB
  • backends/nki-cpp/kernels-gemv-quant.md 3 KB
  • backends/nki-cpp/prefill.md 5 KB
  • backends/nki-cpp/profiling.md 3 KB
  • backends/nki-cpp/runtime.md 6 KB
  • backends/nki-cpp/tensor-parallel.md 10 KB
  • backends/nki-cpp/example/Makefile 1 KB
  • backends/nki-cpp/example/build.sh 1 KB
  • backends/nki-cpp/example/build_neff.py 1 KB
  • backends/nki-cpp/example/kernels.py 16 KB
  • backends/nki-cpp/example/kernels_main.py 2 KB
  • backends/nki-cpp/example/nrt_gemv.cpp 13 KB
  • backends/nki-cpp/example/nrt_inplace.cpp 2 KB
  • examples/lfm2.5/README.md 4 KB
  • examples/lfm2.5/decode-map.md 9 KB
  • examples/lfm2.5/model-card.md 19 KB
  • examples/lfm2.5/optimization-log.md 17 KB
  • examples/qwen3-0.6b-trn2/README.md 2 KB
  • examples/qwen3-0.6b-trn2/model-card.md 2 KB
  • examples/qwen3-0.6b-trn2/optimization-log.md 4 KB
  • references/correctness.md 14 KB
  • references/measurement.md 10 KB
  • references/model-card.md 4 KB
  • references/optimization-method.md 12 KB
  • references/workflow.md 6 KB
  • scripts/README.md 6 KB
  • scripts/ab.py 5 KB
  • scripts/bytes_per_token.py 8 KB
  • scripts/engine_client.py 10 KB
  • scripts/gpu_lock.py 2 KB
  • scripts/same.py 5 KB

Wersje

W bazie nie ma innych skilli o nazwie specialized-inference-engine.

Migawka bazy z 13.11.2026 · wersja paczki fdd498cc69ca (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100enginesmith-setup
    Prowadzi krok po kroku przez budowę wyspecjalizowanego silnika inferencyjnego dla jednego modelu na własnym komputerze i jego publikację.
  • 90build-engine
    Buduje, mierzy i uruchamia własny silnik wnioskowania dla otwartego modelu LLM na karcie NVIDIA, z kwantyzacją i dekodowaniem spekulatywnym.
  • 100profiling
    Ustala, gdzie idzie czas wnioskowania i dlaczego jądro jest wolne na NVIDIA, AMD ROCm, Apple Metal i CPU, bez obiecywania przyspieszenia.
  • 100rocm
    Pomaga pisać jądra HIP, korzystać z bibliotek ROCm, profilować kod, portować CUDA na HIP i debugować kod GPU AMD.
  • 95metal-kernel
    Pisze kernele Metal i MPS dla operatorów PyTorch: dodawanie wsparcia MPS, shadery Metal i portowanie kerneli CUDA na Apple Silicon, z dispatchem w native_functions.yaml.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.