Przejdź do treści
Biblioteka skilli
AI i agenci · Analiza

memo-run-eval

/memo-run-eval

Uruchamia testy wyszukiwania luamemo i odczytuje wyniki: R@1, R@5, R@10, MRR, odsetek chybień i liczbę bajtów przed trafieniem.

Repozytorium
luamemo ↗
Licencja
MIT
Ostatnia zmiana repozytorium
09.11.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
1
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 09.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Uruchamia testy wyszukiwania luamemo i odczytuje wyniki: R@1, R@5, R@10, MRR, odsetek chybień i liczbę bajtów przed trafieniem.

Oryginalny opis (EN)

Run luamemo's retrieval benchmarks on this machine with `memo eval` and read the results out — R@1/R@5/R@10, MRR, the miss rate and the bytes an agent reads before the right memory. Use when the user asks how well luamemo finds memories, wants to check an embedder or a change, or asks for benchmark numbers.

Jak działa

  1. What is measured
  2. 1. Preflight
  3. 2. Choose what to run
  4. 3. Run
  5. 4. Report
  6. 5. Boundaries
  7. 6. Troubleshooting
Sekcje z nagłówków SKILL.md, w oryginale · 181 linii, 1 771 słów
Podgląd SKILL.md
---
name: memo-run-eval
description: Run luamemo's retrieval benchmarks on this machine with `memo eval` and read the results out — R@1/R@5/R@10, MRR, the miss rate and the bytes an agent reads before the right memory. Use when the user asks how well luamemo finds memories, wants to check an embedder or a change, or asks for benchmark numbers.
---

# memo-run-eval — luamemo's benchmark workflow

Use this skill when the user wants to know how well luamemo retrieves the right
memory on their machine: after changing the embedder or a setting, before a release,
or to compare embedders. It is the same workflow the maintainers publish the numbers
with.

**It needs a terminal** (it runs `memo eval …`). If your client gives you no shell
tool, tell the user the commands below and read the output they paste back.

## What is measured

Each benchmark is a set of questions with one right memory each. luamemo searches,
and the benchmark records where the right memory came:

| Number | What it means for the agent |
|---|---|
| **R@1** | The first result is the right one: the agent can answer without reading further. |
| **R@5 / R@10** | The right memory is among the first 5 / 10 results. |
| **MRR** | The average of 1/rank: one number for "how high up". |
| **Miss rate** | The right memory is not in the 10 results the agent gets (1 − R@10). The agent then has to search again, grep or read whole files. It is R@10 seen from the other side, as the event that costs the agent. |
| **Bytes to hit** | What the agent reads before the right memory: the compact rows 1..rank as the MCP server returns them (mean / median / p90). Tokens ≈ bytes / 4. |

Retrieval only: no generated answers are scored, so the numbers are luamemo's part,
not the model's.

## 1. Preflight

```bash
memo eval status --json
```

Read it:

- `kit.installed` false → the **eval kit** is missing: the benchmark runners, the
  project-personalization questions, the fixtures and the published baselines. It is
  not part of the luamemo install. `memo eval install` downloads it from this
  release's GitHub assets and refuses it unless it matches the checksum the install
  ships. **Ask the user before installing it.** Explain what it is first. If
  `kit.release_asset` is false (luamemo run from a checkout), the command is
  `memo eval install --from <the checkout>`.
- `datasets.<name>.present` false → that dataset is not downloaded. `tiers.<tier>`
  says what each tier still needs and how many bytes.
- `embedder.label` → the embedder this project uses (`mine`). When `runnable` is
  false, `why` says why: an HTTP API embedder is never run by `memo eval`.
- `gguf_model` → the GGUF model file, if one is found.
- `history` → the results kept from earlier runs (`results`, the newest `last`). To
  compare without running anything: `memo eval history`.

## 2. Choose what to run

**Tier** — from what the user wants:

| Tier | Benchmarks | Download | When |
|---|---|---|---|
| `quick` | project personalization (36 questions), the session's fixed cost, the code map (40 questions) — all in the kit | none | a sanity check after a change |
| `standard` | quick + LoCoMo, ConvoMem, LongMemEval oracle | ≈ 55 MB | an embedder change; before a release |
| `full` | standard + MemBench (8,500), LongMemEval _s, speed and footprint (`perf`) | ≈ 840 MB | publishing numbers |

What the newer ones measure:
- **`session`:** the bytes an agent receives before its first search (usage rules, tool
  definitions with all tools and with `MEMO_MCP_TOOLS=core`, the brief). Not per embedder.
- **`codemap`:** "where is X done?" over luamemo's own modules. The `index_search` columns
  are as for memories; "grep instead" is what reading files in grep's order costs to reach
  the same answer.
- **`perf`:** this machine's search latency (1k / 20k / 100k memories), embedding and write
  speed, the server's start and memory, the database's size. It is never compared with
  the baseline, only with this machine's own earlier runs.

**`--tier corpus`** (a separate tier, never part of the others): five benchmarks on a local
100,000-row database that `memo eval build-corpus` makes from luamemo's own ingestion of real
source trees on this machine — `scale` (R@k, miss rate, bytes to hit and search time at that
size), `task_cost` (bytes an agent reads with the code map against grep, on the corpus' small
projects), `multiscope` (the same questions over 1, 2, 4 and 8 scopes), `maintenance`
(consolidation cost, redundant rows and recall, on a copy) and `learned` (the learned
reranker's change in R@1 after 100 / 1,000 / 10,000 simulated outcomes). The database is not
built by default: without it these are skipped with one line, and `build-corpus` takes hours
(embedding on the CPU) — **ask before starting it**, and run it in the background; it resumes
where it stopped. `learned` takes ≈ 45 minutes. Read `learned` as a simulated person's
corrections, not a real one's (the caveat is in `eval/README.md`).

Options, each a label of its own in the history:
- `--rerank ollama:<model>` (e.g. `ollama:gemma4:e4b`), `learned`, `noop`, `cross_encoder`:
  a reranker on the retrieval benchmarks. Its table column is its own cost per search
  (ms / model tokens / failed calls). A model on Ollama is slow (seconds per search), so
  run LoCoMo / ConvoMem with it in the background.
- `--prompts auto`: the embedder's task prompts (`MEMO_EMBED_PROMPTS`). Compare it with the
  same embedder without them.
- `--embedder gguf:<model>`: another GGUF embedding model, e.g. `gguf:embeddinggemma-2-Q8_0`.

Every run first checks each runner on tiny fixtures (a few seconds).

**Embedders:** the default `--embedder mine,hash` runs the project's own embedder
plus `hash` as the floor; `hash` is fast and always available. Others are `gguf` (the
in-process model), `ollama` and `tei` (only when that service is running). Only local
embedders run. Never set an API key or switch to a paid embedder for a benchmark.

**Datasets:** before `memo eval fetch NAME …`, tell the user the sizes and sources
(`status` lists them; each dataset has its own license) and **ask**. Downloads over
50 MB need `--yes`; add it only once the user agreed. Each file is checked against
the copy the baselines were measured on; one that differs is refused.

## 3. Run

```bash
memo eval run --tier standard            # or: --tier quick, or benchmarks by name
memo eval run locomo convomem --embedder gguf
memo eval run --tier quick --n 50        # a partial run: compared with nothing, not kept
```

- The runners never touch the user's memories: each one gets a new scratch database
  and none of the project's `MEMO_*` settings. Don't set `MEMO_DB` for it, and never
  point a runner at the user's database yourself.
- What a run keeps: the newest 3 results of each benchmark × embedder, as one memory
  each in the project's database (scope `eval:<project>`, out of the project's own
  searches). That is the record. Don't copy results into notes, logs or other memories.
- `standard` takes about half a minute with `hash`, 7–25 minutes with ollama or TEI on
  a GPU, and about 3 hours with `gguf` on a CPU (ConvoMem alone takes about 2). `full`
  adds MemBench (about 20 minutes with hash) and LongMemEval _s. Run the long ones in
  the background and wait for completion. Don't poll in a loop; tell the user roughly
  how long it will take.
- The output ends with **the last 3 measurements**: for each benchmark × embedder, this
  run (marked `*` / "now") and the two before it, each with its change from the one
  below. The last lines name the results directory: `report.md` (the same table in
  Markdown, ready to show), `summary.json` and each runner's JSON and log, for when
  something failed.

## 4. Report

**Show the user the "The last 3 measurements" table from `report.md`** (Markdown), then
read it, benchmark by benchmark:

- **R@1** → how often the agent's first result is the answer.
- **Miss rate** → how often the agent has to fall back to searching again or reading
  files.
- **Bytes to hit** → the context the agent spends before it reaches the answer.
- **The change column** → what moved since the previous run. Say what changed in
  between when you know it: a luamemo version (the `luamemo` column), an embedder, a
  setting. Identical numbers are expected when nothing changed, because the benchmarks
  are deterministic. A first run has nothing to compare with yet.
- **Rerank** → set its cost against its gain: seconds and model tokens per search, against
  points of R@1 and bytes to hit saved. Failed calls mean the search kept its own order;
  many failures make that row a baseline row.
- **Code map** → `index_search`'s bytes to hit against grep's: the context the agent saves
  when it looks the code up instead of reading files.
- **Session's fixed cost** → paid every session, before any search. Reading it: the
  tool definitions dominate it, and `MEMO_MCP_TOOLS=core` is the lever.

The run also compares each result with the published baseline for the same benchmark,
embedder and question count. It flags a drop of 1 point or more of R@1, or 10 % more
bytes to hit; repeat those flags. Then:

- Never average across benchmarks: they measure different things (a project's own
  conventions, long chats, updates over time).
- A row with no baseline is a new combination, not a failure.
- If a number looks too good (100 % on the LongMemEval oracle), say that this set is
  easy: it holds only the sessions with the answer. Don't celebrate it.

## 5. Boundaries

- Results stay on this machine unless the user asks otherwise. Publishing numbers
  (README tables, `eval/baselines.json`) is the maintainer's step. When asked, keep
  the exact command, the luamemo version and the embedder with the numbers;
  `summary.json` has all three.
- Don't write the results into memory yourself: `memo eval run` already keeps the last
  3. Asked about earlier results, use `memo eval history` (or `memory_search` in the
  scope `eval:<project>`).

## 6. Troubleshooting

| What you see | What to do |
|---|---|
| `no eval kit for luamemo X` | `memo eval install` (ask first), or `--from <checkout>` |
| `missing dataset(s) … memo eval fetch …` | the printed fetch command (ask first; sizes are printed) |
| `no GGUF model found` / gguf runs fail | `memo gguf status`, `memo ping` |
| `smoke … FAILED — <log>` | read that log: the runners don't run on this machine; show the user the error |
| a runner `FAILED after Ns — <log>` | read the log; the other rows are still valid |
| `ollama` / `tei` rows fail | the service isn't running at its URL (`OLLAMA_URL` / `TEI_URL`) |
| a scratch database left behind (`--keep-db`, or a crash) | it is in the results directory's `scratch/`; it is safe to delete |

Wersje

W bazie nie ma innych skilli o nazwie memo-run-eval.

Migawka bazy z 13.11.2026 · wersja paczki 31848ec3d8b8 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100eval-set
    Buduje i uruchamia projektową ewaluację wyszukiwania reguł, by zmiany reguł lub promptu bazowego były mierzone: przypadki prompt do oczekiwanej reguły, MRR@k i trafność przed i po edycji.
  • 95agent-eval
    Uruchamia zestawy testów na agentach przeglądu i ocenia wyniki, by sprawdzić dokładność i regresje po zmianach.
  • 95stats
    Podaje statystyki korpusu Captain Memo, czyli liczbę fragmentów na kanał, obserwacje i postęp indeksowania.
  • 85strata:eval
    Mierzy, czy zmiana wyszukiwania poprawiła wyniki, na stałym zestawie wzorcowym: recall@k i MRR, bez sędziego LLM.
  • 90eval-analysis
    Analizuje wyniki ewaluacji, by ocenić jakość sygnału i wskazać zmiany promptów lub konfiguracji, także przy regresjach.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.