Przejdź do treści
Biblioteka skilli
AI i agenci · Testowanie

testing-llm-features

/testing-llm-features

Buduje offline'owy zestaw ewaluacji dla funkcji opartych na LLM: złote przypadki dla szablonów promptów, deterministyczna ocena, bramka regresji i CI z promptfoo, Vitest lub deepeval.

Licencja
MIT
Ostatnia zmiana repozytorium
26.08.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
4
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 26.08.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Buduje offline'owy zestaw ewaluacji dla funkcji opartych na LLM: złote przypadki dla szablonów promptów, deterministyczna ocena, bramka regresji i CI z promptfoo, Vitest lub deepeval.

Argumenty

/testing-llm-features The feature, its prompts and model config, what "good output" means for it, and the current test/CI setup
Oryginalny opis (EN)

Builds an offline eval suite for LLM-powered features: golden reference cases per prompt template, deterministic scoring where possible, a regression gate on prompt, model and retrieval changes, and CI wiring with promptfoo, Vitest or deepeval. Use when a product ships prompts, agents or RAG, when a model or prompt upgrade needs a regression check, when LLM output is currently verified by someone eyeballing it, or when asked how to test a feature whose output is non-deterministic.

Jak działa

  1. When to Use
  2. Operating Principles
  3. Workflow
  4. Common Failure Modes
  5. Resource Map
  6. Related Skills
  7. Definition of Done
Sekcje z nagłówków SKILL.md, w oryginale · 160 linii, 1 792 słów
Podgląd SKILL.md
---
name: testing-llm-features
description: 'Builds an offline eval suite for LLM-powered features: golden reference cases per prompt template, deterministic scoring where possible, a regression gate on prompt, model and retrieval changes, and CI wiring with promptfoo, Vitest or deepeval. Use when a product ships prompts, agents or RAG, when a model or prompt upgrade needs a regression check, when LLM output is currently verified by someone eyeballing it, or when asked how to test a feature whose output is non-deterministic.'
argument-hint: 'The feature, its prompts and model config, what "good output" means for it, and the current test/CI setup'
user-invocable: true
---

# Testing LLM Features

Use this skill when a product's behaviour depends on a model, and the current verification method is a person looking at output and deciding it seems fine.

The problem is not that LLM output is hard to assert on. It is that **the same input produces different output, and a prompt or model change can degrade quality without breaking anything**. No test in a normal suite fails. Coverage stays green. The feature just gets worse, and you find out from users.

An eval suite is the unit-test layer for this: a fixed set of cases, run on every change that could affect model behaviour, scored consistently enough that a regression is visible.

## When to Use

- a feature calls a model and there is no automated check on output quality
- a prompt, model version, temperature, tool list, or retrieval config is about to change
- a model upgrade is proposed and nobody can say what would regress
- an agent or chain has grown beyond what manual spot-checks can cover
- RAG answers are drifting and it is unclear whether retrieval or generation is at fault
- a CI gate is needed for AI behaviour, not just AI code

## Operating Principles

- **Golden cases, not vibes.** A case has an input, an expectation, and a scoring rule. "Looks good" is not an expectation.
- **Determinism where you can get it.** Pin the model version, set temperature to 0, fix seeds, freeze retrieval snapshots. Non-determinism is a property of the model; it is not an excuse for a non-reproducible test.
- **Test the assertable layer first.** Schema conformance, required fields, refusal behaviour, citation presence, tool-call shape, latency and cost. Most feature-breaking regressions are structural, and structural checks are cheap and exact.
- **Rubrics before judges.** If an LLM judges the output, it needs a written rubric with a scale, and the judge itself needs validating against human labels. An unrubriced judge is a random number with an API bill.
- **A regression is a new failure on a case that used to pass.** That is the gate. An absolute score threshold on a hard case set gates nothing useful.
- **Separate retrieval failures from generation failures.** A RAG answer can be wrong because the right document was never retrieved. Score retrieval independently or you will tune the prompt to fix an index problem.
- **Cases are a product artifact.** They encode what the feature promises. Review them like code and grow them from real failures.
- **Cost and latency are results.** A 12% quality gain for 4× the cost and 3× the latency is a product decision, so report all three.

## Workflow

### Phase 0: Map the surface

Find everything that can change model behaviour, because each is a trigger for the suite:

| Element | Where to look |
| --- | --- |
| Prompt templates | `prompts/`, `*.prompt.*`, template literals passed to the SDK |
| System prompts | client construction, agent definitions |
| Model config | model id, temperature, top_p, max tokens, stop sequences, thinking budget |
| Tool definitions | tool/function schemas, MCP servers, tool grants |
| Retrieval | chunking, embedding model, index, `topK`, reranking, filters |
| Output contract | Zod/schema on the response, structured-output config, parsing code |
| Orchestration | chain and agent-loop structure, retries, fallbacks between models |

Write down what the feature *promises* - the user-visible contract. "Summarise the ticket in ≤3 sentences, never invent a customer name, always cite the ticket id." That sentence is where the cases come from.

### Phase 1: Design the case set

`./resources/eval-case-design.md` has the taxonomy. The minimum viable set per prompt template:

| Class | Purpose | Count to start |
| --- | --- | --- |
| **Golden** | canonical inputs with known-good outputs | 5–10 |
| **Edge** | empty, huge, multilingual, malformed, contradictory input | 5 |
| **Negative** | must refuse, must not answer, must not leak | 3–5 |
| **Regression** | one case per bug ever found in production | grows forever |
| **Adversarial** | injection and jailbreak attempts | see `testing-llm-guardrails` |

Cases carry the metadata that makes a failure diagnosable: which prompt template, which capability, why the expectation is what it is. A case whose expectation nobody can justify is a case nobody will maintain.

**The regression class is the one that compounds.** Every production incident becomes a permanent case. After six months this set is the most valuable test asset the feature has, and it costs one case per incident to build.

### Phase 2: Choose scoring per case

`./resources/scoring-strategies.md` covers each in detail. Pick the cheapest one that can actually fail:

| Strategy | Use when | Cost |
| --- | --- | --- |
| Exact / regex match | classification, extraction, IDs, enum outputs | free, exact |
| Schema validation | any structured output | free, exact |
| Required / forbidden substrings | must cite the ticket id, must never say "as an AI" | free |
| Deterministic property | length limits, valid JSON, no PII pattern, citation resolves to a real source | free |
| Embedding similarity | paraphrase-tolerant equivalence | cheap, fuzzy |
| Rubric-based LLM judge | tone, helpfulness, faithfulness | expensive, needs validation |
| Human review | the final arbiter on a sample | expensive, definitive |

The ordering is the advice. Teams reach for an LLM judge first because it feels closest to "quality", and end up with a noisy score they cannot debug. Most real regressions trip a free check.

### Phase 3: Implement the suite

Two viable shapes, both in `./resources/promptfoo-setup.md`:

- **promptfoo** - config-driven, good matrix support across prompts and models, built-in assertion types, decent reporting. The default choice for a suite that will grow.
- **Vitest + the SDK** - the eval suite as ordinary tests. Better when evals must share fixtures with the app, or when the team will not adopt another tool.

Non-negotiables whichever you pick:

- pin the exact model version (`claude-sonnet-4-5-20250929`, not `latest`) - a floating alias makes every run unreproducible
- temperature 0 unless the feature's value depends on variation
- freeze retrieval: snapshot the documents, do not query a live index
- record model, prompt hash, and case-set version with every result
- cache responses so a re-run of unchanged cases is free
- run N times on cases where variance matters, and report the distribution, not one sample

### Phase 4: Gate on regressions

The gate is comparative:

| Condition | Verdict |
| --- | --- |
| A case that passed on the baseline now fails | **BLOCK** - this is the whole point |
| Aggregate score drops beyond the noise band | **WARN** - investigate before merging |
| New cases fail | **INFO** - they were added because they fail |
| Cost or latency regresses beyond budget | **WARN** |
| Flaky case (passes and fails across N runs) | fix the case or the determinism, do not average it away |

Establish the noise band empirically: run the suite three times on an unchanged system and see how much the score moves. Anything inside that band is not a signal, and gating on it produces the flakiest CI job in the repo.

Store the baseline as a committed artifact so the comparison is reviewable in the diff. See `./resources/promptfoo-setup.md` for the CI job, the trigger paths, and the API-key-free path for forks.

### Phase 5: Report

- **Regressions** - case, what it asserted, baseline versus now, the actual output
- **Score by capability** - not one global number; "citation accuracy 94%, refusal 100%, summary faithfulness 81%" is actionable
- **Retrieval versus generation** - for RAG, always split
- **Cost and latency** - per case and total, against budget
- **Variance** - which cases are unstable across runs
- **Coverage gaps** - prompt templates with no cases, capabilities with no negative case

## Common Failure Modes

- **Evals only in a notebook.** They exist, they are never run in CI, and the regression ships anyway.
- **LLM judge with no rubric.** Produces a number that moves for unrelated reasons and cannot be debugged.
- **Judging with the model under test.** Self-preference is real; use a different model, or a deterministic check.
- **Floating model alias.** `latest` silently changes the system under test between runs.
- **One aggregate score.** Masks a 40% drop in citation accuracy behind a 3% total move.
- **Golden outputs copied from model output.** Recording what the model *did* rather than what it *should do* freezes today's bugs into the expectations. Golden outputs need a human's approval.
- **No negative cases.** The feature that must refuse is never tested for refusing, and the first jailbreak is found by a user.
- **Tuning the prompt to fix a retrieval miss.** Weeks of prompt work on an indexing bug.
- **Cases that grow only from imagination.** Every production failure should become a case; that is where the good cases come from.

## Resource Map

- `./resources/eval-case-design.md` - case taxonomy, how to write a golden expectation, case metadata, growing the set from incidents
- `./resources/scoring-strategies.md` - each scoring method, when to use it, LLM-judge rubrics and how to validate a judge
- `./resources/promptfoo-setup.md` - promptfoo and Vitest implementations, determinism settings, CI job, baseline comparison, cost control

## Related Skills

- `testing-llm-guardrails` - runtime output validation and the adversarial suite: injection, jailbreaks, tool abuse
- `reviewing-ai-output-groundedness` - the human review protocol for factuality, and the attestation an eval suite cannot replace
- `designing-test-data` - building the input corpora these cases consume
- `analyzing-quality-metrics` - defining and trending eval pass rate, variance, and cost as real metrics
- `unslop-tests` - the same discipline applied to test code: an eval that cannot fail is worse than no eval
- `scoping-change-relevance` - which changes should trigger the suite (`modified-prompt`, model config, tool grants, retrieval)

## Definition of Done

This skill is complete when:

- every prompt template has golden, edge, and negative cases with justified expectations
- scoring is the cheapest method that can actually fail, and any LLM judge has a written rubric and validation against human labels
- the model version, temperature, seeds, and retrieval snapshot are pinned and recorded with results
- the suite runs in CI on prompt, model, tool, and retrieval changes, and gates on regressions against a committed baseline
- the noise band is measured, so the gate does not fire on variance
- retrieval and generation failures are reported separately for RAG features
- cost and latency are reported alongside quality
- every past production failure has a permanent regression case
Pliki w paczce (4)
  • SKILL.md 11 KB
  • resources/eval-case-design.md 7 KB
  • resources/promptfoo-setup.md 9 KB
  • resources/scoring-strategies.md 8 KB

Wersje

W bazie nie ma innych skilli o nazwie testing-llm-features.

Identyczna treść występuje jeszcze w 1 innych folderach na GitHubie. Tu pokazujemy jedną, najlepiej ocenioną kopię.

Migawka bazy z 13.11.2026 · wersja paczki 2529fdc6eb65 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100promptfoo-evaluation
    Tworzy i uruchamia ewaluacje Promptfoo dla promptów LLM i potoków RAG: konfiguracja promptfooconfig.yaml, asercje deterministyczne, semantyczne i oceniane przez model, raport HTML oraz bramka regresji w CI.
  • 100accelint-eval-architect
    Pomaga dodać ewaluację do skilla, agenta albo narzędzia, w tym chatbota RAG, i skonfigurować narzędzia takie jak DeepEval, Ragas albo Promptfoo.
  • 100evals
    Buduje harness regresji i ewaluacji dla kodu pisanego przez AI i funkcji AI: testy charakteryzujące blokują obecne zachowanie przed refaktoryzacją, a zestaw Promptfoo sprawdza chatboty, RAG i klasyfikatory w bramce wydania.
  • 100promptfoo-evaluation
    Konfiguruje i uruchamia ewaluację LLM we frameworku Promptfoo: konfiguracje promptfooconfig.yaml, własne asercje w Pythonie, llm-rubric jako sędzia i przykłady few-shot.
  • 100evals
    Buduje i uruchamia ewaluacje funkcji LLM, agentów i RAG: zbiory wzorcowe, asercje, sędziów LLM i bramki regresji w CI.
  • 100llm-feature-eval
    Projektuje zestaw ewaluacyjny dla funkcji opartej na LLM: złoty zbiór danych, rubryki, progi regresji, koszty i przypadki brzegowe.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.