promptfoo-evaluation
/promptfoo-evaluation
Tworzy i uruchamia ewaluacje Promptfoo dla promptów LLM i potoków RAG: konfiguracja promptfooconfig.yaml, asercje deterministyczne, semantyczne i oceniane przez model, raport HTML oraz bramka regresji w CI.
- Autor
- testland ↗
- Repozytorium
- qa ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 20.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 12
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 6
Brak sygnałów. Repozytorium zmienione 20.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Tworzy i uruchamia ewaluacje Promptfoo dla promptów LLM i potoków RAG: konfiguracja promptfooconfig.yaml, asercje deterministyczne, semantyczne i oceniane przez model, raport HTML oraz bramka regresji w CI.
Oryginalny opis (EN)
Authors and runs Promptfoo evals for LLM prompts and RAG pipelines - wires `promptfooconfig.yaml` providers + prompts + tests + assertions (deterministic `equals` / `contains` / `is-json` / `regex`, semantic `similar`, model-graded `llm-rubric` / `factuality` / `g-eval`, performance `latency` / `cost`, custom `javascript` / `python`), runs `npx promptfoo eval`, views HTML report via `promptfoo view`, and integrates CI for regression gating. Use when the user runs Promptfoo, asks about prompt regression suites, or needs an eval-driven workflow for LLM-backed features.
Jak działa
- When to use
- Step 1 - Install
- Step 2 - First eval
- Step 3 - Variable interpolation
- Step 4 - Assertion catalog
- Step 5 - defaultTest pattern
- Step 6 - Output transforms
- Step 7 - assert-set grouping
- Step 8 - CI integration
- Anti-patterns
Podgląd SKILL.md
---
name: promptfoo-evaluation
description: "Authors and runs Promptfoo evals for LLM prompts and RAG pipelines - wires `promptfooconfig.yaml` providers + prompts + tests + assertions (deterministic `equals` / `contains` / `is-json` / `regex`, semantic `similar`, model-graded `llm-rubric` / `factuality` / `g-eval`, performance `latency` / `cost`, custom `javascript` / `python`), runs `npx promptfoo eval`, views HTML report via `promptfoo view`, and integrates CI for regression gating. Use when the user runs Promptfoo, asks about prompt regression suites, or needs an eval-driven workflow for LLM-backed features."
---
# promptfoo-evaluation
[pf-intro]: https://promptfoo.dev/docs/intro/
[pf-config]: https://promptfoo.dev/docs/configuration/guide/
A `promptfooconfig.yaml` declares **providers** (LLMs under test),
**prompts** (templates with `{{var}}` placeholders), **tests**
(input variables + assertions); `promptfoo eval` runs the
cross-product (per [pf-config][pf-config]).
## When to use
- The repo has a `promptfooconfig.yaml` or the user wants to
author one.
- The user needs prompt regression suites - same inputs, multiple
providers, fail-on-diff.
- A CI workflow needs an eval gate on prompt or model changes.
- The team prefers vendor-neutral eval (vs OpenAI Evals' OpenAI-first
posture).
## Step 1 - Install
Per [github.com/promptfoo/promptfoo][pf-gh]:
[pf-gh]: https://github.com/promptfoo/promptfoo
```bash
npm install -g promptfoo
# or
brew install promptfoo
# or one-off
npx promptfoo@latest
```
Per [pf-gh][pf-gh]: "Node.js 20.20+ or 22.22+ is required for npm
and npx usage."
## Step 2 - First eval
Initialize from a built-in example, then run:
```bash
promptfoo init --example getting-started
cd getting-started
promptfoo eval
promptfoo view # opens HTML report
```
(Commands per [pf-gh][pf-gh].)
A minimal `promptfooconfig.yaml` per [pf-config][pf-config]:
```yaml
prompts:
- file://prompt1.txt
providers:
- openai:gpt-5-mini
- anthropic:claude-haiku-4-5
tests:
- vars:
language: French
input: Hello world
assert:
- type: contains-json
```
Provider syntax `<vendor>:<model>` works for OpenAI, Anthropic,
Vertex (`vertex:gemini-2.0-flash-exp`), Ollama (`ollama:llama2`), and
30+ others (per [pf-config][pf-config]).
## Step 3 - Variable interpolation
Vars interpolate into prompts using `{{variable}}`. Arrays create
combinations (per [pf-config][pf-config]):
```yaml
tests:
- vars:
language: [French, German, Spanish]
input: [Hello world, Good morning]
```
This generates 6 test rows (3 languages × 2 inputs). Vars can also
load from files:
```yaml
tests:
- vars:
var3: file://path/to/var3.txt
context: file://fetch_from_vector_database.py
```
## Step 4 - Assertion catalog
Per [promptfoo.dev/docs/configuration/expected-outputs/][pf-asserts]:
[pf-asserts]: https://promptfoo.dev/docs/configuration/expected-outputs/
**Deterministic (string + structure):**
| Type | Example |
|---|---|
| `equals` | `value: 'expected string'` |
| `contains` / `icontains` | `value: 'substring'` (case-sensitive / insensitive) |
| `starts-with` | `value: 'prefix'` |
| `contains-any` / `contains-all` | `value: ['a', 'b']` |
| `regex` | `value: '^pattern$'` |
| `is-json` / `contains-json` | (validates / locates JSON) |
| `is-sql` / `contains-sql` | (SQL syntax) |
| `is-xml` / `is-html` / `contains-xml` / `contains-html` | (markup) |
| `is-valid-openai-tools-call` / `is-valid-openai-function-call` | (tool calls) |
**Custom logic:**
```yaml
- type: javascript
value: 'output.length > 10'
- type: python
value: 'file://script.py'
```
**Text-quality metrics** (default thresholds per [pf-asserts][pf-asserts]):
| Type | Default threshold |
|---|---|
| `rouge-n` | 0.75 |
| `bleu` | 0.5 |
| `gleu` | 0.5 |
| `meteor` | 0.5 |
| `levenshtein` | 5 (edit distance max) |
**Performance + cost:**
```yaml
- type: latency
threshold: 200 # milliseconds
- type: cost
threshold: 0.001 # dollars per response
```
**Model-graded (LLM-as-judge):**
| Type | Use |
|---|---|
| `llm-rubric` | Free-form rubric: `value: 'Is helpful and accurate'` |
| `model-graded-closedqa` | Closed-QA evaluation method |
| `factuality` | Compares against reference facts |
| `g-eval` | Chain-of-thought scoring |
| `answer-relevance` | Checks output relates to query |
| `context-faithfulness` / `context-recall` / `context-relevance` | RAG-specific |
**Semantic similarity:**
```yaml
- type: similar
value: 'reference text'
threshold: 0.8 # cosine similarity via embeddings
```
**Negation:** all deterministic assertions support a `not-` prefix
(`not-equals`, `not-contains`, `not-regex`, etc.) per [pf-asserts][pf-asserts].
## Step 5 - defaultTest pattern
Shared assertions/vars across all tests:
```yaml
defaultTest:
vars:
shared_var: 'shared content'
assert:
- type: llm-rubric
value: does not describe self as AI
options:
provider: openai:gpt-5-mini-0613
tests:
- vars:
unique_var: value1
```
(Per [pf-config][pf-config].)
## Step 6 - Output transforms
Modify LLM output before assertions execute (per [pf-config][pf-config]):
```yaml
tests:
- vars:
body: Hello world
options:
transform: output.toUpperCase()
```
Or load from a file:
```yaml
options:
transform: file://transform.js:customTransform
```
## Step 7 - assert-set grouping
Group assertions and require a threshold percentage to pass (per
[pf-asserts][pf-asserts]):
```yaml
assert:
- type: assert-set
threshold: 0.5 # 50% of grouped asserts must pass
assert:
- type: cost
threshold: 0.001
- type: latency
threshold: 200
```
## Step 8 - CI integration
Provider API keys via env vars (per [pf-gh][pf-gh]: `export
OPENAI_API_KEY=sk-abc123`).
GitHub Actions pattern (per [promptfoo.dev/docs/integrations/github-action][pf-gha]):
[pf-gha]: https://promptfoo.dev/docs/integrations/github-action/
```yaml
- uses: promptfoo/promptfoo-action@v2
with:
openai-api-key: ${{ secrets.OPENAI_API_KEY }}
config: 'promptfooconfig.yaml'
cache-path: '~/.cache/promptfoo'
use-config-prompts: false
no-share: true
promptfoo-version: 'latest'
```
The action posts a PR comment with regression diff vs the base
branch. Caching reuses LLM responses for unchanged tests (per
[promptfoo.dev/docs/configuration/caching/][pf-cache]).
[pf-cache]: https://promptfoo.dev/docs/configuration/caching/
## Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Only deterministic assertions on creative outputs | LLM responses vary; rigid asserts produce flake | Use `llm-rubric` or `similar` (Step 4) |
| Single provider in config | Misses cross-provider regression | At least 2 providers per eval (Step 2) |
| No cost / latency caps | Eval cost balloons per PR | `cost` + `latency` asserts (Step 4) |
| `assert-set` with `threshold: 0` | Effectively disables grouping | Pick a real threshold (Step 7) |
| Fresh provider model on every run without pinning | Output drifts when vendor updates model | Pin specific snapshot (e.g., `openai:gpt-5-mini-0613`) |
| Skip caching in CI | Re-runs every test on every PR | `cache-path` in GHA action (Step 8) |
## Limitations
- Model-graded assertions invoke a judge LLM per test row → cost
per test ≈ 2× a deterministic-only eval.
- Even seeded LLMs drift between provider model updates; pin model
versions in CI to bound the regression surface.
- Promptfoo ships HTML + JSON + JUnit reporters; no native
Markdown PR-comment formatter (use the GHA action for that).
## References
- [pf-intro][pf-intro] - overview
- [pf-config][pf-config] - full configuration guide
- [pf-asserts][pf-asserts] - assertion catalog
- [pf-gha][pf-gha] - GitHub Action
- [pf-cache][pf-cache] - caching mechanics
- [pf-gh][pf-gh] - install commands
- `openai-evals`, `deepeval-evaluation`, `ragas-evaluation` - sister tools (different framework styles)
Pliki w paczce (12)
- SKILL.md 8 KB
- .tessl-plugin/plugin.json 1 KB
- evals/forty-row-answer-sheet-as-a-launch-gate/criteria.json 7 KB
- evals/forty-row-answer-sheet-as-a-launch-gate/task.md 11 KB
- evals/june-board-number-cannot-be-reproduced-in-september/criteria.json 6 KB
- evals/june-board-number-cannot-be-reproduced-in-september/task.md 6 KB
- evals/pr-eval-bill-went-from-nine-dollars-to-four-hundred/criteria.json 6 KB
- evals/pr-eval-bill-went-from-nine-dollars-to-four-hundred/task.md 10 KB
- evals/safety-gate-has-not-gone-red-since-march/criteria.json 5 KB
- evals/safety-gate-has-not-gone-red-since-march/task.md 10 KB
- evals/subject-line-suite-red-three-mornings-a-week/criteria.json 6 KB
- evals/subject-line-suite-red-three-mornings-a-week/task.md 10 KB
Wersje
| Repozytorium | Wynik | Licencja |
|---|---|---|
| Ta strona · testland/qa | 100 | MIT |
| daymade/claude-code-skills | 100 | MIT |
| testland/qa-skills | 100 | MIT |
Migawka bazy z 13.11.2026 · wersja paczki 05031fb10f17 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100testing-llm-features
Buduje offline'owy zestaw ewaluacji dla funkcji opartych na LLM: złote przypadki dla szablonów promptów, deterministyczna ocena, bramka regresji i CI z promptfoo, Vitest lub deepeval. - 100evals
Buduje harness regresji i ewaluacji dla kodu pisanego przez AI i funkcji AI: testy charakteryzujące blokują obecne zachowanie przed refaktoryzacją, a zestaw Promptfoo sprawdza chatboty, RAG i klasyfikatory w bramce wydania. - 100evals
Buduje i uruchamia ewaluacje funkcji LLM, agentów i RAG: zbiory wzorcowe, asercje, sędziów LLM i bramki regresji w CI. - 100eval-harness
Buduje i uruchamia zestawy ewaluacji funkcji AI w aplikacji użytkownika (prompty, wywołania LLM, RAG, agenci): graders, metryki pass@k, opóźnienia i koszt, w trybach tworzenia, uruchamiania i raportowania. - 100accelint-eval-architect
Pomaga dodać ewaluację do skilla, agenta albo narzędzia, w tym chatbota RAG, i skonfigurować narzędzia takie jak DeepEval, Ragas albo Promptfoo. - 100eval-prompt
Ocenia prompty wnioskowania PAL za pomocą promptfoo z katalogu eval/: po zmianie lub podejrzeniu regresji, przy tworzeniu nowej ewaluacji od zera i przy porównaniu kandydata v2 z bazą.