promptfoo-evaluation
/promptfoo-evaluation
Pomaga w Promptfoo: konfiguruje dostawców, asercje deterministyczne i modelowe, ewaluacje oraz bramki w CI dla promptu lub potoku RAG.
- Autor
- testland ↗
- Repozytorium
- qa-skills ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 12.11.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 1
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 7
Brak sygnałów. Repozytorium zmienione 12.11.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Pomaga w Promptfoo: konfiguruje dostawców, asercje deterministyczne i modelowe, ewaluacje oraz bramki w CI dla promptu lub potoku RAG.
Oryginalny opis (EN)
Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating.
Jak działa
- When to use
- Step 1 - Install
- Step 2 - First eval
- Step 3 - Variable interpolation
- Step 4 - Assertion catalog
- Step 5 - defaultTest pattern
- Step 6 - Output transforms
- Step 7 - assert-set grouping
- Step 8 - CI integration
- Anti-patterns
Podgląd SKILL.md
---
name: promptfoo-evaluation
description: "Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating."
---
# promptfoo-evaluation
[pf-intro]: https://promptfoo.dev/docs/intro/
[pf-config]: https://promptfoo.dev/docs/configuration/guide/
A `promptfooconfig.yaml` declares **providers** (LLMs under test),
**prompts** (templates with `{{var}}` placeholders), **tests**
(input variables + assertions); `promptfoo eval` runs the
cross-product (per [pf-config][pf-config]).
## When to use
- The repo has a `promptfooconfig.yaml` or the user wants to
author one.
- The user needs prompt regression suites - same inputs, multiple
providers, fail-on-diff.
- A CI workflow needs an eval gate on prompt or model changes.
- The team prefers vendor-neutral eval (vs OpenAI Evals' OpenAI-first
posture).
## Step 1 - Install
Per [github.com/promptfoo/promptfoo][pf-gh]:
[pf-gh]: https://github.com/promptfoo/promptfoo
```bash
npm install -g promptfoo
# or
brew install promptfoo
# or one-off
npx promptfoo@latest
```
Per [pf-gh][pf-gh]: "Node.js 20.20+ or 22.22+ is required for npm
and npx usage."
## Step 2 - First eval
Initialize from a built-in example, then run:
```bash
promptfoo init --example getting-started
cd getting-started
promptfoo eval
promptfoo view # opens HTML report
```
(Commands per [pf-gh][pf-gh].)
A minimal `promptfooconfig.yaml` per [pf-config][pf-config]:
```yaml
prompts:
- file://prompt1.txt
providers:
- openai:gpt-5-mini
- anthropic:claude-haiku-4-5
tests:
- vars:
language: French
input: Hello world
assert:
- type: contains-json
```
Provider syntax `<vendor>:<model>` works for OpenAI, Anthropic,
Vertex (`vertex:gemini-2.0-flash-exp`), Ollama (`ollama:llama2`), and
30+ others (per [pf-config][pf-config]).
## Step 3 - Variable interpolation
Vars interpolate into prompts using `{{variable}}`. Arrays create
combinations (per [pf-config][pf-config]):
```yaml
tests:
- vars:
language: [French, German, Spanish]
input: [Hello world, Good morning]
```
This generates 6 test rows (3 languages × 2 inputs). Vars can also
load from files:
```yaml
tests:
- vars:
var3: file://path/to/var3.txt
context: file://fetch_from_vector_database.py
```
## Step 4 - Assertion catalog
Per [promptfoo.dev/docs/configuration/expected-outputs/][pf-asserts]:
[pf-asserts]: https://promptfoo.dev/docs/configuration/expected-outputs/
**Deterministic (string + structure):**
| Type | Example |
|---|---|
| `equals` | `value: 'expected string'` |
| `contains` / `icontains` | `value: 'substring'` (case-sensitive / insensitive) |
| `starts-with` | `value: 'prefix'` |
| `contains-any` / `contains-all` | `value: ['a', 'b']` |
| `regex` | `value: '^pattern$'` |
| `is-json` / `contains-json` | (validates / locates JSON) |
| `is-sql` / `contains-sql` | (SQL syntax) |
| `is-xml` / `is-html` / `contains-xml` / `contains-html` | (markup) |
| `is-valid-openai-tools-call` / `is-valid-openai-function-call` | (tool calls) |
**Custom logic:**
```yaml
- type: javascript
value: 'output.length > 10'
- type: python
value: 'file://script.py'
```
**Text-quality metrics** (default thresholds per [pf-asserts][pf-asserts]):
| Type | Default threshold |
|---|---|
| `rouge-n` | 0.75 |
| `bleu` | 0.5 |
| `gleu` | 0.5 |
| `meteor` | 0.5 |
| `levenshtein` | 5 (edit distance max) |
**Performance + cost:**
```yaml
- type: latency
threshold: 200 # milliseconds
- type: cost
threshold: 0.001 # dollars per response
```
**Model-graded (LLM-as-judge):**
| Type | Use |
|---|---|
| `llm-rubric` | Free-form rubric: `value: 'Is helpful and accurate'` |
| `model-graded-closedqa` | Closed-QA evaluation method |
| `factuality` | Compares against reference facts |
| `g-eval` | Chain-of-thought scoring |
| `answer-relevance` | Checks output relates to query |
| `context-faithfulness` / `context-recall` / `context-relevance` | RAG-specific |
**Semantic similarity:**
```yaml
- type: similar
value: 'reference text'
threshold: 0.8 # cosine similarity via embeddings
```
**Negation:** all deterministic assertions support a `not-` prefix
(`not-equals`, `not-contains`, `not-regex`, etc.) per [pf-asserts][pf-asserts].
## Step 5 - defaultTest pattern
Shared assertions/vars across all tests:
```yaml
defaultTest:
vars:
shared_var: 'shared content'
assert:
- type: llm-rubric
value: does not describe self as AI
options:
provider: openai:gpt-5-mini # in CI, pin a dated snapshot id from the provider's model page
tests:
- vars:
unique_var: value1
```
(Per [pf-config][pf-config].)
## Step 6 - Output transforms
Modify LLM output before assertions execute (per [pf-config][pf-config]):
```yaml
tests:
- vars:
body: Hello world
options:
transform: output.toUpperCase()
```
Or load from a file:
```yaml
options:
transform: file://transform.js:customTransform
```
## Step 7 - assert-set grouping
Group assertions and require a threshold percentage to pass (per
[pf-asserts][pf-asserts]):
```yaml
assert:
- type: assert-set
threshold: 0.5 # 50% of grouped asserts must pass
assert:
- type: cost
threshold: 0.001
- type: latency
threshold: 200
```
## Step 8 - CI integration
Provider API keys via env vars (per [pf-gh][pf-gh]: `export
OPENAI_API_KEY=sk-abc123`).
GitHub Actions pattern (per [promptfoo.dev/docs/integrations/github-action][pf-gha]):
[pf-gha]: https://promptfoo.dev/docs/integrations/github-action/
```yaml
- uses: promptfoo/promptfoo-action@v2
with:
openai-api-key: ${{ secrets.OPENAI_API_KEY }}
config: 'promptfooconfig.yaml'
cache-path: '~/.cache/promptfoo'
use-config-prompts: false
no-share: true
promptfoo-version: 'latest'
```
The action posts a PR comment with regression diff vs the base
branch. Caching reuses LLM responses for unchanged tests (per
[promptfoo.dev/docs/configuration/caching/][pf-cache]).
[pf-cache]: https://promptfoo.dev/docs/configuration/caching/
[oa-models]: https://developers.openai.com/api/docs/models/gpt-5-mini
## Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Only deterministic assertions on creative outputs | LLM responses vary; rigid asserts produce flake | Use `llm-rubric` or `similar` (Step 4) |
| Single provider in config | Misses cross-provider regression | At least 2 providers per eval (Step 2) |
| No cost / latency caps | Eval cost balloons per PR | `cost` + `latency` asserts (Step 4) |
| `assert-set` with `threshold: 0` | Effectively disables grouping | Pick a real threshold (Step 7) |
| Fresh provider model on every run without pinning | Output drifts when vendor updates model | Pin a dated snapshot id from the provider's model page ([OpenAI's lists them per model][oa-models]), not a moving alias |
| Skip caching in CI | Re-runs every test on every PR | `cache-path` in GHA action (Step 8) |
## Limitations
- Model-graded assertions invoke a judge LLM per test row → cost
per test ≈ 2× a deterministic-only eval.
- Even seeded LLMs drift between provider model updates; pin model
versions in CI to bound the regression surface.
- Promptfoo ships HTML + JSON + JUnit reporters; no native
Markdown PR-comment formatter (use the GHA action for that).
## References
- [pf-intro][pf-intro] - overview
- [pf-config][pf-config] - full configuration guide
- [pf-asserts][pf-asserts] - assertion catalog
- [pf-gha][pf-gha] - GitHub Action
- [pf-cache][pf-cache] - caching mechanics
- [pf-gh][pf-gh] - install commands
- `openai-evals`, `deepeval-evaluation`, `ragas-evaluation` - sister tools (different framework styles)
Wersje
| Repozytorium | Wynik | Licencja |
|---|---|---|
| Ta strona · testland/qa-skills | 100 | MIT |
| daymade/claude-code-skills | 100 | MIT |
| testland/qa | 100 | MIT |
Migawka bazy z 13.11.2026 · wersja paczki fb59c11393a0 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100evals
Buduje harness regresji i ewaluacji dla kodu pisanego przez AI i funkcji AI: testy charakteryzujące blokują obecne zachowanie przed refaktoryzacją, a zestaw Promptfoo sprawdza chatboty, RAG i klasyfikatory w bramce wydania. - 100testing-llm-features
Buduje offline'owy zestaw ewaluacji dla funkcji opartych na LLM: złote przypadki dla szablonów promptów, deterministyczna ocena, bramka regresji i CI z promptfoo, Vitest lub deepeval. - 100accelint-eval-architect
Pomaga dodać ewaluację do skilla, agenta albo narzędzia, w tym chatbota RAG, i skonfigurować narzędzia takie jak DeepEval, Ragas albo Promptfoo. - 100eval-prompt
Ocenia prompty wnioskowania PAL za pomocą promptfoo z katalogu eval/: po zmianie lub podejrzeniu regresji, przy tworzeniu nowej ewaluacji od zera i przy porównaniu kandydata v2 z bazą. - 90agent-evaluation
Projektuje i uruchamia powtarzalne ewaluacje agentów AI, promptów, narzędzi i skilli: realistyczne zbiory, izolowane punkty odniesienia, asercje, rubryki, analiza trajektorii, koszt i porównanie regresji. - 100evals
Buduje i uruchamia ewaluacje funkcji LLM, agentów i RAG: zbiory wzorcowe, asercje, sędziów LLM i bramki regresji w CI.