promptfoo-evaluation
/promptfoo-evaluation
Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating.
- Author
- testland ↗
- Repository
- qa-skills ↗
- Licence
- MIT
- Last repository change
- 12.11.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 7
No signals detected. Repository changed on 12.11.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating.
Original author description
Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating.
How it works
- When to use
- Step 1 - Install
- Step 2 - First eval
- Step 3 - Variable interpolation
- Step 4 - Assertion catalog
- Step 5 - defaultTest pattern
- Step 6 - Output transforms
- Step 7 - assert-set grouping
- Step 8 - CI integration
- Anti-patterns
SKILL.md preview
---
name: promptfoo-evaluation
description: "Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating."
---
# promptfoo-evaluation
[pf-intro]: https://promptfoo.dev/docs/intro/
[pf-config]: https://promptfoo.dev/docs/configuration/guide/
A `promptfooconfig.yaml` declares **providers** (LLMs under test),
**prompts** (templates with `{{var}}` placeholders), **tests**
(input variables + assertions); `promptfoo eval` runs the
cross-product (per [pf-config][pf-config]).
## When to use
- The repo has a `promptfooconfig.yaml` or the user wants to
author one.
- The user needs prompt regression suites - same inputs, multiple
providers, fail-on-diff.
- A CI workflow needs an eval gate on prompt or model changes.
- The team prefers vendor-neutral eval (vs OpenAI Evals' OpenAI-first
posture).
## Step 1 - Install
Per [github.com/promptfoo/promptfoo][pf-gh]:
[pf-gh]: https://github.com/promptfoo/promptfoo
```bash
npm install -g promptfoo
# or
brew install promptfoo
# or one-off
npx promptfoo@latest
```
Per [pf-gh][pf-gh]: "Node.js 20.20+ or 22.22+ is required for npm
and npx usage."
## Step 2 - First eval
Initialize from a built-in example, then run:
```bash
promptfoo init --example getting-started
cd getting-started
promptfoo eval
promptfoo view # opens HTML report
```
(Commands per [pf-gh][pf-gh].)
A minimal `promptfooconfig.yaml` per [pf-config][pf-config]:
```yaml
prompts:
- file://prompt1.txt
providers:
- openai:gpt-5-mini
- anthropic:claude-haiku-4-5
tests:
- vars:
language: French
input: Hello world
assert:
- type: contains-json
```
Provider syntax `<vendor>:<model>` works for OpenAI, Anthropic,
Vertex (`vertex:gemini-2.0-flash-exp`), Ollama (`ollama:llama2`), and
30+ others (per [pf-config][pf-config]).
## Step 3 - Variable interpolation
Vars interpolate into prompts using `{{variable}}`. Arrays create
combinations (per [pf-config][pf-config]):
```yaml
tests:
- vars:
language: [French, German, Spanish]
input: [Hello world, Good morning]
```
This generates 6 test rows (3 languages × 2 inputs). Vars can also
load from files:
```yaml
tests:
- vars:
var3: file://path/to/var3.txt
context: file://fetch_from_vector_database.py
```
## Step 4 - Assertion catalog
Per [promptfoo.dev/docs/configuration/expected-outputs/][pf-asserts]:
[pf-asserts]: https://promptfoo.dev/docs/configuration/expected-outputs/
**Deterministic (string + structure):**
| Type | Example |
|---|---|
| `equals` | `value: 'expected string'` |
| `contains` / `icontains` | `value: 'substring'` (case-sensitive / insensitive) |
| `starts-with` | `value: 'prefix'` |
| `contains-any` / `contains-all` | `value: ['a', 'b']` |
| `regex` | `value: '^pattern$'` |
| `is-json` / `contains-json` | (validates / locates JSON) |
| `is-sql` / `contains-sql` | (SQL syntax) |
| `is-xml` / `is-html` / `contains-xml` / `contains-html` | (markup) |
| `is-valid-openai-tools-call` / `is-valid-openai-function-call` | (tool calls) |
**Custom logic:**
```yaml
- type: javascript
value: 'output.length > 10'
- type: python
value: 'file://script.py'
```
**Text-quality metrics** (default thresholds per [pf-asserts][pf-asserts]):
| Type | Default threshold |
|---|---|
| `rouge-n` | 0.75 |
| `bleu` | 0.5 |
| `gleu` | 0.5 |
| `meteor` | 0.5 |
| `levenshtein` | 5 (edit distance max) |
**Performance + cost:**
```yaml
- type: latency
threshold: 200 # milliseconds
- type: cost
threshold: 0.001 # dollars per response
```
**Model-graded (LLM-as-judge):**
| Type | Use |
|---|---|
| `llm-rubric` | Free-form rubric: `value: 'Is helpful and accurate'` |
| `model-graded-closedqa` | Closed-QA evaluation method |
| `factuality` | Compares against reference facts |
| `g-eval` | Chain-of-thought scoring |
| `answer-relevance` | Checks output relates to query |
| `context-faithfulness` / `context-recall` / `context-relevance` | RAG-specific |
**Semantic similarity:**
```yaml
- type: similar
value: 'reference text'
threshold: 0.8 # cosine similarity via embeddings
```
**Negation:** all deterministic assertions support a `not-` prefix
(`not-equals`, `not-contains`, `not-regex`, etc.) per [pf-asserts][pf-asserts].
## Step 5 - defaultTest pattern
Shared assertions/vars across all tests:
```yaml
defaultTest:
vars:
shared_var: 'shared content'
assert:
- type: llm-rubric
value: does not describe self as AI
options:
provider: openai:gpt-5-mini # in CI, pin a dated snapshot id from the provider's model page
tests:
- vars:
unique_var: value1
```
(Per [pf-config][pf-config].)
## Step 6 - Output transforms
Modify LLM output before assertions execute (per [pf-config][pf-config]):
```yaml
tests:
- vars:
body: Hello world
options:
transform: output.toUpperCase()
```
Or load from a file:
```yaml
options:
transform: file://transform.js:customTransform
```
## Step 7 - assert-set grouping
Group assertions and require a threshold percentage to pass (per
[pf-asserts][pf-asserts]):
```yaml
assert:
- type: assert-set
threshold: 0.5 # 50% of grouped asserts must pass
assert:
- type: cost
threshold: 0.001
- type: latency
threshold: 200
```
## Step 8 - CI integration
Provider API keys via env vars (per [pf-gh][pf-gh]: `export
OPENAI_API_KEY=sk-abc123`).
GitHub Actions pattern (per [promptfoo.dev/docs/integrations/github-action][pf-gha]):
[pf-gha]: https://promptfoo.dev/docs/integrations/github-action/
```yaml
- uses: promptfoo/promptfoo-action@v2
with:
openai-api-key: ${{ secrets.OPENAI_API_KEY }}
config: 'promptfooconfig.yaml'
cache-path: '~/.cache/promptfoo'
use-config-prompts: false
no-share: true
promptfoo-version: 'latest'
```
The action posts a PR comment with regression diff vs the base
branch. Caching reuses LLM responses for unchanged tests (per
[promptfoo.dev/docs/configuration/caching/][pf-cache]).
[pf-cache]: https://promptfoo.dev/docs/configuration/caching/
[oa-models]: https://developers.openai.com/api/docs/models/gpt-5-mini
## Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Only deterministic assertions on creative outputs | LLM responses vary; rigid asserts produce flake | Use `llm-rubric` or `similar` (Step 4) |
| Single provider in config | Misses cross-provider regression | At least 2 providers per eval (Step 2) |
| No cost / latency caps | Eval cost balloons per PR | `cost` + `latency` asserts (Step 4) |
| `assert-set` with `threshold: 0` | Effectively disables grouping | Pick a real threshold (Step 7) |
| Fresh provider model on every run without pinning | Output drifts when vendor updates model | Pin a dated snapshot id from the provider's model page ([OpenAI's lists them per model][oa-models]), not a moving alias |
| Skip caching in CI | Re-runs every test on every PR | `cache-path` in GHA action (Step 8) |
## Limitations
- Model-graded assertions invoke a judge LLM per test row → cost
per test ≈ 2× a deterministic-only eval.
- Even seeded LLMs drift between provider model updates; pin model
versions in CI to bound the regression surface.
- Promptfoo ships HTML + JSON + JUnit reporters; no native
Markdown PR-comment formatter (use the GHA action for that).
## References
- [pf-intro][pf-intro] - overview
- [pf-config][pf-config] - full configuration guide
- [pf-asserts][pf-asserts] - assertion catalog
- [pf-gha][pf-gha] - GitHub Action
- [pf-cache][pf-cache] - caching mechanics
- [pf-gh][pf-gh] - install commands
- `openai-evals`, `deepeval-evaluation`, `ragas-evaluation` - sister tools (different framework styles)
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · testland/qa-skills | 100 | MIT |
| daymade/claude-code-skills | 100 | MIT |
| testland/qa | 100 | MIT |
Database snapshot: 13.11.2026 · package version fb59c11393a0 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100evals
Build a regression + eval harness for AI-written code and AI features. Generates characterization tests that lock current behavior before a refactor, scaffolds a Promptfoo eval suite for chatbots/RAG/classifiers, and wires it into the ship-gate. Use when the user wants evals,… - 100testing-llm-features
Builds an offline eval suite for LLM-powered features: golden reference cases per prompt template, deterministic scoring where possible, a regression gate on prompt, model and retrieval changes, and CI wiring with promptfoo, Vitest or deepeval. Use when a product ships prompts,… - 100accelint-eval-architect
Use when users say "add an eval to this skill", "evaluate this skill/agent", "evaluate this tool/repo/pipeline", "how should I test this RAG bot / retrieval / parser", "set up DeepEval/Ragas", "should I use Promptfoo/Inspect", "is my eval any good", "audit my eval harness" — or… - 100eval-prompt
Evaluate a PAL inference prompt using the promptfoo harness in eval/. Use when a prompt constant in src/ has changed or may have regressed, when scaffolding a new prompt eval from scratch, or when comparing a v2 candidate against the current baseline. - 90agent-evaluation
Designs and runs reproducible evaluations for AI agents, prompts, tools, skills, and model-backed workflows using realistic datasets, isolated baselines, objective assertions, rubric grading, trajectory analysis, cost/latency tracking, and regression comparison. Use when… - 100evals
Build and run evaluations for LLM features, prompts, agents, and RAG pipelines — golden datasets, assertions, LLM-as-judge graders, regression gates in CI, A/B comparison of prompts or models. Use whenever someone changes a prompt, swaps a model, ships an AI feature, asks \"is…