Skip to content
Skills Library
AI & agents · Testing

promptfoo-evaluation

/promptfoo-evaluation

Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating.

Repository
qa-skills ↗
Licence
MIT
Last repository change
12.11.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
7
Deprecation

No signals detected. Repository changed on 12.11.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating.

Original author description

Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating.

How it works

  1. When to use
  2. Step 1 - Install
  3. Step 2 - First eval
  4. Step 3 - Variable interpolation
  5. Step 4 - Assertion catalog
  6. Step 5 - defaultTest pattern
  7. Step 6 - Output transforms
  8. Step 7 - assert-set grouping
  9. Step 8 - CI integration
  10. Anti-patterns
Original SKILL.md headings · 278 lines, 965 words
SKILL.md preview
---
name: promptfoo-evaluation
description: "Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating."
---

# promptfoo-evaluation

[pf-intro]: https://promptfoo.dev/docs/intro/
[pf-config]: https://promptfoo.dev/docs/configuration/guide/

A `promptfooconfig.yaml` declares **providers** (LLMs under test),
**prompts** (templates with `{{var}}` placeholders), **tests**
(input variables + assertions); `promptfoo eval` runs the
cross-product (per [pf-config][pf-config]).

## When to use

- The repo has a `promptfooconfig.yaml` or the user wants to
  author one.
- The user needs prompt regression suites - same inputs, multiple
  providers, fail-on-diff.
- A CI workflow needs an eval gate on prompt or model changes.
- The team prefers vendor-neutral eval (vs OpenAI Evals' OpenAI-first
  posture).

## Step 1 - Install

Per [github.com/promptfoo/promptfoo][pf-gh]:

[pf-gh]: https://github.com/promptfoo/promptfoo

```bash
npm install -g promptfoo
# or
brew install promptfoo
# or one-off
npx promptfoo@latest
```

Per [pf-gh][pf-gh]: "Node.js 20.20+ or 22.22+ is required for npm
and npx usage."

## Step 2 - First eval

Initialize from a built-in example, then run:

```bash
promptfoo init --example getting-started
cd getting-started
promptfoo eval
promptfoo view  # opens HTML report
```

(Commands per [pf-gh][pf-gh].)

A minimal `promptfooconfig.yaml` per [pf-config][pf-config]:

```yaml
prompts:
  - file://prompt1.txt

providers:
  - openai:gpt-5-mini
  - anthropic:claude-haiku-4-5

tests:
  - vars:
      language: French
      input: Hello world
    assert:
      - type: contains-json
```

Provider syntax `<vendor>:<model>` works for OpenAI, Anthropic,
Vertex (`vertex:gemini-2.0-flash-exp`), Ollama (`ollama:llama2`), and
30+ others (per [pf-config][pf-config]).

## Step 3 - Variable interpolation

Vars interpolate into prompts using `{{variable}}`. Arrays create
combinations (per [pf-config][pf-config]):

```yaml
tests:
  - vars:
      language: [French, German, Spanish]
      input: [Hello world, Good morning]
```

This generates 6 test rows (3 languages × 2 inputs). Vars can also
load from files:

```yaml
tests:
  - vars:
      var3: file://path/to/var3.txt
      context: file://fetch_from_vector_database.py
```

## Step 4 - Assertion catalog

Per [promptfoo.dev/docs/configuration/expected-outputs/][pf-asserts]:

[pf-asserts]: https://promptfoo.dev/docs/configuration/expected-outputs/

**Deterministic (string + structure):**

| Type | Example |
|---|---|
| `equals` | `value: 'expected string'` |
| `contains` / `icontains` | `value: 'substring'` (case-sensitive / insensitive) |
| `starts-with` | `value: 'prefix'` |
| `contains-any` / `contains-all` | `value: ['a', 'b']` |
| `regex` | `value: '^pattern$'` |
| `is-json` / `contains-json` | (validates / locates JSON) |
| `is-sql` / `contains-sql` | (SQL syntax) |
| `is-xml` / `is-html` / `contains-xml` / `contains-html` | (markup) |
| `is-valid-openai-tools-call` / `is-valid-openai-function-call` | (tool calls) |

**Custom logic:**

```yaml
- type: javascript
  value: 'output.length > 10'
- type: python
  value: 'file://script.py'
```

**Text-quality metrics** (default thresholds per [pf-asserts][pf-asserts]):

| Type | Default threshold |
|---|---|
| `rouge-n` | 0.75 |
| `bleu` | 0.5 |
| `gleu` | 0.5 |
| `meteor` | 0.5 |
| `levenshtein` | 5 (edit distance max) |

**Performance + cost:**

```yaml
- type: latency
  threshold: 200    # milliseconds
- type: cost
  threshold: 0.001  # dollars per response
```

**Model-graded (LLM-as-judge):**

| Type | Use |
|---|---|
| `llm-rubric` | Free-form rubric: `value: 'Is helpful and accurate'` |
| `model-graded-closedqa` | Closed-QA evaluation method |
| `factuality` | Compares against reference facts |
| `g-eval` | Chain-of-thought scoring |
| `answer-relevance` | Checks output relates to query |
| `context-faithfulness` / `context-recall` / `context-relevance` | RAG-specific |

**Semantic similarity:**

```yaml
- type: similar
  value: 'reference text'
  threshold: 0.8   # cosine similarity via embeddings
```

**Negation:** all deterministic assertions support a `not-` prefix
(`not-equals`, `not-contains`, `not-regex`, etc.) per [pf-asserts][pf-asserts].

## Step 5 - defaultTest pattern

Shared assertions/vars across all tests:

```yaml
defaultTest:
  vars:
    shared_var: 'shared content'
  assert:
    - type: llm-rubric
      value: does not describe self as AI
  options:
    provider: openai:gpt-5-mini   # in CI, pin a dated snapshot id from the provider's model page

tests:
  - vars:
      unique_var: value1
```

(Per [pf-config][pf-config].)

## Step 6 - Output transforms

Modify LLM output before assertions execute (per [pf-config][pf-config]):

```yaml
tests:
  - vars:
      body: Hello world
    options:
      transform: output.toUpperCase()
```

Or load from a file:

```yaml
options:
  transform: file://transform.js:customTransform
```

## Step 7 - assert-set grouping

Group assertions and require a threshold percentage to pass (per
[pf-asserts][pf-asserts]):

```yaml
assert:
  - type: assert-set
    threshold: 0.5   # 50% of grouped asserts must pass
    assert:
      - type: cost
        threshold: 0.001
      - type: latency
        threshold: 200
```

## Step 8 - CI integration

Provider API keys via env vars (per [pf-gh][pf-gh]: `export
OPENAI_API_KEY=sk-abc123`).

GitHub Actions pattern (per [promptfoo.dev/docs/integrations/github-action][pf-gha]):

[pf-gha]: https://promptfoo.dev/docs/integrations/github-action/

```yaml
- uses: promptfoo/promptfoo-action@v2
  with:
    openai-api-key: ${{ secrets.OPENAI_API_KEY }}
    config: 'promptfooconfig.yaml'
    cache-path: '~/.cache/promptfoo'
    use-config-prompts: false
    no-share: true
    promptfoo-version: 'latest'
```

The action posts a PR comment with regression diff vs the base
branch. Caching reuses LLM responses for unchanged tests (per
[promptfoo.dev/docs/configuration/caching/][pf-cache]).

[pf-cache]: https://promptfoo.dev/docs/configuration/caching/
[oa-models]: https://developers.openai.com/api/docs/models/gpt-5-mini

## Anti-patterns

| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Only deterministic assertions on creative outputs | LLM responses vary; rigid asserts produce flake | Use `llm-rubric` or `similar` (Step 4) |
| Single provider in config | Misses cross-provider regression | At least 2 providers per eval (Step 2) |
| No cost / latency caps | Eval cost balloons per PR | `cost` + `latency` asserts (Step 4) |
| `assert-set` with `threshold: 0` | Effectively disables grouping | Pick a real threshold (Step 7) |
| Fresh provider model on every run without pinning | Output drifts when vendor updates model | Pin a dated snapshot id from the provider's model page ([OpenAI's lists them per model][oa-models]), not a moving alias |
| Skip caching in CI | Re-runs every test on every PR | `cache-path` in GHA action (Step 8) |

## Limitations

- Model-graded assertions invoke a judge LLM per test row → cost
  per test ≈ 2× a deterministic-only eval.
- Even seeded LLMs drift between provider model updates; pin model
  versions in CI to bound the regression surface.
- Promptfoo ships HTML + JSON + JUnit reporters; no native
  Markdown PR-comment formatter (use the GHA action for that).

## References

- [pf-intro][pf-intro] - overview
- [pf-config][pf-config] - full configuration guide
- [pf-asserts][pf-asserts] - assertion catalog
- [pf-gha][pf-gha] - GitHub Action
- [pf-cache][pf-cache] - caching mechanics
- [pf-gh][pf-gh] - install commands
- `openai-evals`, `deepeval-evaluation`, `ragas-evaluation` - sister tools (different framework styles)

Versions

RepositoryScoreLicence
This page · testland/qa-skills100MIT
daymade/claude-code-skills100MIT
testland/qa100MIT

Database snapshot: 13.11.2026 · package version fb59c11393a0 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100evals
    Build a regression + eval harness for AI-written code and AI features. Generates characterization tests that lock current behavior before a refactor, scaffolds a Promptfoo eval suite for chatbots/RAG/classifiers, and wires it into the ship-gate. Use when the user wants evals,…
  • 100testing-llm-features
    Builds an offline eval suite for LLM-powered features: golden reference cases per prompt template, deterministic scoring where possible, a regression gate on prompt, model and retrieval changes, and CI wiring with promptfoo, Vitest or deepeval. Use when a product ships prompts,…
  • 100accelint-eval-architect
    Use when users say "add an eval to this skill", "evaluate this skill/agent", "evaluate this tool/repo/pipeline", "how should I test this RAG bot / retrieval / parser", "set up DeepEval/Ragas", "should I use Promptfoo/Inspect", "is my eval any good", "audit my eval harness" — or…
  • 100eval-prompt
    Evaluate a PAL inference prompt using the promptfoo harness in eval/. Use when a prompt constant in src/ has changed or may have regressed, when scaffolding a new prompt eval from scratch, or when comparing a v2 candidate against the current baseline.
  • 90agent-evaluation
    Designs and runs reproducible evaluations for AI agents, prompts, tools, skills, and model-backed workflows using realistic datasets, isolated baselines, objective assertions, rubric grading, trajectory analysis, cost/latency tracking, and regression comparison. Use when…
  • 100evals
    Build and run evaluations for LLM features, prompts, agents, and RAG pipelines — golden datasets, assertions, LLM-as-judge graders, regression gates in CI, A/B comparison of prompts or models. Use whenever someone changes a prompt, swaps a model, ships an AI feature, asks \"is…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.