Skip to content
Skills Library
AI & agents · Testing

prompt-testing

/prompt-testing

Test prompts like code, with a case set, pass criteria, and regression runs on every change including model upgrades. Use when a prompt is in production and its behaviour matters.

Repository
AI-SKILLS ↗
Licence
MIT
Last repository change
05.09.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 05.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Test prompts like code, with a case set, pass criteria, and regression runs on every change including model upgrades. Use when a prompt is in production and its behaviour matters.

Original author description

Test prompts like code, with a case set, pass criteria, and regression runs on every change including model upgrades. Use when a prompt is in production and its behaviour matters.

How it works

  1. Method
  2. Boundaries
Original SKILL.md headings · 38 lines, 245 words
SKILL.md preview
---
name: prompt-testing
description: Test prompts like code, with a case set, pass criteria, and regression runs on every change including model upgrades. Use when a prompt is in production and its behaviour matters.
---

# Prompt testing

A prompt in production is code with non-deterministic behaviour, which
makes testing harder and more necessary. Without it, a model upgrade or
a small edit silently changes behaviour for every user.

## Method

1. **Define pass criteria per case.** Exact match where the output is
   structured, and a rubric or a judge where it is prose. Undefined
   criteria mean the test proves nothing.
2. **Run each case several times.** Output varies between runs, so a
   single pass may be luck, and the pass rate matters more than a single
   result (see agent-ensemble-voting).
3. **Include adversarial and boundary cases.** Empty input, very long
   input, contradictory instructions, and injection attempts (see
   llm-guardrails).
4. **Test on every model you deploy.** Behaviour differs between
   providers and versions enough that one model's results do not
   transfer.
5. **Re-run before every model upgrade.** A provider updating a model
   changes your product, and this is the only warning you will get.
6. **Track cost and latency alongside accuracy.** A prompt that is
   marginally better and twice the cost is a trade-off to make
   deliberately.
7. **Automate it in the pipeline.** A test suite that must be run
   manually is run rarely and never at the moment it matters (see
   agent-eval-design).

## Boundaries

Testing shows behaviour on the tested cases; the long tail remains
unknown. Judge-based scoring inherits the judge's biases and needs
periodic human calibration. Non-determinism means thresholds rather than
guarantees, and flaky tests need statistical handling (see
test-flakiness-budget).

Versions

RepositoryScoreLicence
This page · Amey-Thakur/AI-SKILLS90MIT
fusengine/agents100MIT

Database snapshot: 13.11.2026 · package version 2d41cee0c865 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100llm-evals
    Builds evaluation suites for LLM features and agents - error analysis on real outputs, pass criteria, code-based checks, calibrated LLM-as-judge graders, dataset splits, CI regression runs, and noise-aware comparisons with cost and latency. Use when testing prompts or models,…
  • 100ai-prompt-regression-testing
    Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence.
  • 90prompt-version
    Build prompt versioning systems — storage, A/B testing, regression tracking, rollback. Use when asked to "version our prompts", "A/B test prompts", or "roll back a prompt".
  • 100cortex-prompt
    Build a production-ready prompt package — system prompt, few-shot examples, output format, edge case handling, eval criteria. Use when asked to "prompt engineering", "build a prompt", "write a system prompt", or "improve this prompt".
  • 100agent-eval-harness
    Use to test whether an AI agent, prompt, or skill actually works before or after it ships — build a small eval set, score runs, and catch regressions when the prompt or model changes. Trigger when the user asks whether a prompt is good, wants to compare models or versions, sees…
  • 90set-up-ab-testing
    Use this to test an LLM change (new prompt, new model, new retrieval) on real traffic before rolling it out to everyone. Trigger on "A/B test my prompt", "roll out a new model safely", "compare two prompts in production", "canary this change", "does this actually improve things…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.