prompt-testing
/prompt-testing
Test prompts like code, with a case set, pass criteria, and regression runs on every change including model upgrades. Use when a prompt is in production and its behaviour matters.
- Author
- Amey-Thakur ↗
- Repository
- AI-SKILLS ↗
- Licence
- MIT
- Last repository change
- 05.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 05.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Test prompts like code, with a case set, pass criteria, and regression runs on every change including model upgrades. Use when a prompt is in production and its behaviour matters.
Original author description
Test prompts like code, with a case set, pass criteria, and regression runs on every change including model upgrades. Use when a prompt is in production and its behaviour matters.
How it works
- Method
- Boundaries
SKILL.md preview
--- name: prompt-testing description: Test prompts like code, with a case set, pass criteria, and regression runs on every change including model upgrades. Use when a prompt is in production and its behaviour matters. --- # Prompt testing A prompt in production is code with non-deterministic behaviour, which makes testing harder and more necessary. Without it, a model upgrade or a small edit silently changes behaviour for every user. ## Method 1. **Define pass criteria per case.** Exact match where the output is structured, and a rubric or a judge where it is prose. Undefined criteria mean the test proves nothing. 2. **Run each case several times.** Output varies between runs, so a single pass may be luck, and the pass rate matters more than a single result (see agent-ensemble-voting). 3. **Include adversarial and boundary cases.** Empty input, very long input, contradictory instructions, and injection attempts (see llm-guardrails). 4. **Test on every model you deploy.** Behaviour differs between providers and versions enough that one model's results do not transfer. 5. **Re-run before every model upgrade.** A provider updating a model changes your product, and this is the only warning you will get. 6. **Track cost and latency alongside accuracy.** A prompt that is marginally better and twice the cost is a trade-off to make deliberately. 7. **Automate it in the pipeline.** A test suite that must be run manually is run rarely and never at the moment it matters (see agent-eval-design). ## Boundaries Testing shows behaviour on the tested cases; the long tail remains unknown. Judge-based scoring inherits the judge's biases and needs periodic human calibration. Non-determinism means thresholds rather than guarantees, and flaky tests need statistical handling (see test-flakiness-budget).
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · Amey-Thakur/AI-SKILLS | 90 | MIT |
| fusengine/agents | 100 | MIT |
Database snapshot: 13.11.2026 · package version 2d41cee0c865 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100llm-evals
Builds evaluation suites for LLM features and agents - error analysis on real outputs, pass criteria, code-based checks, calibrated LLM-as-judge graders, dataset splits, CI regression runs, and noise-aware comparisons with cost and latency. Use when testing prompts or models,… - 100ai-prompt-regression-testing
Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence. - 90prompt-version
Build prompt versioning systems — storage, A/B testing, regression tracking, rollback. Use when asked to "version our prompts", "A/B test prompts", or "roll back a prompt". - 100cortex-prompt
Build a production-ready prompt package — system prompt, few-shot examples, output format, edge case handling, eval criteria. Use when asked to "prompt engineering", "build a prompt", "write a system prompt", or "improve this prompt". - 100agent-eval-harness
Use to test whether an AI agent, prompt, or skill actually works before or after it ships — build a small eval set, score runs, and catch regressions when the prompt or model changes. Trigger when the user asks whether a prompt is good, wants to compare models or versions, sees… - 90set-up-ab-testing
Use this to test an LLM change (new prompt, new model, new retrieval) on real traffic before rolling it out to everyone. Trigger on "A/B test my prompt", "roll out a new model safely", "compare two prompts in production", "canary this change", "does this actually improve things…