fai-eval-driven-dev
/fai-eval-driven-dev
Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says "how do we know it's getting better".
Assessment and security
- Files in this package
- 3
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says "how do we know it's getting better".
Triggers from the description
Original author description
Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says "how do we know it's getting better".
How it works
- Overview
- Parameters
- Steps
- Output
- WAF Alignment
- Compatible Solution Plays
- Error Handling
- Usage
- Metrics Reference
- Test Set Format
SKILL.md preview
---
name: fai-eval-driven-dev
description: Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says "how do we know it's getting better".
---
# Fai Eval Driven Dev
Implements test-driven AI development — define eval metrics first, then build to pass them.
## Overview
This skill provides a structured, repeatable procedure for implements test-driven ai development — define eval metrics first, then build to pass them.. It can be used standalone as a LEGO block or auto-wired inside solution plays via the FAI Protocol.
**Category:** Evaluation
**Complexity:** Medium
**Estimated Time:** 10-30 minutes
## Parameters
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `target` | string | Yes | — | Target resource, file, or endpoint |
| `environment` | enum | No | `dev` | Target environment: `dev`, `staging`, `prod` |
| `verbose` | boolean | No | `false` | Enable detailed output logging |
| `dry_run` | boolean | No | `false` | Validate without making changes |
| `config_path` | string | No | `config/` | Path to configuration directory |
## Steps
### Step 1: Validate Prerequisites
Verify all required tools, credentials, and dependencies are available.
```bash
# Check required tools
command -v node >/dev/null 2>&1 || { echo 'Node.js required'; exit 1; }
command -v az >/dev/null 2>&1 || { echo 'Azure CLI required'; exit 1; }
```
### Step 2: Load Configuration
Read settings from the FAI manifest and TuneKit config files.
```bash
# Load from fai-manifest.json if inside a play
CONFIG_DIR="${config_path:-config}"
if [ -f "fai-manifest.json" ]; then
echo "FAI Protocol detected — auto-wiring context"
fi
```
### Step 3: Execute Core Logic
Perform the primary operation: implements test-driven ai development — define eval metrics first, then build to pass them..
### Step 4: Validate Results
Verify the output meets quality thresholds and WAF compliance.
```bash
# Validate output
if [ "$?" -eq 0 ]; then
echo "✅ Skill completed successfully"
else
echo "❌ Skill failed — check logs"
exit 1
fi
```
## Output
| Output | Type | Description |
|--------|------|-------------|
| `status` | enum | `success`, `warning`, `failure` |
| `duration_ms` | number | Execution time in milliseconds |
| `artifacts` | string[] | List of generated/modified files |
| `logs` | string | Detailed execution log |
## WAF Alignment
| Pillar | How This Skill Contributes |
|--------|---------------------------|
| responsible-ai | Validates content safety, checks for bias, enforces groundedness |
| reliability | Includes retry logic, validates outputs, provides rollback steps |
## Compatible Solution Plays
- **Play 03**
- **Play 60**
## Error Handling
| Exit Code | Meaning | Action |
|-----------|---------|--------|
| 0 | Success | Proceed to next step |
| 1 | Validation failure | Check input parameters |
| 2 | Dependency missing | Install required tools |
| 3 | Runtime error | Check logs, retry with `--verbose` |
## Usage
### Standalone
```bash
# Run this skill directly
npx frootai skill run fai-eval-driven-dev
```
### Inside a Solution Play
When referenced in `fai-manifest.json`, this skill auto-wires with the play's context:
```json
{
"primitives": {
"skills": ["skills/fai-eval-driven-dev/"]
}
}
```
### Via Agent Invocation
Agents can invoke this skill using the `/skill` command in Copilot Chat.
## Metrics Reference
| Metric | Range | Threshold | Description |
|--------|-------|-----------|-------------|
| Groundedness | 0.0-1.0 | ≥ 0.85 | Answer supported by retrieved context |
| Coherence | 0.0-1.0 | ≥ 0.80 | Logical flow and consistency |
| Relevance | 0.0-1.0 | ≥ 0.80 | Answer addresses the question |
| Fluency | 0.0-1.0 | ≥ 0.75 | Natural language quality |
| Safety | 0-4 | 0 | Content safety violations |
| Faithfulness | 0.0-1.0 | ≥ 0.90 | No hallucinated facts |
## Test Set Format
```jsonl
{"question": "What is RAG?", "context": "RAG combines...", "expected": "Retrieval-Augmented Generation..."}
{"question": "How does chunking work?", "context": "Documents are split...", "expected": "Chunking divides..."}
```
## CI/CD Integration
```yaml
# .github/workflows/eval.yml
- name: Run FAI Evaluation
run: |
python evaluation/eval.py --test-set evaluation/test-set.jsonl
python evaluation/check-thresholds.py --groundedness 0.85 --coherence 0.80
```
## Regression Tracking
Track evaluation scores over time to detect quality regressions:
```bash
# Compare with baseline
python evaluation/regression.py --baseline results/baseline.json --current results/latest.json
```
## Notes
- This skill follows the FAI SKILL.md specification
- All outputs are deterministic when `dry_run=true`
- Integrates with FAI Engine for automated pipeline execution
- Part of the Evaluation category in the FAI primitives catalog
Files in this package (3)
- SKILL.lean.md 5 KB
- SKILL.md 5 KB
- agents/openai.yaml 1 KB
Versions
The database has no other skills named fai-eval-driven-dev.
Database snapshot: 13.11.2026 · package version 0e57d91e5aac (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100eval-driven-development
Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate… - 100eval-driven-dev
Improve AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user… - 100evals-harness
Design eval harnesses — task schemas, metrics, dataset versioning, eval-as-code patterns. Use when asked to "build an eval harness", "set up eval-as-code", or "version our eval datasets". - 100agentic-eval-first-development
Architect, execute, and iterate on AI evaluations using the Data-Task-Score framework. Treats evals as the modern, quantifiable version of a PRD. Use when the user asks to "build an eval," "improve model quality," "test an agent workflow," "quantify product intuition," "move… - 100eval-harness
Eval-Driven Development (EDD) framework for AI development. Use when implementing AI features, prompt engineering, or LLM integration. Keywords: eval, evaluation, test, AI, LLM, prompt, benchmark, quality, EDD. - 90eval-analysis
Analyze eval results to understand signal quality and guide prompt/config changes. Use when the user says "eval-analysis", "analyze eval results", "what do the evals show", "eval regression", or asks about eval signal enrichment.