Skip to content
Skills Library
AI & agents · Testing

fai-eval-driven-dev

/fai-eval-driven-dev

Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says "how do we know it's getting better".

Repository
frootai ↗
Licence
MIT
Last repository change
26.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says "how do we know it's getting better".

Triggers from the description

how do we know it
Original author description

Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says "how do we know it's getting better".

How it works

  1. Overview
  2. Parameters
  3. Steps
  4. Output
  5. WAF Alignment
  6. Compatible Solution Plays
  7. Error Handling
  8. Usage
  9. Metrics Reference
  10. Test Set Format
Original SKILL.md headings · 164 lines, 716 words
SKILL.md preview
---
name: fai-eval-driven-dev
description: Adopt evaluation-driven development for AI — define eval sets, gate changes on metrics, and iterate on prompts/models with measured feedback. Use this skill when quality regresses unpredictably, before shipping prompt changes, or to set up CI evals — even if the user only says "how do we know it's getting better".
---

# Fai Eval Driven Dev

Implements test-driven AI development — define eval metrics first, then build to pass them.

## Overview

This skill provides a structured, repeatable procedure for implements test-driven ai development — define eval metrics first, then build to pass them.. It can be used standalone as a LEGO block or auto-wired inside solution plays via the FAI Protocol.

**Category:** Evaluation
**Complexity:** Medium
**Estimated Time:** 10-30 minutes

## Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `target` | string | Yes | — | Target resource, file, or endpoint |
| `environment` | enum | No | `dev` | Target environment: `dev`, `staging`, `prod` |
| `verbose` | boolean | No | `false` | Enable detailed output logging |
| `dry_run` | boolean | No | `false` | Validate without making changes |
| `config_path` | string | No | `config/` | Path to configuration directory |

## Steps

### Step 1: Validate Prerequisites

Verify all required tools, credentials, and dependencies are available.

```bash
# Check required tools
command -v node >/dev/null 2>&1 || { echo 'Node.js required'; exit 1; }
command -v az >/dev/null 2>&1 || { echo 'Azure CLI required'; exit 1; }
```

### Step 2: Load Configuration

Read settings from the FAI manifest and TuneKit config files.

```bash
# Load from fai-manifest.json if inside a play
CONFIG_DIR="${config_path:-config}"
if [ -f "fai-manifest.json" ]; then
  echo "FAI Protocol detected — auto-wiring context"
fi
```

### Step 3: Execute Core Logic

Perform the primary operation: implements test-driven ai development — define eval metrics first, then build to pass them..

### Step 4: Validate Results

Verify the output meets quality thresholds and WAF compliance.

```bash
# Validate output
if [ "$?" -eq 0 ]; then
  echo "✅ Skill completed successfully"
else
  echo "❌ Skill failed — check logs"
  exit 1
fi
```

## Output

| Output | Type | Description |
|--------|------|-------------|
| `status` | enum | `success`, `warning`, `failure` |
| `duration_ms` | number | Execution time in milliseconds |
| `artifacts` | string[] | List of generated/modified files |
| `logs` | string | Detailed execution log |

## WAF Alignment

| Pillar | How This Skill Contributes |
|--------|---------------------------|
| responsible-ai | Validates content safety, checks for bias, enforces groundedness |
| reliability | Includes retry logic, validates outputs, provides rollback steps |

## Compatible Solution Plays

- **Play 03**
- **Play 60**

## Error Handling

| Exit Code | Meaning | Action |
|-----------|---------|--------|
| 0 | Success | Proceed to next step |
| 1 | Validation failure | Check input parameters |
| 2 | Dependency missing | Install required tools |
| 3 | Runtime error | Check logs, retry with `--verbose` |

## Usage

### Standalone

```bash
# Run this skill directly
npx frootai skill run fai-eval-driven-dev
```

### Inside a Solution Play

When referenced in `fai-manifest.json`, this skill auto-wires with the play's context:

```json
{
  "primitives": {
    "skills": ["skills/fai-eval-driven-dev/"]
  }
}
```

### Via Agent Invocation

Agents can invoke this skill using the `/skill` command in Copilot Chat.

## Metrics Reference

| Metric | Range | Threshold | Description |
|--------|-------|-----------|-------------|
| Groundedness | 0.0-1.0 | ≥ 0.85 | Answer supported by retrieved context |
| Coherence | 0.0-1.0 | ≥ 0.80 | Logical flow and consistency |
| Relevance | 0.0-1.0 | ≥ 0.80 | Answer addresses the question |
| Fluency | 0.0-1.0 | ≥ 0.75 | Natural language quality |
| Safety | 0-4 | 0 | Content safety violations |
| Faithfulness | 0.0-1.0 | ≥ 0.90 | No hallucinated facts |

## Test Set Format

```jsonl
{"question": "What is RAG?", "context": "RAG combines...", "expected": "Retrieval-Augmented Generation..."}
{"question": "How does chunking work?", "context": "Documents are split...", "expected": "Chunking divides..."}
```

## CI/CD Integration

```yaml
# .github/workflows/eval.yml
- name: Run FAI Evaluation
  run: |
    python evaluation/eval.py --test-set evaluation/test-set.jsonl
    python evaluation/check-thresholds.py --groundedness 0.85 --coherence 0.80
```

## Regression Tracking

Track evaluation scores over time to detect quality regressions:

```bash
# Compare with baseline
python evaluation/regression.py --baseline results/baseline.json --current results/latest.json
```

## Notes

- This skill follows the FAI SKILL.md specification
- All outputs are deterministic when `dry_run=true`
- Integrates with FAI Engine for automated pipeline execution
- Part of the Evaluation category in the FAI primitives catalog
Files in this package (3)
  • SKILL.lean.md 5 KB
  • SKILL.md 5 KB
  • agents/openai.yaml 1 KB

Versions

The database has no other skills named fai-eval-driven-dev.

Database snapshot: 13.11.2026 · package version 0e57d91e5aac (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100eval-driven-development
    Use this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes. Trigger on "how do I improve this prompt", "my changes keep breaking other things", "how do I know if this is better", "iterate…
  • 100eval-driven-dev
    Improve AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user…
  • 100evals-harness
    Design eval harnesses — task schemas, metrics, dataset versioning, eval-as-code patterns. Use when asked to "build an eval harness", "set up eval-as-code", or "version our eval datasets".
  • 100agentic-eval-first-development
    Architect, execute, and iterate on AI evaluations using the Data-Task-Score framework. Treats evals as the modern, quantifiable version of a PRD. Use when the user asks to "build an eval," "improve model quality," "test an agent workflow," "quantify product intuition," "move…
  • 100eval-harness
    Eval-Driven Development (EDD) framework for AI development. Use when implementing AI features, prompt engineering, or LLM integration. Keywords: eval, evaluation, test, AI, LLM, prompt, benchmark, quality, EDD.
  • 90eval-analysis
    Analyze eval results to understand signal quality and guide prompt/config changes. Use when the user says "eval-analysis", "analyze eval results", "what do the evals show", "eval regression", or asks about eval signal enrichment.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.