evals-context
/evals-context
Provides context about the CoStrict evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).
- Author
- zgsm-ai ↗
- Repository
- costrict ↗
- Licence
- Apache-2.0
- Last repository change
- 30.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 3
No signals detected. Repository changed on 30.09.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Provides context about the CoStrict evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).
Triggers from the description
Original author description
Provides context about the CoStrict evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).
How it works
- When to Use This Skill
- When NOT to Use This Skill
- Key Disambiguation: Two "Evals" Locations
- Directory Structure Reference
- Architecture Overview
- Common Tasks Quick Reference
- Running Evals Locally
- Testing
- Key Types/Exports from @roo-code/evals
SKILL.md preview
---
name: evals-context
description: Provides context about the CoStrict evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).
---
# Evals Codebase Context
## When to Use This Skill
Use this skill when the task involves:
- Modifying or debugging the evals execution infrastructure
- Adding new eval exercises or languages
- Working with the evals web interface (apps/web-evals)
- Modifying the public evals display page on roocode.com
- Understanding where evals code lives in this monorepo
## When NOT to Use This Skill
Do NOT use this skill when:
- Working on unrelated parts of the codebase (extension, webview-ui, etc.)
- The task is purely about the VS Code extension's core functionality
- Working on the main website pages that don't involve evals
## Key Disambiguation: Two "Evals" Locations
This monorepo has **two distinct evals-related locations** that can cause confusion:
| Component | Path | Purpose |
| --------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------- |
| **Evals Execution System** | `packages/evals/` | Core eval infrastructure: CLI, DB schema, Docker configs |
| **Evals Management UI** | `apps/web-evals/` | Next.js app for creating/monitoring eval runs (localhost:3446) |
| **Website Evals Page** | `apps/web-roo-code/src/app/evals/` | Public roocode.com page displaying eval results |
| **External Exercises Repo** | [Roo-Code-Evals](https://github.com/RooCodeInc/Roo-Code-Evals) | Actual coding exercises (NOT in this monorepo) |
## Directory Structure Reference
### `packages/evals/` - Core Evals Package
```
packages/evals/
├── ARCHITECTURE.md # Detailed architecture documentation
├── ADDING-EVALS.md # Guide for adding new exercises/languages
├── README.md # Setup and running instructions
├── docker-compose.yml # Container orchestration
├── Dockerfile.runner # Runner container definition
├── Dockerfile.web # Web app container
├── drizzle.config.ts # Database ORM config
├── src/
│ ├── index.ts # Package exports
│ ├── cli/ # CLI commands for running evals
│ │ ├── runEvals.ts # Orchestrates complete eval runs
│ │ ├── runTask.ts # Executes individual tasks in containers
│ │ ├── runUnitTest.ts # Validates task completion via tests
│ │ └── redis.ts # Redis pub/sub integration
│ ├── db/
│ │ ├── schema.ts # Database schema (runs, tasks)
│ │ ├── queries/ # Database query functions
│ │ └── migrations/ # SQL migrations
│ └── exercises/
│ └── index.ts # Exercise loading utilities
└── scripts/
└── setup.sh # Local macOS setup script
```
### `apps/web-evals/` - Evals Management Web App
```
apps/web-evals/
├── src/
│ ├── app/
│ │ ├── page.tsx # Home page (runs list)
│ │ ├── runs/
│ │ │ ├── new/ # Create new eval run
│ │ │ └── [id]/ # View specific run status
│ │ └── api/runs/ # SSE streaming endpoint
│ ├── actions/ # Server actions
│ │ ├── runs.ts # Run CRUD operations
│ │ ├── tasks.ts # Task queries
│ │ ├── exercises.ts # Exercise listing
│ │ └── heartbeat.ts # Controller health checks
│ ├── hooks/ # React hooks (SSE, models, etc.)
│ └── lib/ # Utilities and schemas
```
### `apps/web-roo-code/src/app/evals/` - Public Website Evals Page
```
apps/web-roo-code/src/app/evals/
├── page.tsx # Fetches and displays public eval results
├── evals.tsx # Main evals display component
├── plot.tsx # Visualization component
└── types.ts # EvalRun type (extends packages/evals types)
```
This page **displays** eval results on the public roocode.com website. It imports types from `@roo-code/evals` but does NOT run evals.
## Architecture Overview
The evals system is a distributed evaluation platform that runs AI coding tasks in isolated VS Code environments:
```
┌─────────────────────────────────────────────────────────────┐
│ Web App (apps/web-evals) ──────────────────────────────── │
│ │ │
│ ▼ │
│ PostgreSQL ◄────► Controller Container │
│ │ │ │
│ ▼ ▼ │
│ Redis ◄───► Runner Containers (1-25 parallel) │
└─────────────────────────────────────────────────────────────┘
```
**Key components:**
- **Controller**: Orchestrates eval runs, spawns runners, manages task queue (p-queue)
- **Runner**: Isolated Docker container with VS Code + CoStrict extension + language runtimes
- **Redis**: Pub/sub for real-time events (NOT task queuing)
- **PostgreSQL**: Stores runs, tasks, metrics
## Common Tasks Quick Reference
### Adding a New Eval Exercise
1. Add exercise to [Roo-Code-Evals](https://github.com/RooCodeInc/Roo-Code-Evals) repo (external)
2. See [`packages/evals/ADDING-EVALS.md`](packages/evals/ADDING-EVALS.md) for structure
### Modifying Eval CLI Behavior
Edit files in [`packages/evals/src/cli/`](packages/evals/src/cli/):
- [`runEvals.ts`](packages/evals/src/cli/runEvals.ts) - Run orchestration
- [`runTask.ts`](packages/evals/src/cli/runTask.ts) - Task execution
- [`runUnitTest.ts`](packages/evals/src/cli/runUnitTest.ts) - Test validation
### Modifying the Evals Web Interface
Edit files in [`apps/web-evals/src/`](apps/web-evals/src/):
- [`app/runs/new/new-run.tsx`](apps/web-evals/src/app/runs/new/new-run.tsx) - New run form
- [`actions/runs.ts`](apps/web-evals/src/actions/runs.ts) - Run server actions
### Modifying the Public Evals Display Page
Edit files in [`apps/web-roo-code/src/app/evals/`](apps/web-roo-code/src/app/evals/):
- [`evals.tsx`](apps/web-roo-code/src/app/evals/evals.tsx) - Display component
- [`plot.tsx`](apps/web-roo-code/src/app/evals/plot.tsx) - Charts
### Database Schema Changes
1. Edit [`packages/evals/src/db/schema.ts`](packages/evals/src/db/schema.ts)
2. Generate migration: `cd packages/evals && pnpm drizzle-kit generate`
3. Apply migration: `pnpm drizzle-kit migrate`
## Running Evals Locally
```bash
# From repo root
pnpm evals
# Opens web UI at http://localhost:3446
```
**Ports (defaults):**
- PostgreSQL: 5433
- Redis: 6380
- Web: 3446
## Testing
```bash
# packages/evals tests
cd packages/evals && npx vitest run
# apps/web-evals tests
cd apps/web-evals && npx vitest run
```
## Key Types/Exports from `@roo-code/evals`
The package exports are defined in [`packages/evals/src/index.ts`](packages/evals/src/index.ts):
- Database queries: `getRuns`, `getTasks`, `getTaskMetrics`, etc.
- Schema types: `Run`, `Task`, `TaskMetrics`
- Used by both `apps/web-evals` and `apps/web-roo-code`
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · zgsm-ai/costrict | 100 | Apache-2.0 |
| roocodeinc/roo-code | 100 | Apache-2.0 |
Identical content appears in 1 other GitHub folders. We show one copy with the highest score.
Database snapshot: 13.11.2026 · package version 13dbc767baea (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100skill-eval-scaffolder
Scaffold evals/evals.json manifests for repo skills with baseline cases. Use when adding behavioral eval coverage. NOT for live LLM eval execution. - 100agent-eval-tests
Use when writing, editing, or reviewing evalite-scored agent evals in packages/core/compute/assistant-evals/src/evals. Use when creating new eval files, adding deterministic assertions or an LLM-judge scorer, or fixing a failing/mis-scoring eval. - 84design
Design an evaluation suite for an LLM-based application or a Claude Code skill: interview for measurable success criteria, pick a grading method per criterion, and scaffold a criteria doc plus eval cases into the consumer repo. Use when: 'design evals', 'create an eval suite',… - 100evals-harness
Design eval harnesses — task schemas, metrics, dataset versioning, eval-as-code patterns. Use when asked to "build an eval harness", "set up eval-as-code", or "version our eval datasets". - 100skill-eval
Measure whether a skill actually helps: build a `claude plugin eval` suite, run it with and without the skill under a required spend cap, and read the delta. Use when adding evals to a skill, converting evals.json, re-checking skills after a model upgrade, or deciding if a skill… - 100evals-analyze
Analyze LLM eval results — score breakdowns by category, regression detection vs baseline, failure clustering. Use when asked to "analyze our eval results", "did the model regress", or "cluster the eval failures".