create-evaluator
/create-evaluator
Scaffolds a new Evaluator for the Dokimos LLM evaluation framework. Use this skill whenever the user wants to create, add, or implement a new evaluator, metric, or scoring function for evaluating LLM outputs. Also use when the user mentions adding evaluation criteria, custom metrics, or grading logic.
- Author
- dokimos-dev ↗
- Repository
- dokimos ↗
- Licence
- MIT
- Last repository change
- 12.10.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 12.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Scaffolds a new Evaluator for the Dokimos LLM evaluation framework. Use this skill whenever the user wants to create, add, or implement a new evaluator, metric, or scoring function for evaluating LLM outputs. Also use when the user mentions adding evaluation criteria, custom metrics, or grading logic.
Original author description
Scaffolds a new Evaluator for the Dokimos LLM evaluation framework. Use this skill whenever the user wants to create, add, or implement a new evaluator, metric, or scoring function for evaluating LLM outputs. Also use when the user mentions adding evaluation criteria, custom metrics, or grading logic.
How it works
- Where things live
- Evaluator structure
- Test structure
- Conventions checklist
SKILL.md preview
---
name: create-evaluator
description: Scaffolds a new Evaluator for the Dokimos LLM evaluation framework. Use this skill whenever the user wants to create, add, or implement a new evaluator, metric, or scoring function for evaluating LLM outputs. Also use when the user mentions adding evaluation criteria, custom metrics, or grading logic.
---
# Create Evaluator
Scaffold a new Evaluator implementation for Dokimos following project conventions.
The user will describe what the evaluator should do via `$ARGUMENTS`. Use that description to determine the evaluator's name, scoring logic, which test case parameters it requires, and what constitutes a passing score.
## Where things live
- **Evaluator source**: `dokimos-core/src/main/java/dev/dokimos/core/evaluators/`
- **Tests**: `dokimos-core/src/test/java/dev/dokimos/core/`
- **Base class**: `dev.dokimos.core.BaseEvaluator`
- **Key types**: `EvalResult`, `EvalTestCase`, `EvalTestCaseParam`, `JudgeLM`
Before writing code, read these files to understand the current conventions:
- `dokimos-core/src/main/java/dev/dokimos/core/BaseEvaluator.java`
- `dokimos-core/src/main/java/dev/dokimos/core/evaluators/ExactMatchEvaluator.java` (simple evaluator reference)
- `dokimos-core/src/main/java/dev/dokimos/core/evaluators/LLMJudgeEvaluator.java` (LLM-based evaluator reference, if the new evaluator needs a JudgeLM)
## Evaluator structure
Every evaluator follows this pattern:
1. **Extends `BaseEvaluator`** with a private constructor that takes a `Builder`
2. **Implements `runEvaluation(EvalTestCase testCase)`** — this is where the scoring logic goes
3. **Has a static `Builder` class** with sensible defaults for name, threshold, and evaluationParams
4. **Returns an `EvalResult`** via `EvalResult.builder().name(name).score(score).threshold(threshold).reason(reason).build()`
The `evaluationParams` field declares which `EvalTestCaseParam` values the evaluator requires (INPUT, ACTUAL_OUTPUT, EXPECTED_OUTPUT). `BaseEvaluator` validates these are non-null before calling `runEvaluation`.
If the evaluator needs an LLM to judge (semantic similarity, faithfulness, etc.), accept a `JudgeLM` in the builder. `JudgeLM` is a functional interface: `String generate(String prompt)`.
### Template
```java
package dev.dokimos.core.evaluators;
import dev.dokimos.core.BaseEvaluator;
import dev.dokimos.core.EvalResult;
import dev.dokimos.core.EvalTestCase;
import dev.dokimos.core.EvalTestCaseParam;
import java.util.List;
/**
* [One-line description of what this evaluator checks.]
*/
public class NameEvaluator extends BaseEvaluator {
// Add any evaluator-specific fields here (e.g., JudgeLM judge)
private NameEvaluator(Builder builder) {
super(builder.name, builder.threshold, builder.evaluationParams);
// Initialize evaluator-specific fields from builder
}
/**
* Creates a new builder for constructing [Name] evaluators.
*
* @return a new builder
*/
public static Builder builder() {
return new Builder();
}
@Override
protected EvalResult runEvaluation(EvalTestCase testCase) {
double score = 0.0;
String reason = "...";
return EvalResult.builder()
.name(name)
.score(score)
.threshold(threshold)
.reason(reason)
.build();
}
public static class Builder {
private String name = "Name";
private double threshold = 1.0;
private List<EvalTestCaseParam> evaluationParams = List.of(
EvalTestCaseParam.ACTUAL_OUTPUT,
EvalTestCaseParam.EXPECTED_OUTPUT
);
public Builder name(String name) { this.name = name; return this; }
public Builder threshold(double threshold) { this.threshold = threshold; return this; }
public Builder evaluationParams(List<EvalTestCaseParam> params) {
this.evaluationParams = List.copyOf(params);
return this;
}
// Add evaluator-specific builder methods
public NameEvaluator build() { return new NameEvaluator(this); }
}
}
```
## Test structure
Create a test class in `dokimos-core/src/test/java/dev/dokimos/core/`. Tests use JUnit 6 (Jupiter) and AssertJ. Name it `<EvaluatorName>Test.java` in the `dev.dokimos.core` package (not the `evaluators` subpackage — this matches existing convention).
Cover at minimum:
- A case where the evaluator gives a high/passing score
- A case where the evaluator gives a low/failing score
- A case with a missing required parameter (should throw `IllegalArgumentException`)
- A case verifying custom threshold behavior
If the evaluator uses a `JudgeLM`, mock it with Mockito in tests.
## Conventions checklist
- [ ] Class is in `dev.dokimos.core.evaluators` package
- [ ] Extends `BaseEvaluator`, not `Evaluator` directly
- [ ] Constructor is private, takes `Builder`
- [ ] Has `public static Builder builder()` method
- [ ] Builder uses `List.copyOf()` for defensive copying of evaluationParams
- [ ] All public methods and the class have Javadoc
- [ ] Score is between 0.0 and 1.0
- [ ] Test class is in `dev.dokimos.core` package (not `evaluators`)
- [ ] Tests use AssertJ assertions (`assertThat(...)`)
- [ ] No Java features beyond Java 17
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · dokimos-dev/dokimos | 100 | MIT |
| alex-kopylov/zweihander | 100 | Apache-2.0 |
Database snapshot: 13.11.2026 · package version cb9400a763e8 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 1 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100create-tests
Scaffolds eval-driven tests using dokimos-junit. Use this skill when the user wants to create evaluation tests, set up eval-driven development, write parameterized evaluation tests, use @DatasetSource, or test their LLM application with Dokimos in a JUnit test. Also use when the… - 100evaluate-koog
Sets up evaluation of Koog AI agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark a Koog agent, or integrate Koog with Dokimos evaluations. Also use when the user mentions Koog agents, AI agent evaluation, or agent testing with Dokimos. - 100update-evaluator
Use when updating an existing Langfuse LLM-as-a-Judge evaluator. Trigger phrases include "update evaluator", "modify evaluator", "change evaluator prompt", "update evaluation filters", and "change evaluator model". Handles template-level changes (new version) and… - 98agentic-eval
Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when: - Implementing self-critique and reflection loops - Building evaluator-optimizer pipelines for quality-critical generation - Creating test-driven code refinement workflows - Designing… - 100phoenix-evals-new-metric
Add a new built-in (pre-built) LLM classification evaluator to the Phoenix evals libraries, and optionally promote it to a server-side dataset evaluator. Use when working in the Phoenix repo and asked to add a new built-in eval, pre-built metric, builtin evaluator, or… - 100agent-evaluation
This skill should be used when the user asks to "evaluate agent performance", "build test framework", "measure agent quality", "create evaluation rubrics", "implement LLM-as-judge", "compare model outputs", "mitigate evaluation bias", or mentions multi-dimensional evaluation,…