Skip to content
Skills Library
AI & agents · Building & coding

create-evaluator

/create-evaluator

Scaffolds a new Evaluator for the Dokimos LLM evaluation framework. Use this skill whenever the user wants to create, add, or implement a new evaluator, metric, or scoring function for evaluating LLM outputs. Also use when the user mentions adding evaluation criteria, custom metrics, or grading logic.

Repository
dokimos ↗
Licence
MIT
Last repository change
12.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 12.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Scaffolds a new Evaluator for the Dokimos LLM evaluation framework. Use this skill whenever the user wants to create, add, or implement a new evaluator, metric, or scoring function for evaluating LLM outputs. Also use when the user mentions adding evaluation criteria, custom metrics, or grading logic.

Original author description

Scaffolds a new Evaluator for the Dokimos LLM evaluation framework. Use this skill whenever the user wants to create, add, or implement a new evaluator, metric, or scoring function for evaluating LLM outputs. Also use when the user mentions adding evaluation criteria, custom metrics, or grading logic.

How it works

  1. Where things live
  2. Evaluator structure
  3. Test structure
  4. Conventions checklist
Original SKILL.md headings · 124 lines, 550 words
SKILL.md preview
---
name: create-evaluator
description: Scaffolds a new Evaluator for the Dokimos LLM evaluation framework. Use this skill whenever the user wants to create, add, or implement a new evaluator, metric, or scoring function for evaluating LLM outputs. Also use when the user mentions adding evaluation criteria, custom metrics, or grading logic.
---

# Create Evaluator

Scaffold a new Evaluator implementation for Dokimos following project conventions.

The user will describe what the evaluator should do via `$ARGUMENTS`. Use that description to determine the evaluator's name, scoring logic, which test case parameters it requires, and what constitutes a passing score.

## Where things live

- **Evaluator source**: `dokimos-core/src/main/java/dev/dokimos/core/evaluators/`
- **Tests**: `dokimos-core/src/test/java/dev/dokimos/core/`
- **Base class**: `dev.dokimos.core.BaseEvaluator`
- **Key types**: `EvalResult`, `EvalTestCase`, `EvalTestCaseParam`, `JudgeLM`

Before writing code, read these files to understand the current conventions:
- `dokimos-core/src/main/java/dev/dokimos/core/BaseEvaluator.java`
- `dokimos-core/src/main/java/dev/dokimos/core/evaluators/ExactMatchEvaluator.java` (simple evaluator reference)
- `dokimos-core/src/main/java/dev/dokimos/core/evaluators/LLMJudgeEvaluator.java` (LLM-based evaluator reference, if the new evaluator needs a JudgeLM)

## Evaluator structure

Every evaluator follows this pattern:

1. **Extends `BaseEvaluator`** with a private constructor that takes a `Builder`
2. **Implements `runEvaluation(EvalTestCase testCase)`** — this is where the scoring logic goes
3. **Has a static `Builder` class** with sensible defaults for name, threshold, and evaluationParams
4. **Returns an `EvalResult`** via `EvalResult.builder().name(name).score(score).threshold(threshold).reason(reason).build()`

The `evaluationParams` field declares which `EvalTestCaseParam` values the evaluator requires (INPUT, ACTUAL_OUTPUT, EXPECTED_OUTPUT). `BaseEvaluator` validates these are non-null before calling `runEvaluation`.

If the evaluator needs an LLM to judge (semantic similarity, faithfulness, etc.), accept a `JudgeLM` in the builder. `JudgeLM` is a functional interface: `String generate(String prompt)`.

### Template

```java
package dev.dokimos.core.evaluators;

import dev.dokimos.core.BaseEvaluator;
import dev.dokimos.core.EvalResult;
import dev.dokimos.core.EvalTestCase;
import dev.dokimos.core.EvalTestCaseParam;

import java.util.List;

/**
 * [One-line description of what this evaluator checks.]
 */
public class NameEvaluator extends BaseEvaluator {

    // Add any evaluator-specific fields here (e.g., JudgeLM judge)

    private NameEvaluator(Builder builder) {
        super(builder.name, builder.threshold, builder.evaluationParams);
        // Initialize evaluator-specific fields from builder
    }

    /**
     * Creates a new builder for constructing [Name] evaluators.
     *
     * @return a new builder
     */
    public static Builder builder() {
        return new Builder();
    }

    @Override
    protected EvalResult runEvaluation(EvalTestCase testCase) {
        double score = 0.0;
        String reason = "...";

        return EvalResult.builder()
                .name(name)
                .score(score)
                .threshold(threshold)
                .reason(reason)
                .build();
    }

    public static class Builder {
        private String name = "Name";
        private double threshold = 1.0;
        private List<EvalTestCaseParam> evaluationParams = List.of(
                EvalTestCaseParam.ACTUAL_OUTPUT,
                EvalTestCaseParam.EXPECTED_OUTPUT
        );

        public Builder name(String name) { this.name = name; return this; }
        public Builder threshold(double threshold) { this.threshold = threshold; return this; }
        public Builder evaluationParams(List<EvalTestCaseParam> params) {
            this.evaluationParams = List.copyOf(params);
            return this;
        }
        // Add evaluator-specific builder methods

        public NameEvaluator build() { return new NameEvaluator(this); }
    }
}
```

## Test structure

Create a test class in `dokimos-core/src/test/java/dev/dokimos/core/`. Tests use JUnit 6 (Jupiter) and AssertJ. Name it `<EvaluatorName>Test.java` in the `dev.dokimos.core` package (not the `evaluators` subpackage — this matches existing convention).

Cover at minimum:
- A case where the evaluator gives a high/passing score
- A case where the evaluator gives a low/failing score
- A case with a missing required parameter (should throw `IllegalArgumentException`)
- A case verifying custom threshold behavior

If the evaluator uses a `JudgeLM`, mock it with Mockito in tests.

## Conventions checklist

- [ ] Class is in `dev.dokimos.core.evaluators` package
- [ ] Extends `BaseEvaluator`, not `Evaluator` directly
- [ ] Constructor is private, takes `Builder`
- [ ] Has `public static Builder builder()` method
- [ ] Builder uses `List.copyOf()` for defensive copying of evaluationParams
- [ ] All public methods and the class have Javadoc
- [ ] Score is between 0.0 and 1.0
- [ ] Test class is in `dev.dokimos.core` package (not `evaluators`)
- [ ] Tests use AssertJ assertions (`assertThat(...)`)
- [ ] No Java features beyond Java 17

Versions

RepositoryScoreLicence
This page · dokimos-dev/dokimos100MIT
alex-kopylov/zweihander100Apache-2.0

Database snapshot: 13.11.2026 · package version cb9400a763e8 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 1 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100create-tests
    Scaffolds eval-driven tests using dokimos-junit. Use this skill when the user wants to create evaluation tests, set up eval-driven development, write parameterized evaluation tests, use @DatasetSource, or test their LLM application with Dokimos in a JUnit test. Also use when the…
  • 100evaluate-koog
    Sets up evaluation of Koog AI agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark a Koog agent, or integrate Koog with Dokimos evaluations. Also use when the user mentions Koog agents, AI agent evaluation, or agent testing with Dokimos.
  • 100update-evaluator
    Use when updating an existing Langfuse LLM-as-a-Judge evaluator. Trigger phrases include "update evaluator", "modify evaluator", "change evaluator prompt", "update evaluation filters", and "change evaluator model". Handles template-level changes (new version) and…
  • 98agentic-eval
    Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when: - Implementing self-critique and reflection loops - Building evaluator-optimizer pipelines for quality-critical generation - Creating test-driven code refinement workflows - Designing…
  • 100phoenix-evals-new-metric
    Add a new built-in (pre-built) LLM classification evaluator to the Phoenix evals libraries, and optionally promote it to a server-side dataset evaluator. Use when working in the Phoenix repo and asked to add a new built-in eval, pre-built metric, builtin evaluator, or…
  • 100agent-evaluation
    This skill should be used when the user asks to "evaluate agent performance", "build test framework", "measure agent quality", "create evaluation rubrics", "implement LLM-as-judge", "compare model outputs", "mitigate evaluation bias", or mentions multi-dimensional evaluation,…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.