Skip to content
Skills Library
AI & agents · Testing

evaluate-agent

/evaluate-agent

Sets up evaluation of AI agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark an AI agent that uses tools, or assess tool call correctness, task completion, argument hallucination, or tool definition quality. Also use when the user mentions agent evaluation, tool call validation, agent testing, or tool reliability checks.

Repository
dokimos ↗
Licence
MIT
Last repository change
12.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
1
Deprecation

No signals detected. Repository changed on 12.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Sets up evaluation of AI agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark an AI agent that uses tools, or assess tool call correctness, task completion, argument hallucination, or tool definition quality. Also use when the user mentions agent evaluation, tool call validation, agent testing, or tool reliability checks.

Original author description

Sets up evaluation of AI agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark an AI agent that uses tools, or assess tool call correctness, task completion, argument hallucination, or tool definition quality. Also use when the user mentions agent evaluation, tool call validation, agent testing, or tool reliability checks.

How it works

  1. Where things live
  2. Available evaluators
  3. Data model essentials
  4. Extracting traces from a framework
  5. EvalTestCase key conventions
  6. Evaluator configuration
  7. Minimal pattern — single test case
  8. Experiment pattern — across a dataset
  9. Kotlin DSL
  10. Steps
Original SKILL.md headings · 184 lines, 1,108 words
SKILL.md preview
---
name: evaluate-agent
description: Sets up evaluation of AI agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark an AI agent that uses tools, or assess tool call correctness, task completion, argument hallucination, or tool definition quality. Also use when the user mentions agent evaluation, tool call validation, agent testing, or tool reliability checks.
---

# Evaluate AI Agent

Set up Dokimos agent evaluation for an AI agent that uses tools. The user will describe their agent and evaluation goals via `$ARGUMENTS`.

## Where things live

- **Agent data model**: `dokimos-core/src/main/java/dev/dokimos/core/agents/` — `ToolCall.java`, `ToolDefinition.java`, `AgentTrace.java`
- **Agent evaluators**: `dokimos-core/src/main/java/dev/dokimos/core/evaluators/agents/`
- **Kotlin DSL**: `dokimos-kotlin/src/main/kotlin/dev/dokimos/kotlin/dsl/evaluators/EvaluatorDsl.kt` and `CoreDsl.kt`
- **Example**: `dokimos-examples/src/main/java/dev/dokimos/examples/basic/AgentEvaluationExample.java`
- **Integration tests**: `dokimos-core/src/test/java/dev/dokimos/core/integration/AgentEvaluatorIT.java`
- **Maven dependency**: `dev.dokimos:dokimos-core` (agent evaluation is built in, no extra dependencies)

Before writing code, read the data model files and any relevant evaluator files to understand exact APIs.

## Available evaluators

| Evaluator | What it checks | LLM required? | Default threshold |
|-----------|---------------|:---:|:---:|
| `ToolCallValidityEvaluator` | Tool calls match JSON schema (names, required params, types, enums) | No | 1.0 |
| `ToolCorrectnessEvaluator` | Agent used the expected set of tools | No | 1.0 |
| `ToolTrajectoryEvaluator` | Tool-call sequence matches an expected trajectory (selectable match mode) | No | 1.0 |
| `ToolErrorEvaluator` | Tool calls succeeded (no error results) | No | 1.0 |
| `ToolEfficiencyEvaluator` | No redundant or duplicate tool calls | No | 1.0 |
| `TaskCompletionEvaluator` | Agent completed the user's tasks | Yes | 0.5 |
| `ToolArgumentHallucinationEvaluator` | Arguments are grounded in user input | Yes | 0.8 |
| `ToolNameReliabilityEvaluator` | Tool names follow conventions (snake_case, conciseness, clarity, ordering, intent) | Optional | 0.8 |
| `ToolDescriptionReliabilityEvaluator` | Tool descriptions are well-crafted (structure, clarity, args documented, examples, usage notes) | Optional | 0.8 |

Five of the nine are deterministic and need no LLM (`ToolCallValidity`, `ToolCorrectness`, `ToolTrajectory`, `ToolError`, `ToolEfficiency`), so they run in a unit test or CI gate with no API key.

`ToolTrajectoryEvaluator` match modes: `STRICT`, `IN_ORDER` (default, LCS), `ANY_ORDER`, `SUPERSET`, `SUBSET`, `PRECISION`, `RECALL`. Supply an `ArgumentMatcher` to also assert arguments, optionally per tool.

## Data model essentials

- **`ToolCall`**: A tool invocation record (name, arguments map, optional result string, optional metadata). Create with `ToolCall.of(name, args)` or `ToolCall.builder()`.
- **`ToolDefinition`**: A tool's contract (name, description, JSON Schema map for arguments). Create with `ToolDefinition.of(name, desc, schema)`. The schema must follow JSON Schema format with `"type": "object"`, `"properties"`, and optional `"required"`.
- **`AgentTrace`**: Wraps a full agent execution (tool calls, reasoning steps, final response). Build with `AgentTrace.builder().addToolCall(...).finalResponse(...).build()`. Call `toOutputMap()` to get a map with keys `"output"`, `"toolCalls"`, `"reasoningSteps"` for use in `EvalTestCase`. `trace.toTestCase(input, tools, tasks)` is a shortcut that builds a ready-to-use `EvalTestCase` (tools and tasks are optional overloads).
- **Multi-turn (`ConversationTrajectory` in `dev.dokimos.core.conversation`)**: when tools are called across a conversation, attach them to each assistant turn with `Message.assistant(content, toolCalls)` (or the builder's `assistantMessage(content, toolCalls)`). Score per turn with `trajectory.toolCallsByTurn()` (one `List<ToolCall>` per assistant turn) plus the deterministic evaluators, or build a whole-conversation case with `toTestCase(tools)` (deterministic, input is the last user turn) / `toTestCase(tools, tasks)` (judge, input is the full transcript). `TrajectoryEvaluator.includeToolCalls(true)` renders the calls into the judge prompt.

## Extracting traces from a framework

Do not hand-build `AgentTrace` if the agent runs on a supported framework. Each extractor turns a framework run into a trace:

- **LangChain4j** (`dokimos-langchain4j`): `LangChain4jSupport.toAgentTrace(result)` from an `AiServices` `Result<T>`; `toToolDefinitions(specs)` for the tools.
- **Spring AI** (`dokimos-spring-ai`): `SpringAiSupport.toAgentTrace(assistantMessage, toolResponseMessages)` (results matched by tool-call id); `toToolDefinitions(defs)`.
- **Koog** (`dokimos-koog`): install a `KoogTraceCollector` via `collectAgentTrace(collector)` in the event handler, run the agent, then `collector.toAgentTrace(response)`.
- **OpenAI Java SDK** (`dokimos-openai`): `OpenAiSupport.toAgentTrace(message, resultLookup)` from a `ChatCompletionMessage`; `toToolDefinitions(tools)` for the tools.

See the agent-evaluation guide for the full extractor reference: https://dokimos.dev/evaluation/agent-evaluation

## EvalTestCase key conventions

Evaluators read from specific keys in `EvalTestCase` maps:

| Map | Key | Type | Used by |
|-----|-----|------|---------|
| `actualOutputs` | `"toolCalls"` | `List<ToolCall>` | Validity, Correctness, Trajectory, Tool Error, Tool Efficiency, Hallucination |
| `actualOutputs` | `"output"` | `String` | Task Completion |
| `expectedOutputs` | `"toolCalls"` | `List<ToolCall>` | Correctness, Trajectory |
| `metadata` | `"tools"` | `List<ToolDefinition>` | Validity, Name Reliability, Description Reliability |
| `metadata` | `"tasks"` | `List<String>` | Task Completion |
| `metadata` | `"constraints"` | `String` | Task Completion |

**IMPORTANT**: In an Experiment, evaluators read metadata from the `Example`, NOT from the `Experiment`. Put `"tools"` and `"tasks"` in each Example's metadata (in the dataset), not on the Experiment builder.

## Evaluator configuration

```java
// Rule-based — just use defaults or set threshold/strictMode
ToolCallValidityEvaluator.builder().strictMode(true).threshold(1.0).build();
ToolCorrectnessEvaluator.builder().matchMode(ToolCorrectnessEvaluator.MatchMode.NAMES_ONLY).build();

// LLM-based — require a JudgeLM
JudgeLM judge = prompt -> openAiClient.generate(prompt);
TaskCompletionEvaluator.builder().judge(judge).threshold(0.5).build();
ToolArgumentHallucinationEvaluator.builder().judge(judge).threshold(0.8).build();

// Tool reliability — optional JudgeLM for semantic checks
ToolNameReliabilityEvaluator.builder().judge(judge).threshold(0.8).build();
ToolDescriptionReliabilityEvaluator.builder().maxInputArgs(5).maxOptionalArgs(3).judge(judge).build();
```

`ToolCorrectnessEvaluator` match modes: `NAMES_ONLY` (default, F1 score), `NAMES_AND_ORDER` (LCS similarity), `NAMES_AND_ARGS` (full structural comparison).

## Minimal pattern — single test case

```java
List<ToolDefinition> tools = List.of(
    ToolDefinition.of("search_flights", "Search for flights", Map.of(
        "type", "object",
        "properties", Map.of(
            "origin", Map.of("type", "string", "description", "Origin airport code"),
            "destination", Map.of("type", "string", "description", "Destination airport code")
        ),
        "required", List.of("origin", "destination")
    ))
);

AgentTrace trace = AgentTrace.builder()
    .addToolCall(ToolCall.of("search_flights", Map.of("origin", "JFK", "destination", "CDG")))
    .finalResponse("Found flights to Paris.")
    .build();

var testCase = EvalTestCase.builder()
    .input("Find flights from NYC to Paris")
    .actualOutput("toolCalls", trace.toolCalls())
    .actualOutput("output", trace.finalResponse())
    .metadata("tools", tools)
    .build();

var result = ToolCallValidityEvaluator.builder().build().evaluate(testCase);
```

## Experiment pattern — across a dataset

```java
JudgeLM judge = prompt -> openAiClient.generate(prompt);

// Tools and tasks go in each Example's metadata
Dataset dataset = Dataset.builder()
    .name("Agent Evaluation")
    .addExample(Example.builder()
        .input("input", "Find flights to Paris and book a hotel")
        .expectedOutput("toolCalls", List.of(
            ToolCall.of("search_flights", Map.of()),
            ToolCall.of("book_hotel", Map.of())
        ))
        .metadata("tools", tools)
        .metadata("tasks", List.of("Search flights", "Book hotel"))
        .build())
    .build();

ExperimentResult result = Experiment.builder()
    .name("Agent Evaluation")
    .dataset(dataset)
    .task(example -> {
        AgentTrace trace = myAgent.run(example.input());
        return trace.toOutputMap();
    })
    .evaluators(List.of(
        ToolCallValidityEvaluator.builder().build(),
        ToolCorrectnessEvaluator.builder().build(),
        TaskCompletionEvaluator.builder().judge(judge).build(),
        ToolArgumentHallucinationEvaluator.builder().judge(judge).build()
    ))
    .build()
    .run();
```

## Kotlin DSL

```kotlin
val judge = JudgeLM { prompt -> openAiClient.generate(prompt) }

// Standalone evaluator
val validity = toolCallValidity { threshold = 1.0 }

// In an experiment
evaluators {
    toolCallValidity { strictMode = true }
    toolCorrectness { matchMode = ToolCorrectnessEvaluator.MatchMode.NAMES_ONLY }
    toolTrajectory { matchMode = ToolTrajectoryEvaluator.MatchMode.IN_ORDER }
    toolError { }
    toolEfficiency { }
    taskCompletion(judge) { threshold = 0.5 }
    toolArgumentHallucination(judge) { threshold = 0.8 }
    toolNameReliability { judge = judgeLM }
    toolDescriptionReliability { maxInputArgs = 5; maxOptionalArgs = 3 }
}
```

## Steps

1. Understand from `$ARGUMENTS` what the agent does, what tools it uses, and the evaluation goals
2. Read the data model files (`ToolCall.java`, `ToolDefinition.java`, `AgentTrace.java`) and any relevant evaluator source files
3. Determine which evaluators are needed based on the table above
4. Define `ToolDefinition` objects for each tool the agent can use (with JSON Schema for arguments including `"type"`, `"properties"`, `"required"`)
5. Create a dataset with examples — each Example should include `metadata("tools", tools)` and optionally `metadata("tasks", taskList)` and `expectedOutput("toolCalls", expectedCalls)`
6. Build the `Task` using `AgentTrace.toOutputMap()` to capture tool calls and reasoning
7. Wire evaluators and run the experiment
8. Start with rule-based evaluators (`ToolCallValidityEvaluator`, `ToolCorrectnessEvaluator`) first — they don't need an LLM and give fast deterministic feedback. Add LLM-based evaluators once basics pass.

Versions

The database has no other skills named evaluate-agent.

Database snapshot: 13.11.2026 · package version 83398c5ff92c (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100evaluate-openai
    Sets up evaluation of an OpenAI Java SDK agent using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark an agent built directly on the OpenAI Java SDK that uses tools (function calling), or assess its tool call correctness, task completion, or argument…
  • 100evaluate-embabel
    Sets up evaluation of Embabel agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark an Embabel agent, its tool calls, or its execution trace. Also use when the user mentions Embabel evaluation or integrating Dokimos with an Embabel project.
  • 100agent-evaluation
    Evaluate LLM agents and tool-using workflows—task success, tool accuracy, latency/cost, safety, and regression suites. Use when shipping agent features, comparing prompts/models, or debugging agent failures. Triggers: "agent eval", "benchmark agent", "tool accuracy", "agent regression".
  • 100evaluate-mcp-gateway
    Evaluate MCP Gateway — test tool invocation accuracy, input validation, transport reliability, LLM tool selection quality, error handling. Use when: evaluate, test MCP server.
  • 100evaluate-and-harden-agent
    Evaluate an agent and harden its failure modes before it touches real traffic — an offline eval harness scored on task-completion, trajectory, and tool-use correctness against a fixed task set, plus loop hardening (step/tool-call caps, timeouts + retries, stop conditions,…
  • 100agent-evaluation
    Use when testing skills, commands, or agents for quality. Use after creating new skills, before deploying agents, or when debugging inconsistent agent behavior. Triggers on "evaluate", "test quality", "is this skill working", or QA of AI workflows.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.