evaluate-agent
/evaluate-agent
Konfiguruje ewaluację agentów AI z Dokimos: poprawność wywołań narzędzi, ukończenie zadań, halucynacje argumentów i jakość definicji narzędzi.
- Autor
- dokimos-dev ↗
- Repozytorium
- dokimos ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 12.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 1
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 1
Brak sygnałów. Repozytorium zmienione 12.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Konfiguruje ewaluację agentów AI z Dokimos: poprawność wywołań narzędzi, ukończenie zadań, halucynacje argumentów i jakość definicji narzędzi.
Oryginalny opis (EN)
Sets up evaluation of AI agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark an AI agent that uses tools, or assess tool call correctness, task completion, argument hallucination, or tool definition quality. Also use when the user mentions agent evaluation, tool call validation, agent testing, or tool reliability checks.
Jak działa
- Where things live
- Available evaluators
- Data model essentials
- Extracting traces from a framework
- EvalTestCase key conventions
- Evaluator configuration
- Minimal pattern — single test case
- Experiment pattern — across a dataset
- Kotlin DSL
- Steps
Podgląd SKILL.md
---
name: evaluate-agent
description: Sets up evaluation of AI agents using Dokimos. Use this skill when the user wants to evaluate, test, or benchmark an AI agent that uses tools, or assess tool call correctness, task completion, argument hallucination, or tool definition quality. Also use when the user mentions agent evaluation, tool call validation, agent testing, or tool reliability checks.
---
# Evaluate AI Agent
Set up Dokimos agent evaluation for an AI agent that uses tools. The user will describe their agent and evaluation goals via `$ARGUMENTS`.
## Where things live
- **Agent data model**: `dokimos-core/src/main/java/dev/dokimos/core/agents/` — `ToolCall.java`, `ToolDefinition.java`, `AgentTrace.java`
- **Agent evaluators**: `dokimos-core/src/main/java/dev/dokimos/core/evaluators/agents/`
- **Kotlin DSL**: `dokimos-kotlin/src/main/kotlin/dev/dokimos/kotlin/dsl/evaluators/EvaluatorDsl.kt` and `CoreDsl.kt`
- **Example**: `dokimos-examples/src/main/java/dev/dokimos/examples/basic/AgentEvaluationExample.java`
- **Integration tests**: `dokimos-core/src/test/java/dev/dokimos/core/integration/AgentEvaluatorIT.java`
- **Maven dependency**: `dev.dokimos:dokimos-core` (agent evaluation is built in, no extra dependencies)
Before writing code, read the data model files and any relevant evaluator files to understand exact APIs.
## Available evaluators
| Evaluator | What it checks | LLM required? | Default threshold |
|-----------|---------------|:---:|:---:|
| `ToolCallValidityEvaluator` | Tool calls match JSON schema (names, required params, types, enums) | No | 1.0 |
| `ToolCorrectnessEvaluator` | Agent used the expected set of tools | No | 1.0 |
| `ToolTrajectoryEvaluator` | Tool-call sequence matches an expected trajectory (selectable match mode) | No | 1.0 |
| `ToolErrorEvaluator` | Tool calls succeeded (no error results) | No | 1.0 |
| `ToolEfficiencyEvaluator` | No redundant or duplicate tool calls | No | 1.0 |
| `TaskCompletionEvaluator` | Agent completed the user's tasks | Yes | 0.5 |
| `ToolArgumentHallucinationEvaluator` | Arguments are grounded in user input | Yes | 0.8 |
| `ToolNameReliabilityEvaluator` | Tool names follow conventions (snake_case, conciseness, clarity, ordering, intent) | Optional | 0.8 |
| `ToolDescriptionReliabilityEvaluator` | Tool descriptions are well-crafted (structure, clarity, args documented, examples, usage notes) | Optional | 0.8 |
Five of the nine are deterministic and need no LLM (`ToolCallValidity`, `ToolCorrectness`, `ToolTrajectory`, `ToolError`, `ToolEfficiency`), so they run in a unit test or CI gate with no API key.
`ToolTrajectoryEvaluator` match modes: `STRICT`, `IN_ORDER` (default, LCS), `ANY_ORDER`, `SUPERSET`, `SUBSET`, `PRECISION`, `RECALL`. Supply an `ArgumentMatcher` to also assert arguments, optionally per tool.
## Data model essentials
- **`ToolCall`**: A tool invocation record (name, arguments map, optional result string, optional metadata). Create with `ToolCall.of(name, args)` or `ToolCall.builder()`.
- **`ToolDefinition`**: A tool's contract (name, description, JSON Schema map for arguments). Create with `ToolDefinition.of(name, desc, schema)`. The schema must follow JSON Schema format with `"type": "object"`, `"properties"`, and optional `"required"`.
- **`AgentTrace`**: Wraps a full agent execution (tool calls, reasoning steps, final response). Build with `AgentTrace.builder().addToolCall(...).finalResponse(...).build()`. Call `toOutputMap()` to get a map with keys `"output"`, `"toolCalls"`, `"reasoningSteps"` for use in `EvalTestCase`. `trace.toTestCase(input, tools, tasks)` is a shortcut that builds a ready-to-use `EvalTestCase` (tools and tasks are optional overloads).
- **Multi-turn (`ConversationTrajectory` in `dev.dokimos.core.conversation`)**: when tools are called across a conversation, attach them to each assistant turn with `Message.assistant(content, toolCalls)` (or the builder's `assistantMessage(content, toolCalls)`). Score per turn with `trajectory.toolCallsByTurn()` (one `List<ToolCall>` per assistant turn) plus the deterministic evaluators, or build a whole-conversation case with `toTestCase(tools)` (deterministic, input is the last user turn) / `toTestCase(tools, tasks)` (judge, input is the full transcript). `TrajectoryEvaluator.includeToolCalls(true)` renders the calls into the judge prompt.
## Extracting traces from a framework
Do not hand-build `AgentTrace` if the agent runs on a supported framework. Each extractor turns a framework run into a trace:
- **LangChain4j** (`dokimos-langchain4j`): `LangChain4jSupport.toAgentTrace(result)` from an `AiServices` `Result<T>`; `toToolDefinitions(specs)` for the tools.
- **Spring AI** (`dokimos-spring-ai`): `SpringAiSupport.toAgentTrace(assistantMessage, toolResponseMessages)` (results matched by tool-call id); `toToolDefinitions(defs)`.
- **Koog** (`dokimos-koog`): install a `KoogTraceCollector` via `collectAgentTrace(collector)` in the event handler, run the agent, then `collector.toAgentTrace(response)`.
- **OpenAI Java SDK** (`dokimos-openai`): `OpenAiSupport.toAgentTrace(message, resultLookup)` from a `ChatCompletionMessage`; `toToolDefinitions(tools)` for the tools.
See the agent-evaluation guide for the full extractor reference: https://dokimos.dev/evaluation/agent-evaluation
## EvalTestCase key conventions
Evaluators read from specific keys in `EvalTestCase` maps:
| Map | Key | Type | Used by |
|-----|-----|------|---------|
| `actualOutputs` | `"toolCalls"` | `List<ToolCall>` | Validity, Correctness, Trajectory, Tool Error, Tool Efficiency, Hallucination |
| `actualOutputs` | `"output"` | `String` | Task Completion |
| `expectedOutputs` | `"toolCalls"` | `List<ToolCall>` | Correctness, Trajectory |
| `metadata` | `"tools"` | `List<ToolDefinition>` | Validity, Name Reliability, Description Reliability |
| `metadata` | `"tasks"` | `List<String>` | Task Completion |
| `metadata` | `"constraints"` | `String` | Task Completion |
**IMPORTANT**: In an Experiment, evaluators read metadata from the `Example`, NOT from the `Experiment`. Put `"tools"` and `"tasks"` in each Example's metadata (in the dataset), not on the Experiment builder.
## Evaluator configuration
```java
// Rule-based — just use defaults or set threshold/strictMode
ToolCallValidityEvaluator.builder().strictMode(true).threshold(1.0).build();
ToolCorrectnessEvaluator.builder().matchMode(ToolCorrectnessEvaluator.MatchMode.NAMES_ONLY).build();
// LLM-based — require a JudgeLM
JudgeLM judge = prompt -> openAiClient.generate(prompt);
TaskCompletionEvaluator.builder().judge(judge).threshold(0.5).build();
ToolArgumentHallucinationEvaluator.builder().judge(judge).threshold(0.8).build();
// Tool reliability — optional JudgeLM for semantic checks
ToolNameReliabilityEvaluator.builder().judge(judge).threshold(0.8).build();
ToolDescriptionReliabilityEvaluator.builder().maxInputArgs(5).maxOptionalArgs(3).judge(judge).build();
```
`ToolCorrectnessEvaluator` match modes: `NAMES_ONLY` (default, F1 score), `NAMES_AND_ORDER` (LCS similarity), `NAMES_AND_ARGS` (full structural comparison).
## Minimal pattern — single test case
```java
List<ToolDefinition> tools = List.of(
ToolDefinition.of("search_flights", "Search for flights", Map.of(
"type", "object",
"properties", Map.of(
"origin", Map.of("type", "string", "description", "Origin airport code"),
"destination", Map.of("type", "string", "description", "Destination airport code")
),
"required", List.of("origin", "destination")
))
);
AgentTrace trace = AgentTrace.builder()
.addToolCall(ToolCall.of("search_flights", Map.of("origin", "JFK", "destination", "CDG")))
.finalResponse("Found flights to Paris.")
.build();
var testCase = EvalTestCase.builder()
.input("Find flights from NYC to Paris")
.actualOutput("toolCalls", trace.toolCalls())
.actualOutput("output", trace.finalResponse())
.metadata("tools", tools)
.build();
var result = ToolCallValidityEvaluator.builder().build().evaluate(testCase);
```
## Experiment pattern — across a dataset
```java
JudgeLM judge = prompt -> openAiClient.generate(prompt);
// Tools and tasks go in each Example's metadata
Dataset dataset = Dataset.builder()
.name("Agent Evaluation")
.addExample(Example.builder()
.input("input", "Find flights to Paris and book a hotel")
.expectedOutput("toolCalls", List.of(
ToolCall.of("search_flights", Map.of()),
ToolCall.of("book_hotel", Map.of())
))
.metadata("tools", tools)
.metadata("tasks", List.of("Search flights", "Book hotel"))
.build())
.build();
ExperimentResult result = Experiment.builder()
.name("Agent Evaluation")
.dataset(dataset)
.task(example -> {
AgentTrace trace = myAgent.run(example.input());
return trace.toOutputMap();
})
.evaluators(List.of(
ToolCallValidityEvaluator.builder().build(),
ToolCorrectnessEvaluator.builder().build(),
TaskCompletionEvaluator.builder().judge(judge).build(),
ToolArgumentHallucinationEvaluator.builder().judge(judge).build()
))
.build()
.run();
```
## Kotlin DSL
```kotlin
val judge = JudgeLM { prompt -> openAiClient.generate(prompt) }
// Standalone evaluator
val validity = toolCallValidity { threshold = 1.0 }
// In an experiment
evaluators {
toolCallValidity { strictMode = true }
toolCorrectness { matchMode = ToolCorrectnessEvaluator.MatchMode.NAMES_ONLY }
toolTrajectory { matchMode = ToolTrajectoryEvaluator.MatchMode.IN_ORDER }
toolError { }
toolEfficiency { }
taskCompletion(judge) { threshold = 0.5 }
toolArgumentHallucination(judge) { threshold = 0.8 }
toolNameReliability { judge = judgeLM }
toolDescriptionReliability { maxInputArgs = 5; maxOptionalArgs = 3 }
}
```
## Steps
1. Understand from `$ARGUMENTS` what the agent does, what tools it uses, and the evaluation goals
2. Read the data model files (`ToolCall.java`, `ToolDefinition.java`, `AgentTrace.java`) and any relevant evaluator source files
3. Determine which evaluators are needed based on the table above
4. Define `ToolDefinition` objects for each tool the agent can use (with JSON Schema for arguments including `"type"`, `"properties"`, `"required"`)
5. Create a dataset with examples — each Example should include `metadata("tools", tools)` and optionally `metadata("tasks", taskList)` and `expectedOutput("toolCalls", expectedCalls)`
6. Build the `Task` using `AgentTrace.toOutputMap()` to capture tool calls and reasoning
7. Wire evaluators and run the experiment
8. Start with rule-based evaluators (`ToolCallValidityEvaluator`, `ToolCorrectnessEvaluator`) first — they don't need an LLM and give fast deterministic feedback. Add LLM-based evaluators once basics pass.
Wersje
W bazie nie ma innych skilli o nazwie evaluate-agent.
Migawka bazy z 13.11.2026 · wersja paczki 83398c5ff92c (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100evaluate-openai
Konfiguruje ewaluację agenta zbudowanego bezpośrednio na OpenAI Java SDK z wywoływaniem funkcji w Dokimos: poprawność wywołań narzędzi, ukończenie zadań i halucynacje argumentów. - 100evaluate-embabel
Konfiguruje ewaluację agentów Embabel z użyciem Dokimos, w tym ich wywołań narzędzi i śladu wykonania. - 100agent-evaluation
Ocenia agentów LLM i workflow z narzędziami: powodzenie zadań, trafność narzędzi, opóźnienia i koszt, bezpieczeństwo oraz zestawy regresji. - 100evaluate-mcp-gateway
Ocenia bramę MCP: dokładność wywołań narzędzi, walidację wejścia, niezawodność transportu, jakość wyboru narzędzi przez LLM i obsługę błędów. - 100evaluate-and-harden-agent
Ocenia agenta i utwardza jego tryby awarii przed realnym ruchem: ewaluacja offline (ukończenie zadań, trajektoria, poprawność narzędzi), limity kroków, timeouty, ponowienia, zatwierdzenia przez człowieka i śledzenie kosztów. - 100agent-evaluation
Testuje jakość skilli, komend i agentów: po utworzeniu nowego, przed wdrożeniem agenta lub gdy zachowanie jest niespójne.