implement-eval

What it does and when to use it

Implement or update scenario-focused evaluations for the Prefect MCP server. Use when adding eval coverage from a GitHub issue or a described support scenario, preparing Prefect server state, prompting the test agent, asserting user-facing behavior, running evals, and updating the eval catalog.

How it works

SKILL.md has no level-two headings · 20 lines, 192 words

---
name: implement-eval
description: Implement or update scenario-focused evaluations for the Prefect MCP server. Use when adding eval coverage from a GitHub issue or a described support scenario, preparing Prefect server state, prompting the test agent, asserting user-facing behavior, running evals, and updating the eval catalog.
---

# Implement a Prefect MCP eval

1. Read `AGENTS.md` and `evals/README.md` before making changes.
2. Resolve the requested scenario:
   - If the user provides a GitHub issue number or URL, read it with an available GitHub integration or the `gh` CLI.
   - Otherwise, use the scenario described by the user.
   - Identify the user-facing question, required Prefect state, expected investigation, and success criteria.
3. Inspect related evals and fixtures before choosing a file location. Extend an existing scenario directory when appropriate; otherwise add a focused test under `evals/`.
4. Implement the eval:
   - Create server state in a fixture.
   - Prompt the agent in language a Prefect user or support engineer would use.
   - Assert on the final behavior or answer, not incidental wording or private implementation details.
   - Keep protocol behavior in unit tests rather than evals.
5. Run the narrowest relevant eval first, then run the full suite with `just evals`.
6. Add or update the eval's row in `evals/README.md`.
7. Review the diff for unrelated changes and report the scenario covered and verification performed.

Do not add live credentials, harness-specific argument placeholders, or client-specific tool syntax to the skill.

Versions

The database has no other skills named implement-eval.

Database snapshot: 13.11.2026 · package version dd3a228f852a (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

• 100prefect-flow-builder
  Build prefect flow builder operations. Auto-activating skill for Data Pipelines. Triggers on: prefect flow builder, prefect flow builder Part of the Data Pipelines skill category. Use when working with prefect flow builder functionality. Trigger with phrases like "prefect flow…
• 100workflows
  Inspect and troubleshoot Prefect workflows, deployments, work pools, and releases with Prefect MCP tools. Use for Prefect operational questions, failure diagnosis, documentation lookup, and safe guidance when a requested mutation requires a separately configured Prefect CLI.
• 100python-prefect-to-orchestra
  Use this skill when the user wants to convert a Prefect @task function containing arbitrary Python logic (computation, API calls, pandas, boto3, etc.) into an equivalent Orchestra Python pipeline task. Triggers: any mention of migrating or rewriting Prefect @task functions to…
• 100prefect-data-workflow-orchestration
  Use this skill when building, scheduling, and observing resilient Python data workflows and tasks using Prefect 2+.
• 100add-eval
  Add or update an LLM eval test for a specific tool or collection. Creates a new eval scenario or updates an existing one to cover new tools. Use when adding eval coverage for new or modified tools.
• 100prefect-webhooks
  Receive and authenticate Prefect webhooks. Use when setting up an endpoint for Prefect automation notifications sent by the Custom Webhook notification block, checking the secret header you configured, debugging 401s, or handling flow run Failed or Completed notifications.…