model-evaluation
/model-evaluation
Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.
- Author
- awslabs ↗
- Repository
- agent-plugins ↗
- Licence
- Apache-2.0
- Last repository change
- 09.10.2026
Assessment and security
- Files in this package
- 15
- Scripts
- 5
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 09.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
- 5 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: code_templates/custom_scorer_evaluator.py, code_templates/llmaaj_evaluator.py, scripts/nova_reward_function_source_template.py, scripts/reward_function_source_template.py, scripts/validate_custom_metrics.py
What it does and when to use it
Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.
Triggers from the description
Original author description
Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.
How it works
- Prerequisites
- Principles
- Scope
- Evaluation Types
- Workflow
SKILL.md preview
--- name: model-evaluation description: 'Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.' metadata: version: "3.0.0" --- # Model Evaluation Generate code that evaluates a SageMaker model. ## Prerequisites - The SDK environment has been verified (SDK version, region, execution role). If not done, activate the `sdk-getting-started` skill first. ## Principles 1. **One thing at a time.** Each response advances exactly one decision. Never combine multiple questions in a single turn. 2. **Confirm before proceeding.** Wait for the user to agree before moving to the next step. 3. **Don't read files until you need them.** Only read reference files when you've reached the step that requires them. 4. **Don't ask what you already know.** If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask. 5. **No narration.** Share outcomes and ask questions. Keep responses short. 6. **No repetition.** If you said something before a tool call, don't repeat it after. ## Scope This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models. Tell the user when the skill is activated: > "I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization." If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill. ## Evaluation Types There are two evaluation types: - **LLM-as-Judge** — an LLM grades your model's responses. (OSS models only — not supported for Nova.) - **Custom Scorer** — programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models. ## Workflow ### Step 1: Determine evaluation type **Do you already know which evaluation type to use?** Check conversation history, plan.md, workflow_state.json, or anything else you've already read. **If yes:** confirm with the user. > "It sounds like you want to run [evaluation type]. Is that right?" ⏸ Wait for confirmation. If confirmed → go to Step 2. **If no:** ask. > "What kind of evaluation would you like to run? I support: > > 1. **LLM-as-Judge** — an LLM grades your model's responses > 2. **Custom Scorer** — programmatic scoring (math, code, or your own logic) > > Pick one, or say 'help me decide' if you're not sure." ⏸ Wait for user. - If user picks one → go to Step 2. - If user indicates uncertainty, by saying something like "help me decide," "whatever you think," "I'm not sure" → read `references/evaluation-type-guide.md` and follow its instructions. It will guide the user to a choice and then return here. You MUST NEVER make a recommendation to the user on eval type without reading `references/evaluation-type-guide.md`. ### Step 2: Validate and hand off to evaluation workflow Before reading the reference file, validate that the chosen evaluation type is compatible with the user's situation. You may already know these answers from conversation context — don't ask if you don't need to. #### LLM-as-Judge validation 1. **What model type are we evaluating?** LLM-as-Judge is not supported for Nova models. To determine model type (if you don't already know it): - If you have the **training job name or ARN**, use the AWS MCP tool `list-tags` on the training job ARN and look for the `sagemaker-studio:jumpstart-model-id` tag. Contains "nova" → Nova. Anything else → OSS. - If you have a **Model Package ARN**, use the AWS MCP tool `describe-model-package` and check the model description or source tags. - If neither is available, ask the user. 2. **Does the user have an evaluation dataset?** LLM-as-Judge requires one. #### Custom Scorer validation 1. **Does the user have an evaluation dataset?** Custom Scorer requires one. (Works with both OSS and Nova models, though for Nova only custom lambdas are supported.) --- If validation fails, tell the user which requirement(s) aren't met and offer alternatives: > "[Evaluation type] won't work because [reason]." If the failure reason was lack of an eval dataset, there's nothing we can do. Inform the user: > "Unfortunately all of the supported eval types require an eval dataset. I can't help you with model evaluation." If the failure reason is something else, offer to help them pick a different evaluation type. ⏸ Wait for user. If they say they do want help choosing a different eval type → read `references/evaluation-type-guide.md`. If validation passes, read the corresponding reference file: | User chose | Read | | ------------- | ---------------------------------------- | | LLM-as-Judge | `references/llmaaj-evaluation.md` | | Custom Scorer | `references/custom-scorer-evaluation.md` | Follow the reference file's instructions from the beginning.
Files in this package (15)
- SKILL.md 5 KB
- code_templates/custom_scorer_evaluator.py 2 KB
- code_templates/llmaaj_evaluator.py 2 KB
- references/code_output_guide.md 3 KB
- references/create-reward-function.md 3 KB
- references/custom-lambda-scorer.md 4 KB
- references/custom-scorer-evaluation.md 10 KB
- references/evaluation-type-guide.md 8 KB
- references/llmaaj-builtin-evaluation.md 5 KB
- references/llmaaj-custom-evaluation.md 2 KB
- references/llmaaj-evaluation.md 14 KB
- references/supported-judge-models.md 2 KB
- scripts/nova_reward_function_source_template.py 13 KB
- scripts/reward_function_source_template.py 9 KB
- scripts/validate_custom_metrics.py 4 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · awslabs/agent-plugins | 100 | Apache-2.0 |
| SimonFoote-greensparx/CopilotForge | 100 | CC-BY-4.0 |
| param087/agent-ml-skills | 100 | MIT |
| Amey-Thakur/AI-SKILLS | 90 | MIT |
| 420company/artemis | 85 | MIT |
| Harrypatria/data-mining-ml | 85 | MIT |
Database snapshot: 13.11.2026 · package version 6c64a14fd1f3 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 1 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100agentic-evaluation-framework
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality". - 100agent-evaluation
This skill should be used when the user asks to "evaluate agent performance", "build test framework", "measure agent quality", "create evaluation rubrics", "implement LLM-as-judge", "compare model outputs", "mitigate evaluation bias", or mentions multi-dimensional evaluation,… - 100advanced-evaluation
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment. - 100advanced-evaluation
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment. - 87genai-hub-prompt-evaluation
Run GenAI evaluations on SAP AI Core: benchmark LLMs and prompts against a test dataset using system-defined and custom (LLM-as-judge) metrics, then retrieve aggregate results. Supports all three evaluation types — prompt registry, orchestration registry, and grounding (RAG)… - 90compare-llm-models
Use this to pick or switch the LLM behind a feature, based on evidence instead of hype or the newest release. Trigger on "which model should I use", "is GPT/Claude/Gemini/Llama better for this", "should I switch models", "can a cheaper model do this", "compare models for my use…