Skip to content
Skills Library
AI & agents · Testing

setup

/setup

Set up regression testing (CI) for this repository's Claude Code configuration (plugin, skills, hooks, CLAUDE.md). Use when the user says "set up config-drift-checker", "add evals for my plugin/skills/hooks", "test my Claude Code setup", "make sure my hooks keep working after updates", or "wire the eval GitHub Action". Do NOT use for evaluating an LLM application or prompts; this is for agent configuration only.

Licence
Apache-2.0
Last repository change
27.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Set up regression testing (CI) for this repository's Claude Code configuration (plugin, skills, hooks, CLAUDE.md). Use when the user says "set up config-drift-checker", "add evals for my plugin/skills/hooks", "test my Claude Code setup", "make sure my hooks keep working after updates", or "wire the eval GitHub Action". Do NOT use for evaluating an LLM application or prompts; this is for agent configuration only.

Triggers from the description

set up config-drift-checkeradd evals for my plugin/skills/hookstest my Claude Code setupmake sure my hooks keep working after updateswire the eval GitHub Action
Original author description

Set up regression testing (CI) for this repository's Claude Code configuration (plugin, skills, hooks, CLAUDE.md). Use when the user says "set up config-drift-checker", "add evals for my plugin/skills/hooks", "test my Claude Code setup", "make sure my hooks keep working after updates", or "wire the eval GitHub Action". Do NOT use for evaluating an LLM application or prompts; this is for agent configuration only.

How it works

  1. 1. Discover the setup
  2. 2. Generate starter cases (three, from real content)
  3. 3. Smoke run
  4. 4. Pin and budget: .cdc.yml
  5. 5. Wire CI
  6. 6. Hand-off checklist (print exactly)
Original SKILL.md headings · 98 lines, 985 words
SKILL.md preview
---
name: setup
description: Set up regression testing (CI) for this repository's Claude Code configuration (plugin, skills, hooks, CLAUDE.md). Use when the user says "set up config-drift-checker", "add evals for my plugin/skills/hooks", "test my Claude Code setup", "make sure my hooks keep working after updates", or "wire the eval GitHub Action". Do NOT use for evaluating an LLM application or prompts; this is for agent configuration only.
---

# config-drift-checker: setup

Goal: leave the repo with (1) an `evals/` suite generated from the user's **real** configuration,
(2) a passing local smoke run, (3) the GitHub workflow in place, (4) a checklist of the one or two
things only the user can do. Ask nothing you can discover from the repo.

## 1. Discover the setup

Look, in order, for: `.claude-plugin/plugin.json` (a plugin; its dir is the target);
`.claude/skills/*/SKILL.md`, `.claude/hooks/*`, `.claude/settings.json` hooks, `CLAUDE.md`.
If there is no plugin manifest, create a minimal one at the repo root pointing at the skills and
hooks you found (`claude plugin init --help` shows the shape); evals need a plugin target.
**Check ignores first:** run `git check-ignore -v .claude/evals/x`, because many repos ignore `.claude/`
(worktrees, local settings). If it is ignored, put skills, hooks and evals under `agent-config/`
at the repo root and set `"experimental": {"evals": "agent-config/evals"}` in the manifest; never
edit the user's `.gitignore` for this. Note that a plugin does not carry `CLAUDE.md`, so cases that
test CLAUDE.md rules must copy it into the workspace via `scaffold_script`.
List what you found to the user in five lines or fewer.

## 2. Generate starter cases (three, from real content)

Turn budgets: `max_turns: 30` and `timeout_seconds: 900` for real-code cases (the agent reads,
edits several files and writes a test), `12` for single-file or prose cases, `6` for hook cases. A
run that hits `max_turns` is scored as-is and flagged TRUNCATED; treat that as "raise the budget".
Give every case a `description:` line in `prompt.md` frontmatter, one sentence stating what the
case proves and which part of the setup (skill, hook, CLAUDE.md rule) it exercises; the HTML report
shows it under the case heading. Also give it a `covers.yaml` next to prompt.md listing the ids of the rules it exercises
(never a `covers:` frontmatter key: the official runner rejects unknown keys),
from `node ${CLAUDE_PLUGIN_ROOT}/tools/config-coverage.mjs <plugin> --list` (run it after step 1;
a negative-trigger case covers nothing, which is correct). The coverage number in every report
comes from these. Write them under `<plugin>/evals/<case>/` using the official layout (prompt.md + graders/*.md,
optional case.yaml). Use `${CLAUDE_PLUGIN_ROOT}/../examples/komo-stack/evals/` as the reference if present,
or the format section of the `write-case` skill.

- **One positive skill case** per important skill: a prompt that should trigger it, graders that
  check the skill's *observable* conventions in the output (regex on code position, not prose),
  a `tool_used: Skill` indicator, and one `llm` grader with the rubric taken from the SKILL.md.
  For backend/code skills prefer a **real-code case**: a `scaffold.sh` (named in `case.yaml` as
  `context.scaffold_script`) copies the real source plus `CLAUDE.md`, locating the plugin with
  `ROOT="${EVAL_PLUGIN_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"` because the
  official runner passes no `EVAL_*` variables to it. The prompt asks for a feature that does
  **not** exist yet (grep first: an existing feature makes the agent correctly do nothing and the
  case fails for the wrong reason) and ends "Print the complete source of every file you change
  in your reply", so graders can read `last_message`; grade a specific file the agent must write
  with `target: { source: file, path: <path> }`. Never use `target: files` for code: in the
  official format it is only the list of created paths.
- **One negative-trigger case**: a plausible request the skill must NOT fire on
  (`tool_used: Skill, min: 0, max: 0, arm: both`).
- **One hook case** per guard hook: a `scaffold.sh` creates a scratch state (e.g. `git init`, a
  commit), ask the agent to run a command the hook blocks *and that the model will actually
  attempt* (not force-push to main, which models refuse unprompted), graders: attempted
  (`tool_used: Bash`), reported-blocked (regex), not-succeeded (regex not_contains).

Beyond the three starters: write **one case per remaining hook** and a **trigger + negative-trigger
case per remaining skill**, each with a `covers.yaml` listing ids from the `--list` run above. Then re-run
`node ${CLAUDE_PLUGIN_ROOT}/tools/config-coverage.mjs <plugin>` and report the coverage % to the
user. CI can enforce it with the Action's `coverage-min` input.

## 3. Smoke run

```
node ${CLAUDE_PLUGIN_ROOT}/tools/eval-shim.mjs <plugin> --runs 1 --ablation none --scaffold
```
Read the per-grader verdicts. Fix graders that fail for the wrong reason (prose match, missing
scaffold, unsatisfiable negative). Re-run once. Show the user the table.

## 4. Pin and budget: `.cdc.yml`

Write `<plugin>/.cdc.yml` from what the smoke run resolved:
```
node ${CLAUDE_PLUGIN_ROOT}/tools/cdc-config.mjs <plugin> init --model <aggregates.resolvedModels[0] of the smoke run> --harness <claude --version, first token>
```
Then **ask the user one question only**: how much they are willing to spend per month on this suite,
and set `budget.per_month_usd` to it (default 10). Explain in two sentences: the pinned track is the
baseline every PR is checked against; the canary runs the alias model on the latest Claude Code when
a release ships, at most every 72 h, never past the budget, and opens a bump PR after two greens.

## 5. Wire CI

Copy `${CLAUDE_PLUGIN_ROOT}/ci/config-drift-checker.yml` to `.github/workflows/config-drift-checker.yml`
**as is**. It is generic: it checks out the published `jameskomo/config-drift-checker` for the watch job
and uses `jameskomo/config-drift-checker/action@v1` with `plugin-dir: .`. Only adjust the `paths:` filters
to the directories that hold this repo's CLAUDE.md, skills, hooks and source. Never point it at
in-repo tool paths. If `gh` is available and authenticated, offer to run
`gh secret set ANTHROPIC_API_KEY` (the user pastes the key; never echo it) and
`gh secret set SLACK_WEBHOOK_URL`.

## 6. Hand-off checklist (print exactly)

1. Push. 2. Settings → Actions → General: workflow permissions *Read and write*, and tick *Allow
GitHub Actions to create and approve pull requests* (for bump/pin PRs). 3. Actions →
config-drift-checker → Run workflow (records the baseline; merge the pin PR it opens if `.cdc.yml`
had no pin). 4. The next Claude Code or model release runs the canary; regressions appear as a red
check, a PR comment, Slack and the drift index (`eval-results` branch → Pages). 5. After intentional
changes: Run workflow with `promote-baseline: true`. 6. Spend never passes `budget.per_month_usd`
in `.cdc.yml`; a skipped run says so in the job summary.

Versions

RepositoryScoreLicence
This page · jameskomo/config-drift-checker100Apache-2.0
HwangSlater/harness-fit100MIT
nesquikm/dev-process-toolkit100MIT
userKael/rastro100MIT
als-apg/osprey100BSD-3-Clause
dedujeong/gemba100Apache-2.0
eigenwise/eigenwise-toolshed100MIT
awrshift/agent-memory-kit100MIT
lifecycle-innovations-limited/claude-ops100MIT

Database snapshot: 13.11.2026 · package version d51e3c1a6c9f (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100cc-stack-setup
    Deploy or audit the full 5-layer Claude Code stack (CLAUDE.md + skills + hooks + agents + memory) in a repository. Use this skill whenever the user runs /cc-setup, asks to "set up Claude Code on this repo", "configure a new repo for Claude", "audit my Claude Code setup",…
  • 100working-with-claude-code
    Use when working with Claude Code CLI, plugins, hooks, MCP servers, skills, configuration, or any Claude Code feature - provides comprehensive official documentation for all aspects of Claude Code
  • 100working-with-claude-code
    Use when working with Claude Code CLI, plugins, hooks, MCP servers, skills, configuration, or any Claude Code feature - provides comprehensive official documentation for all aspects of Claude Code
  • 90write-case
    Author or fix an eval case (prompt.md, graders/*.md, case.yaml) for a Claude Code plugin, skill, or hook in the official `claude plugin eval` format. Use when the user says "write an eval for", "add a test case for this skill/hook", "my grader is wrong", or "how do I assert the…
  • 95plugin-structure
    This skill should be used when the user asks to "create a plugin", "scaffold a plugin", "understand plugin structure", "organize plugin components", "set up plugin.json", "use ${CLAUDE_PLUGIN_ROOT}", "add commands/agents/skills/hooks", "configure auto-discovery", or needs…
  • 100plugin-conventions
    ACTIVATE when creating a Claude Code plugin, writing plugin.json, marketplace.json, hooks.json, a hook script, or an evals/evals.json file. ACTIVATE for 'Claude plugin', 'plugin.json', 'marketplace.json', 'hooks.json', 'CLAUDE_PLUGIN_ROOT', 'permissionDecision', 'evals.json'.…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.