Skip to content
Skills Library
AI & agents · Testing

skill-forge

/skill-forge

Use when authoring, iterating, or refining a skill against an Ailly eval harness, driving a SKILL.md to a falsification gate (improved > 0 and regressed == 0) through assemble/run/eval/report. The harness-assembling loop pairs with ailly-skill-eval (which describes the harness anatomy) and delegates the authoring algorithm to skill-creator. Distinct from building the harness by hand or running one ad-hoc eval against a single prompt.

Repository
ailly ↗
Licence
MIT
Last repository change
15.10.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
0
HTML comments
0
URLs in SKILL.md
2
Deprecation

No signals detected. Repository changed on 15.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Use when authoring, iterating, or refining a skill against an Ailly eval harness, driving a SKILL.md to a falsification gate (improved > 0 and regressed == 0) through assemble/run/eval/report. The harness-assembling loop pairs with ailly-skill-eval (which describes the harness anatomy) and delegates the authoring algorithm to skill-creator. Distinct from building the harness by hand or running one ad-hoc eval against a single prompt.

Original author description

Use when authoring, iterating, or refining a skill against an Ailly eval harness, driving a SKILL.md to a falsification gate (improved > 0 and regressed == 0) through assemble/run/eval/report. The harness-assembling loop pairs with ailly-skill-eval (which describes the harness anatomy) and delegates the authoring algorithm to skill-creator. Distinct from building the harness by hand or running one ad-hoc eval against a single prompt.

How it works

  1. Entry-state resume
  2. The loop
  3. The two gates
  4. Two disciplines the loop must hold
  5. Pointers
Original SKILL.md headings · 131 lines, 1,006 words
SKILL.md preview
---
name: skill-forge
description: Use when authoring, iterating, or refining a skill against an Ailly eval harness, driving a SKILL.md to a falsification gate (improved > 0 and regressed == 0) through assemble/run/eval/report. The harness-assembling loop pairs with ailly-skill-eval (which describes the harness anatomy) and delegates the authoring algorithm to skill-creator. Distinct from building the harness by hand or running one ad-hoc eval against a single prompt.
---

# skill-forge

`skill-forge` runs an authoring algorithm on an Ailly eval harness as one loop.
It **delegates** the authoring steps (capture intent, write or refine the
`SKILL.md`, reason about improvements) to an *authoring role*, and it **owns**
the substrate steps (build the harness, run the arms, read the gate, commit).
The loop is **refine-centric and eval-first**: the falsification eval is the
spec, built from intent, and the loop drives a skill body toward passing it.

This skill is the rare, idempotent harness-assembling *loop*. Its pair,
[ailly-skill-eval](../ailly-skill-eval/SKILL.md), is the frequent within-harness
*reference*: the project anatomy, the two axes, the assertion palette, and the
falsification gate. `skill-forge` defers to it for all of that and never restates
it. It delegates the authoring algorithm to an authoring role, preferring
Anthropic's `skill-creator` when present (see
[references/loop.md](references/loop.md), "The authoring role"); the substrate
loop is identical regardless of which authoring strategy resolves.

For the long-form walk-through of each phase, read
[references/loop.md](references/loop.md). For the generation contract (what the
scaffold emits and the disciplines it must hold), read
[references/scaffold.md](references/scaffold.md). The forged artifact is a
`SKILL.md` conforming to the open
[Agent Skills standard](https://agentskills.io/home); the `assembly`,
`conversation`, and `evaluation` schemas live in [DESIGN.md](../../DESIGN.md).
This skill restates none of them.

## Entry-state resume

On entry, determine the resume point from what already exists, mirroring
`developer:ailly`'s resume table. The eval is built from **intent** (the
situations the skill should route on, and the output properties a correct
response must exhibit), never derived from the skill body, so it can exist before
a meaningful body does.

| Skill body? | Eval/harness? | Resume at |
| --- | --- | --- |
| no  | no  | build the eval from intent, create an empty/skeleton `SKILL.md`, then refine |
| yes | no  | build the eval from intent, then refine |
| no  | yes | create an empty/skeleton `SKILL.md`, then refine |
| yes | yes | refine directly |

Every row converges on **refine against the gate**. The rows differ only in how
much scaffolding precedes the first refinement. Re-entry is idempotent: a present
eval or skill is used as-is, never clobbered, so a second invocation on a green
skill is a no-op.

## The loop

The phases split across the delegate/own boundary. Steps marked *(delegate)* go
to the authoring role; steps marked *(own)* are the substrate `skill-forge`
drives. [references/loop.md](references/loop.md) walks each with its rationale.

0. **Determine the entry state** and resume at the first unsatisfied step (the
   table above).
1. **Capture intent** *(delegate)*. The role conducts the interview and research.
   The intent feeds both the eval and any skill draft.
2. **Author, stub, or refine `SKILL.md`** *(delegate)*: a `description:` that
   routes, a body that shapes output, a "Common Mistakes" section. An
   empty/skeleton body is a valid starting point. *Draft gate:* the author
   confirms each revision before it is evaluated.
3. **Reference the skill in place** *(own)*. The invocation assembly names the
   authored `skills/<name>/SKILL.md` via a `kind: external` prefix block. Nothing
   is vendored; an edit is picked up at the next `assemble`.
4. **Build the harness from intent** *(own)*: generate the discovery, invocation,
   and baseline assemblies, the prompts, the suites, and `check_<skill>.py`, per
   [references/scaffold.md](references/scaffold.md). *Review gate:* the author
   tightens every generated artifact before evaluating.
5. **Evaluate** *(own)*: `assemble`, `run`, `eval`, `report` for the three arms.
   The committed `runs/<ts-id>/` conversations and `evals/reports/` are the
   record.
6. **Read the gate** *(own)*: `report <baseline-id> <invocation-id>`, then check
   `improved > 0 && regressed == 0` and discovery routing at or above 0.9.
7. **Refine** *(delegate)*. If the skilled arm still fails an assertion it should
   satisfy (or `regressed > 0`, or a checker errored), hand the committed eval
   report back to the role for a revised `SKILL.md`, and re-evaluate. Loop to
   green.
8. **Done.** Commit. Every iteration's conversation, trace, and comparison report
   is already committed; there is no separate ledger.

## The two gates

- **Draft gate (after step 2):** the author confirms a skill revision before it
  is evaluated.
- **Review gate (after step 4):** the author tightens every generated artifact
  before the first evaluation.

These mirror `developer:ailly`'s draft discipline: generation proposes, the human
confirms, then the substrate runs.

## Two disciplines the loop must hold

- **Never encode the answer.** A generated discovery prompt describes the
  situation and never names the skill. The baseline-shared prefix and every
  baseline prompt contain no skill routing identifier (the bare `name`, or its
  `<plugin>:<name>` form). A leaky scaffold makes the baseline arm pass too, so
  `improved == 0`. When the gate reads `improved == 0`, the response is "re-frame
  the prompt at the situation," **never** "weaken the checker." The falsification
  gate catches a leak by failing loudly; it is the safety net that makes full
  scaffolding safe.
- **A null result is not a red.** Both arms passing (`unchanged_pass`) is a true
  null, a legitimate statement that a capable model already produces the pattern,
  and the loop does not iterate on it. The loop iterates only while the *skilled*
  arm fails, `regressed > 0`, or a checker errors. It never clears `improved == 0`
  by weakening the checker. The `check_<skill>.py` rules and the SKILL.md "Common
  Mistakes" bullets are kept 1:1 as a maintained invariant (both project the same
  intent), so refining a failure mode updates both, never the checker alone.

## Pointers

- [ailly-skill-eval](../ailly-skill-eval/SKILL.md) — the paired harness
  reference: project anatomy, the two axes, the assertion palette, the
  falsification gate. `skill-forge` defers to it for all harness anatomy.
- [references/loop.md](references/loop.md) — the long-form per-phase walk-through,
  the entry-state resume table, and the authoring-role contract and resolution
  order.
- [references/scaffold.md](references/scaffold.md) — the generation contract: what
  the scaffold emits, the in-place `kind: external` reference, the prompt-framing
  discipline, and the checker ↔ Common-Mistakes invariant.
- [e2e/clean-comments-review/](../../e2e/clean-comments-review/) — a worked forge,
  whose skill body lives at
  [skills/clean-comments-review/SKILL.md](../clean-comments-review/SKILL.md) and
  is referenced in place via `kind: external`.
- `skill-creator` — the authoring algorithm `skill-forge` delegates to (capture
  intent, the Skill Writing Guide, improvement reasoning) when installed.
- [Agent Skills standard](https://agentskills.io/home) — the open spec the forged
  `SKILL.md` conforms to.
- [DESIGN.md](../../DESIGN.md) — the authoritative `assembly`, `conversation`, and
  `evaluation` schemas.
Files in this package (3)
  • SKILL.md 8 KB
  • references/loop.md 8 KB
  • references/scaffold.md 6 KB

Versions

RepositoryScoreLicence
This page · davidsouther/ailly90MIT
tonydzi/second-brain-starter-kit100MIT
frankxai/starlight-intelligence-system100MIT
antoniocascais/claude-code-toolkit100AGPL-3.0
Adit-Jain-srm/skill-forge100MIT
sanyuan0704/sanyuan-skills100MIT
mphinance/alpha-skills98MIT
devpilotX/ai-skill-ai98MIT
rune-kit/rune95MIT

Database snapshot: 13.11.2026 · package version 9fa939047918 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100ailly-skill-eval
    Use when building or regression-testing an evaluation suite for a collection of LLM-agent skills — asserting that a SKILL.md edit improves or degrades the skills' discovery and invocation — distinct from running a single ad-hoc eval against one prompt.
  • 10004-ai-eval-harness-spec
    Use when specifying a reusable AI feature evaluation harness with datasets, graders, thresholds, calibration, regression gates, and run evidence; use ai-agent-eval-spec for multi-step tool-using agent evaluation.
  • 100eval-prompt
    Evaluate a PAL inference prompt using the promptfoo harness in eval/. Use when a prompt constant in src/ has changed or may have regressed, when scaffolding a new prompt eval from scratch, or when comparing a v2 candidate against the current baseline.
  • 100evals-harness
    Design eval harnesses — task schemas, metrics, dataset versioning, eval-as-code patterns. Use when asked to "build an eval harness", "set up eval-as-code", or "version our eval datasets".
  • 100harness-eval
    Write and run evals that prove a harness beats Claude without it, in the claude plugin eval format (prompt.md + graders/, with-plugin vs no-plugin arms, delta per case), with cost estimates and an experiment ledger. Use when writing eval cases or graders for a plugin, skill or…
  • 100eval-harness
    Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles. Use when a Claude Code workflow needs a formal eval before it is trusted or changed.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.