Skip to content
Skills Library
AI & agents · Testing

init

/init

Initialize a new agentic-usability benchmark pipeline project. Use when setting up a new SDK benchmark, creating a config.json, or starting a new evaluation project.

Licence
Apache-2.0
Last repository change
11.06.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
6
Deprecation

No signals detected. Repository changed on 11.06.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Initialize a new agentic-usability benchmark pipeline project. Use when setting up a new SDK benchmark, creating a config.json, or starting a new evaluation project.

Arguments

/init [project-directory]
Original author description

Initialize a new agentic-usability benchmark pipeline project. Use when setting up a new SDK benchmark, creating a config.json, or starting a new evaluation project.

How it works

  1. Option 1: Interactive Wizard
  2. Option 2: Direct Config Creation
Original SKILL.md headings · 181 lines, 737 words
SKILL.md preview
---
name: init
description: Initialize a new agentic-usability benchmark pipeline project. Use when setting up a new SDK benchmark, creating a config.json, or starting a new evaluation project.
argument-hint: "[project-directory]"
disable-model-invocation: true
allowed-tools: Bash(agentic-usability *) Write Read Glob
---

# Initialize Pipeline Project

Set up a new agentic-usability benchmark pipeline in the given project directory.

```!
echo "Project directory: $ARGUMENTS"
```

You have two approaches:

## Option 1: Interactive Wizard

Run `agentic-usability init -p $ARGUMENTS` for a step-by-step interactive setup.

## Option 2: Direct Config Creation

If the user has described their SDK, create `config.json` directly. This is faster and allows you to tailor the config to their exact setup.

### Project Directory Structure

After init, the project should have:
```
<project>/
  config.json       # Configuration (you create this)
  suite.json        # Test suite (created by generate)
  results/          # Run results (created by eval/execute)
  cache/repos/      # Git repo cache (created automatically)
```

### config.json Schema

```json
{
  "privateInfo": [],
  "publicInfo": [],
  "agents": {},
  "targets": [],
  "sandbox": {}
}
```

#### `privateInfo` (required, non-empty array)

SDK source code and internal docs. Visible to generator and judge, **never to executor**. Each entry is a SourceConfig with a `type` discriminator:

**Local source** — filesystem path:
```json
{ "type": "local", "path": "./src", "subpath": "packages/core", "additionalContext": "Focus on the Builder API" }
```
Fields: `path` (required), `subpath` (optional), `additionalContext` (optional)

**Git source** — clone a repository:
```json
{ "type": "git", "url": "https://github.com/org/sdk.git", "branch": "main", "subpath": "src", "sparse": ["src/api"], "additionalContext": "..." }
```
Fields: `url` (required), `branch`, `subpath`, `sparse` (sparse checkout paths), `additionalContext` (all optional)

**URL source** — fetch documentation:
```json
{ "type": "url", "url": "https://internal-docs.example.com/api-ref", "additionalContext": "..." }
```
Fields: `url` (required), `additionalContext` (optional)

**Package source** — metadata about the SDK package:
```json
{ "type": "package", "name": "@example/sdk", "installCommand": "npm install @example/sdk", "language": "typescript", "additionalContext": "..." }
```
Fields: `name` (required), `installCommand`, `language`, `additionalContext` (all optional)

#### `publicInfo` (optional array)

Public docs and package info visible to **both executor and judge**. Same SourceConfig types as above. Typically includes:
- A `package` source so executors know what to install
- A `url` source for public documentation

#### `agents` (optional object)

| Role | Type | Runs in sandbox? | Secret required? |
|------|------|-------------------|-----------------|
| `generator` | AgentConfig | No (host) | No |
| `executor` | SandboxAgentConfig | **Yes** | **Yes** |
| `judge` | SandboxAgentConfig | **Yes** | **Yes** |
| `insights` | AgentConfig | No (host) | No |

**AgentConfig fields** (generator, insights):
- `command` (required): `"claude"`, `"codex"`, `"gemini"`, or custom CLI name
- `systemPrompt` (optional): supports `{{packageName}}` and `{{docsUrl}}` placeholders

**SandboxAgentConfig** — extends AgentConfig with required `secret`:
```json
{
  "command": "claude",
  "secret": { "value": "$ANTHROPIC_API_KEY" }
}
```

**AgentSecretConfig fields**:
- `value` (required): raw API key or `"$ENV_VAR"` reference
- `envVar`: env var name for key inside sandbox — auto-detected for known agents
- `baseUrl`: API base URL — auto-detected for known agents
- `baseUrlEnvVar`: env var for base URL override — auto-detected for known agents

**Known agent defaults** (auto-filled, user only needs `value`):

| command | envVar | baseUrl | baseUrlEnvVar |
|---------|--------|---------|---------------|
| `claude` | `ANTHROPIC_API_KEY` | `https://api.anthropic.com` | `ANTHROPIC_BASE_URL` |
| `codex` | `CODEX_API_KEY` | `https://api.openai.com/v1` | `OPENAI_BASE_URL` |
| `gemini` | `GEMINI_API_KEY` | `https://generativelanguage.googleapis.com` | `GEMINI_API_BASE_URL` |

**Custom agents** must explicitly set `envVar` and `baseUrl` in the secret.

#### `targets` (required, non-empty array)

Docker images for sandboxed execution:
```json
{ "name": "node-20", "image": "node:20-slim", "timeout": 1200, "additionalContext": "Node.js 20 with npm" }
```
Fields: `name` (required), `image` (required), `timeout` (optional, seconds), `additionalContext` (optional, included in generator prompt)

#### `sandbox` (required object, can be `{}`)

```json
{
  "concurrency": 3,
  "defaultTimeout": 600,
  "memoryMib": 2048,
  "cpus": 2,
  "secrets": {
    "EXTRA_API_KEY": {
      "value": "$EXTRA_KEY",
      "allowHosts": ["api.extra-service.com"],
      "allowHostPatterns": ["*.extra-service.com"]
    }
  },
  "env": {
    "LICENSE_KEY": "$MY_LICENSE_KEY"
  }
}
```

- `secrets`: TLS-injected secrets that never enter the VM. Each needs `value` and non-empty `allowHosts`.
- `env`: Plain env vars passed directly into sandbox. Values can use `$VAR` to reference host env.

#### `workspace` (optional)

```json
{ "template": "./workspace-template", "setupScript": "./setup.sh" }
```

For the full schema with all validation rules, see [config-schema.md](../_reference/config-schema.md).

### Complete Example

```json
{
  "privateInfo": [
    { "type": "local", "path": "./sdk-source", "additionalContext": "Main SDK source code" }
  ],
  "publicInfo": [
    { "type": "package", "name": "my-sdk", "installCommand": "npm install my-sdk", "language": "typescript" },
    { "type": "url", "url": "https://docs.my-sdk.io/getting-started" }
  ],
  "agents": {
    "generator": { "command": "claude" },
    "executor": { "command": "claude", "secret": { "value": "$ANTHROPIC_API_KEY" } },
    "judge": { "command": "claude", "secret": { "value": "$ANTHROPIC_API_KEY" } }
  },
  "targets": [
    { "name": "node-20", "image": "node:20-slim", "timeout": 1200 }
  ],
  "sandbox": {
    "concurrency": 3,
    "defaultTimeout": 600
  }
}
```

After creating config.json, run `agentic-usability generate -p <project>` to create the test suite.

Versions

RepositoryScoreLicence
This page · pspdfkit-labs/agentic-usability100Apache-2.0
CodaColaLight/nightmill100Apache-2.0
userKael/rastro100MIT
elalfner/relevo100MIT
abatyuk/okf100MIT
duronext/jig100MIT
sh5623/e2e-rail100MIT
Calvin-LLC/skills100MIT
arxdsilva/ai-loops100MIT

Database snapshot: 13.11.2026 · package version 5a285ad41881 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100eval
    Run the full evaluation pipeline (execute, judge, report) for an SDK usability benchmark. Use when running a complete benchmark end-to-end, resuming an interrupted pipeline, or checking pipeline status.
  • 100scaffold-init
    Initialize an AI-native development workspace with spec-driven workflow infrastructure. Use when starting a new project, setting up Claude Code on a codebase for the first time, or when the user says scaffold, initialize, set up AI workflow, onboard this repo, or similar. Do not…
  • 100scaffold-init
    Initialize an AI-native development workspace with spec-driven workflow infrastructure. Use when starting a new project, setting up Codex on a codebase for the first time, or when the user says scaffold, initialize, set up AI workflow, onboard this repo, or similar. Do not use…
  • 100init-pipeline
    This skill should be used when the user asks to set up or initialize the subagent-system pipeline for a project ('init pipeline', 'set up the pipeline', 'khoi tao pipeline'). Creates PIPELINE_CONFIG.md and checkpoint scaffolding.
  • 92init-project
    Use when setting up a new or existing project for AI-assisted development.
  • 100pm-init
    Initialize or refresh Project Memory for a repository. Use after amplifying a new repo, when the user says "init PM", "set up project memory", or when pm-index.md is missing or empty. Surveys the codebase, creates starter PM units, and populates the index.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.