test-cli-usability
/test-cli-usability
Write scenario tests that verify your CLI tool is usable by AI agents. Ensures commands work non-interactively, provide clear output, and don't hang on prompts. Use when you want to prove your CLI is agent-friendly.
- Author
- langwatch ↗
- Repository
- langwatch ↗
- Licence
- Apache-2.0
- Last repository change
- 27.10.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Write scenario tests that verify your CLI tool is usable by AI agents. Ensures commands work non-interactively, provide clear output, and don't hang on prompts. Use when you want to prove your CLI is agent-friendly.
Original author description
Write scenario tests that verify your CLI tool is usable by AI agents. Ensures commands work non-interactively, provide clear output, and don't hang on prompts. Use when you want to prove your CLI is agent-friendly.
How it works
- Prerequisites
- Step 1: Identify Your CLI Commands
- Step 2: Write Scenario Tests
- Step 3: Assert No Interactive Workarounds
- Step 4: Test Error Recovery
- Common Mistakes
SKILL.md preview
---
name: test-cli-usability
description: Write scenario tests that verify your CLI tool is usable by AI agents. Ensures commands work non-interactively, provide clear output, and don't hang on prompts. Use when you want to prove your CLI is agent-friendly.
license: MIT
compatibility: Requires @langwatch/scenario. Works with Claude Code and similar coding agents.
metadata:
category: recipe
---
# Test Your CLI's Agent Usability
This recipe helps you write scenario tests that verify your CLI tool works well when operated by AI agents (Claude Code, Cursor, Codex, etc.). A CLI that's agent-friendly means:
- All commands can run non-interactively (no stdin prompts that hang)
- Output is parseable and informative
- Error messages are clear enough for an agent to self-correct
- Help text enables discovery (`--help` works on every subcommand)
## Prerequisites
Install the Scenario SDK:
```bash
npm install @langwatch/scenario vitest @ai-sdk/openai
# or: pip install langwatch-scenario pytest
```
## Step 1: Identify Your CLI Commands
List every command your CLI supports. For each, note:
- Does it require interactive input? (MUST have a non-interactive alternative)
- What flags/options does it accept?
- What does it output on success/failure?
## Step 2: Write Scenario Tests
For each command, write a scenario test where an AI agent discovers and uses it:
```typescript
import scenario, { type AgentAdapter, AgentRole } from "@langwatch/scenario";
import { openai } from "@ai-sdk/openai";
import { describe, expect, it } from "vitest";
const myAgent: AgentAdapter = {
role: AgentRole.AGENT,
call: async (input) => {
// Your Claude Code adapter here
},
};
describe("CLI agent usability", () => {
it("discovers and uses the command non-interactively", async () => {
const result = await scenario.run({
name: "CLI command discovery",
description: "Agent discovers and uses the CLI to accomplish a task",
agents: [
myAgent,
scenario.userSimulatorAgent({ model: openai("gpt-6-luna") }),
scenario.judgeAgent({
model: openai("gpt-6-luna"),
criteria: [
"Agent used the CLI command correctly",
"Agent did not get stuck on interactive prompts",
"Agent did not need to pipe 'yes' or use 'expect' scripting",
],
}),
],
});
expect(result.success).toBe(true);
});
});
```
## Step 3: Assert No Interactive Workarounds
Add this assertion to every test:
```typescript
function assertNoInteractiveWorkarounds(state) {
const output = state.messages
.map((m) => (typeof m.content === "string" ? m.content : JSON.stringify(m.content)))
.join("\n");
expect(output).not.toMatch(/echo\s+["']?[yY](?:es)?["']?\s*\|/);
expect(output).not.toMatch(/\byes\s*\|/);
expect(output).not.toMatch(/expect\s+-c/);
expect(output).not.toMatch(/printf\s+["']\\n["']\s*\|/);
}
```
If this assertion fails, your CLI has an interactivity bug -- add `--yes`, `--force`, or `--non-interactive` flags to the offending commands.
## Step 4: Test Error Recovery
Write scenarios where the agent makes a mistake and must recover:
- Wrong command name -> agent reads `--help` and self-corrects
- Missing required argument -> agent reads error message and retries
- Authentication failure -> agent follows instructions in error output
## Common Mistakes
- Do NOT make commands that require stdin for essential operations -- always provide flag alternatives
- Do NOT use interactive prompts for confirmation without a `--yes` or `--force` flag
- Do NOT output errors without actionable guidance (the agent needs to know how to fix it)
- DO make `--help` comprehensive on every subcommand
- DO use non-zero exit codes for failures (agents check exit codes)
- DO output structured information (the agent can parse it)
Versions
The database has no other skills named test-cli-usability.
Identical content appears in 1 other GitHub folders. We show one copy with the highest score.
Database snapshot: 13.11.2026 · package version 45989e027d7c (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100cli-agent-friendly-audit
Audit a CLI tool against the agent-friendliness checklist from Zbigniew Sobiecki's "Building Agent-Friendly CLIs". Use when the user says "quality-cli-agent-friendly-audit", "cli-agent-friendly-audit", "audit this cli", "is my cli agent-friendly", or asks to check a CLI for agent ergonomics. - 95cli-for-agent
Design or review CLIs so coding agents can run them reliably: non-interactive flags, layered --help with examples, stdin/pipelines, fast actionable errors, idempotency, dry-run, and predictable structure. Use when building a CLI, adding commands, writing --help, or when the user… - 95tarn-api-testing
Write, run, debug, and iterate on API tests using Tarn — a CLI-first API testing tool with .tarn.yaml files and structured JSON output. Use when: writing API tests, running tarn commands, debugging test failures, generating tests from OpenAPI specs or curl commands, setting up… - 100write-tests
Write tests that pin real behavior instead of implementation details, config values, or lucky samples. Use when adding tests for new behavior, writing a regression test, fixing a brittle or flaky test, reviewing a test diff, or when a test breaks after a refactor or config… - 100write-tests
Write tests that pin real behavior instead of implementation details, config values, or lucky samples. Use when adding tests for new behavior, writing a regression test, fixing a brittle or flaky test, reviewing a test diff, or when a test breaks after a refactor or config… - 100agent-cli
Add agent-friendly --json NDJSON output to Python CLI scripts, or scaffold a complete cli_utils package for a project. Use this skill when the user wants to make scripts machine-readable for AI agents, add --json flags, convert print statements to structured JSON, build a CLI…