cli-qa
/cli-qa
Weryfikuje narzędzia wiersza poleceń end-to-end: uruchamia binarkę z przykładowymi wejściami, zbiera stdout, stderr i kod wyjścia jako osobne dowody oraz porównuje z plikami golden z redakcją zmiennych pól.
- Autor
- kylesnowschwartz ↗
- Repozytorium
- ralph-ban ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 24.08.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 2
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 24.08.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Weryfikuje narzędzia wiersza poleceń end-to-end: uruchamia binarkę z przykładowymi wejściami, zbiera stdout, stderr i kod wyjścia jako osobne dowody oraz porównuje z plikami golden z redakcją zmiennych pól.
Wyzwalacze z opisu
Argumenty
/cli-qa [scope of the CLI change to verify]
Oryginalny opis (EN)
This skill should be used when the user asks to "QA a CLI tool", "verify a command-line program", "check the binary's output", "validate exit codes", "compare against a golden file", or wants end-to-end behavioural verification of command-line tools. Drives a binary with representative inputs, captures stdout / stderr / exit code as separate evidence channels, asserts via golden-file comparison with volatile-field redaction.
Jak działa
- Bundled Resources
- Workflow
- Driving the binary
- Exit code conventions
- Pipeline gotchas
- Environment that warps output
- TTY-vs-pipe divergence
- Signal-handling assertions
- Golden-file comparison
- Evidence capture
Podgląd SKILL.md
---
name: cli-qa
description: This skill should be used when the user asks to "QA a CLI tool", "verify a command-line program", "check the binary's output", "validate exit codes", "compare against a golden file", or wants end-to-end behavioural verification of command-line tools. Drives a binary with representative inputs, captures stdout / stderr / exit code as separate evidence channels, asserts via golden-file comparison with volatile-field redaction.
argument-hint: "[scope of the CLI change to verify]"
---
# cli-QA
Drive the binary, capture stdout / stderr / exit separately, compare against the spec or a golden fixture.
**Scope**: $ARGUMENTS
If no scope was provided, read the recent changeset to determine which subcommands, flags, or output formats changed.
## Bundled Resources
- `references/golden-files.md` — golden-file redaction (timestamps, PIDs, paths, widths, ANSI) and regenerate-on-purpose idiom. Load when the spec asserts byte-equality against a fixture.
## Workflow
1. Read the changeset — `git diff`, `git log` — and identify the surface that changed: subcommand, flags, input parsing, output formatting, error reporting, exit-code semantics.
2. Build the binary in the worktree (`GOWORK=off go build ./...` for Go, project's build for others).
3. For each spec asserting CLI behaviour, drive the binary with a representative input.
4. Capture stdout, stderr, and exit code to *three separate files*. Never `2>&1` for the Oracle's transcript.
5. Compare the captures against the spec — equality, regex, golden-file with redaction.
6. Persist the transcript under `.agent-history/oracle/<card-id>/<timestamp>/`.
## Driving the binary
```bash
TXN=.agent-history/oracle/$CARD_ID/$(date +%Y%m%dT%H%M%S)
mkdir -p "$TXN"
# Three channels, three files; preserve exit code
./bin/mytool subcommand --flag value \
> "$TXN/stdout.txt" \
2> "$TXN/stderr.txt"
echo "$?" > "$TXN/exit.txt"
```
`echo $?` captures the exit before anything else clobbers it. Pipelines clobber `$?` unless `set -o pipefail`.
## Exit code conventions
`126`, `127`, `128+N` are POSIX-defined. `0`, `1`, `2` are *program-defined* and mean different things in different tools (`git` uses `1` for merge conflict; `grep` uses `1` for "no match"). Record the exit code as a fact; interpret only when the program documents its codes.
| Code | Defined by | What it tells the Oracle |
|---|---|---|
| 0 | program | exit was graceful — the program decided everything succeeded |
| 1, 2, 3, … | program | program-specific; check the program's docs/man page before interpreting |
| 126 | shell | command was found but not executable — a setup defect, not behaviour under test |
| 127 | shell | command was not found — the binary was not built; halt the QA |
| 128 + N | shell | the program was killed by signal N (e.g., 130 = 128 + SIGINT, 137 = 128 + SIGKILL) |
If the spec says "shall exit non-zero on bad input" without naming a code, mark `could-not-determine` and surface to the planner.
## Pipeline gotchas
`2>&1` collapses stderr into stdout — after that, "what went to stderr" is unanswerable. For multi-stage pipelines, `set -euo pipefail` makes any-stage failures propagate; without it, only the last stage's exit is consulted.
## Environment that warps output
CLIs check more than `isatty(1)`. Fix env-var levers for reproducibility:
| Variable | Effect when unset / default | What to set for deterministic capture |
|---|---|---|
| `TERM` | colour, cursor codes vary | `TERM=dumb` for plainest output |
| `NO_COLOR` | many tools respect this | `NO_COLOR=1` to disable colour even on TTYs |
| `CI` | many tools change output for CI | `CI=` (unset) or `CI=1` per intent |
| `LC_ALL` / `LANG` | number / date format vary | `LC_ALL=C` |
| `TZ` | local-time output drifts | `TZ=UTC` |
| `PAGER` | tools like `git log` paginate | `PAGER=cat` to disable |
| `COLUMNS` | line-wrapping width | `COLUMNS=80` for stable wraps |
`env -i HOME=$HOME PATH=$PATH TERM=dumb LC_ALL=C TZ=UTC ./bin/mytool ...` is the heavy form when the spec asserts byte-exact output.
## TTY-vs-pipe divergence
Many CLIs check `isatty(1)` and change output: colours, progress bars, pagination. The Oracle's redirected captures are pipes, so they record the non-TTY path. To assert TTY-only behaviour, force a PTY with `script`:
- `script` merges stdout and stderr under the PTY — separate channels are not recoverable. Pick TTY mode *or* separate channels.
- PTY transcripts carry `\r`, backspace, and ANSI control bytes; goldens need normalisation (`tr -d '\r'`, ANSI strip).
- Flag spelling differs by platform:
```bash
if [[ "$OSTYPE" == darwin* ]]; then
script -q /dev/null ./bin/mytool subcommand > "$TXN/transcript.txt"
else
script -q -c './bin/mytool subcommand' /dev/null > "$TXN/transcript.txt"
fi
echo "$?" > "$TXN/exit.txt"
```
## Signal-handling assertions
When the spec asserts shutdown behaviour ("shall exit cleanly on SIGINT"), drive the signal explicitly:
```bash
./bin/mytool serve >"$TXN/stdout.txt" 2>"$TXN/stderr.txt" &
PID=$!
# Poll for readiness — sleep is flaky for slow-starting binaries
for i in $(seq 1 60); do
if grep -q 'listening' "$TXN/stderr.txt" 2>/dev/null; then break; fi
sleep 0.1
done
kill -INT "$PID"
wait "$PID"; CODE=$?
echo "$CODE" > "$TXN/exit.txt"
```
If the binary handles SIGINT and exits with its own code, `wait` returns that code. If it doesn't and the kernel kills it, `wait` returns `130` (= 128 + SIGINT). The spec must say which case ("shall exit 0 on SIGINT" or "shall exit 130 uncaught") for the verdict to be unambiguous.
## Golden-file comparison
For large or structurally complex output, compare against a fixture. Redaction (replacing volatiles with stable placeholders) is what makes golden files non-flaky. See `references/golden-files.md`.
```bash
# Quick recipe: redact obvious volatiles, then diff
sed -E '
s/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:.+-]+/[TIMESTAMP]/g;
s/(pid )[0-9]+/\1[PID]/g;
s/\/tmp\/[A-Za-z0-9._-]+/[TMPPATH]/g;
' "$TXN/stdout.txt" > "$TXN/stdout.redacted.txt"
diff -u testdata/expected.txt "$TXN/stdout.redacted.txt" > "$TXN/diff.txt" || true
```
## Evidence capture
Save artefacts under `.agent-history/oracle/<card-id>/<timestamp>/`:
- `command.txt` — the exact command line invoked, one line
- `stdin.txt` — input fed to the program, if any
- `stdout.txt` — raw stdout
- `stdout.redacted.txt` — stdout after volatile-field redaction (when comparing to a golden)
- `stderr.txt` — raw stderr
- `exit.txt` — single line: the exit code as a number
- `diff.txt` — output of `diff -u golden observed`, when applicable
- `verdict.md` — APPROVE / REJECT / ESCALATE with the spec table filled in
## Side-effect assertions
When the spec asserts effects beyond stdout/stderr/exit, delegate:
- Database → `db-state-qa` (snapshot before/after, structural diff).
- Log content → `log-tail-qa` (bounded wait for pattern).
- File-system → `find` + `stat`.
Link the side-effect transcript from `verdict.md`.
## Rules
- **Three channels, three files.** Never `2>&1` in the capture. Stderr carries information the spec may assert.
- **Capture exit code immediately.** `echo $?` on the next line. Pipelines and chained commands clobber it.
- **Build the binary the Oracle exercises.** `GOWORK=off` for Go in worktrees. A stale binary makes the verdict meaningless.
- **Force a PTY when the spec asserts TTY-mode output.** `script -q` on macOS, `script -q -c` on Linux. Capture both paths when both matter.
- **Reproduce signal behaviour explicitly.** Send the signal; wait; capture the code. Implicit shutdowns (SIGTERM at end of pipeline) are different from asserted SIGINT handling.
- **Redact before diff.** Timestamps, PIDs, temp paths, terminal widths, ANSI escapes. The redaction list belongs in `references/golden-files.md`.
- **Regenerate goldens deliberately.** A golden updated to match observed output is no longer a spec — it is a record. The Oracle does not regenerate; it reports mismatch and lets the worker decide.
## Report Format
```
## CLI QA Report
**Scope**: <which subcommands / flags verified>
**Verdict**: APPROVE | REJECT | ESCALATE
### Build
- [PASS/FAIL] `<build command>` — <notes>
### Channels
| Trial | stdout | stderr | exit |
|---|---|---|---|
| 1 | <one-line summary or "see file"> | <same> | <code> |
### Specifications Verified
| Spec # | Assertion | Verified by | Verdict |
|--------|-----------|-------------|---------|
| 1 | (paste from bl show) | (file path / diff) | satisfied / unsatisfied / could-not-determine |
### Findings
1. <description with reproduction command and evidence path>
### Transcript
Path: `.agent-history/oracle/<card-id>/<timestamp>/`
Contents: <brief listing>
```
Pliki w paczce (2)
- SKILL.md 9 KB
- references/golden-files.md 6 KB
Wersje
W bazie nie ma innych skilli o nazwie cli-qa.
Migawka bazy z 13.11.2026 · wersja paczki 957adf757ab4 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100build-cli-project
Implementuje narzędzia wiersza poleceń w Rust po wyborze kształtu projektu: parsowanie argumentów, dyspozytor poleceń, stdin, stdout i stderr, kody wyjścia, konfiguracja, komunikaty błędów, testy i walidacja Cargo. - 100unix-cli
Podaje wytyczne projektowania narzędzi wiersza poleceń w duchu UNIX: konwencje POSIX, kody wyjścia, strumienie stdout i stderr, sygnały oraz skrypty powłoki. - 100build-cli
Pomaga tworzyć i rozbudowywać narzędzia wiersza poleceń w TypeScript, Go, Pythonie lub Rust: podpolecenia, typowane argumenty, priorytety konfiguracji, --json, kody wyjścia, --dry-run i testy uruchamiające binarkę. - 100effect-v4-cli
Buduje i przegląda programy wiersza poleceń w Effect v4: kody wyjścia, czysty stdout, komunikaty na stderr, podpowiedzi i testy interfejsów. - 100effect-v4-cli
Pomaga budować i przeglądać programy wiersza poleceń na Effect v4: kontrakt kodu wyjścia, czysty stdout, raportowanie błędów na stderr, prompty i widoki postępu TUI. - 86exploratory-run
Uruchamia jednorazowy kontrolowany eksperyment w sprawdzonym środowisku, zapisuje kod wyjścia i przycięty stdout oraz stderr.