Przejdź do treści
Biblioteka skilli
Programowanie · Testowanie

cli-qa

/cli-qa

Weryfikuje narzędzia wiersza poleceń end-to-end: uruchamia binarkę z przykładowymi wejściami, zbiera stdout, stderr i kod wyjścia jako osobne dowody oraz porównuje z plikami golden z redakcją zmiennych pól.

Repozytorium
ralph-ban ↗
Licencja
MIT
Ostatnia zmiana repozytorium
24.08.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
2
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 24.08.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Weryfikuje narzędzia wiersza poleceń end-to-end: uruchamia binarkę z przykładowymi wejściami, zbiera stdout, stderr i kod wyjścia jako osobne dowody oraz porównuje z plikami golden z redakcją zmiennych pól.

Wyzwalacze z opisu

QA a CLI toolverify a command-line programcheck the binaryvalidate exit codescompare against a golden file

Argumenty

/cli-qa [scope of the CLI change to verify]
Oryginalny opis (EN)

This skill should be used when the user asks to "QA a CLI tool", "verify a command-line program", "check the binary's output", "validate exit codes", "compare against a golden file", or wants end-to-end behavioural verification of command-line tools. Drives a binary with representative inputs, captures stdout / stderr / exit code as separate evidence channels, asserts via golden-file comparison with volatile-field redaction.

Jak działa

  1. Bundled Resources
  2. Workflow
  3. Driving the binary
  4. Exit code conventions
  5. Pipeline gotchas
  6. Environment that warps output
  7. TTY-vs-pipe divergence
  8. Signal-handling assertions
  9. Golden-file comparison
  10. Evidence capture
Sekcje z nagłówków SKILL.md, w oryginale · 185 linii, 1 239 słów
Podgląd SKILL.md
---
name: cli-qa
description: This skill should be used when the user asks to "QA a CLI tool", "verify a command-line program", "check the binary's output", "validate exit codes", "compare against a golden file", or wants end-to-end behavioural verification of command-line tools. Drives a binary with representative inputs, captures stdout / stderr / exit code as separate evidence channels, asserts via golden-file comparison with volatile-field redaction.
argument-hint: "[scope of the CLI change to verify]"
---

# cli-QA

Drive the binary, capture stdout / stderr / exit separately, compare against the spec or a golden fixture.

**Scope**: $ARGUMENTS

If no scope was provided, read the recent changeset to determine which subcommands, flags, or output formats changed.

## Bundled Resources

- `references/golden-files.md` — golden-file redaction (timestamps, PIDs, paths, widths, ANSI) and regenerate-on-purpose idiom. Load when the spec asserts byte-equality against a fixture.

## Workflow

1. Read the changeset — `git diff`, `git log` — and identify the surface that changed: subcommand, flags, input parsing, output formatting, error reporting, exit-code semantics.
2. Build the binary in the worktree (`GOWORK=off go build ./...` for Go, project's build for others).
3. For each spec asserting CLI behaviour, drive the binary with a representative input.
4. Capture stdout, stderr, and exit code to *three separate files*. Never `2>&1` for the Oracle's transcript.
5. Compare the captures against the spec — equality, regex, golden-file with redaction.
6. Persist the transcript under `.agent-history/oracle/<card-id>/<timestamp>/`.

## Driving the binary

```bash
TXN=.agent-history/oracle/$CARD_ID/$(date +%Y%m%dT%H%M%S)
mkdir -p "$TXN"

# Three channels, three files; preserve exit code
./bin/mytool subcommand --flag value \
  > "$TXN/stdout.txt" \
  2> "$TXN/stderr.txt"
echo "$?" > "$TXN/exit.txt"
```

`echo $?` captures the exit before anything else clobbers it. Pipelines clobber `$?` unless `set -o pipefail`.

## Exit code conventions

`126`, `127`, `128+N` are POSIX-defined. `0`, `1`, `2` are *program-defined* and mean different things in different tools (`git` uses `1` for merge conflict; `grep` uses `1` for "no match"). Record the exit code as a fact; interpret only when the program documents its codes.

| Code | Defined by | What it tells the Oracle |
|---|---|---|
| 0 | program | exit was graceful — the program decided everything succeeded |
| 1, 2, 3, … | program | program-specific; check the program's docs/man page before interpreting |
| 126 | shell | command was found but not executable — a setup defect, not behaviour under test |
| 127 | shell | command was not found — the binary was not built; halt the QA |
| 128 + N | shell | the program was killed by signal N (e.g., 130 = 128 + SIGINT, 137 = 128 + SIGKILL) |

If the spec says "shall exit non-zero on bad input" without naming a code, mark `could-not-determine` and surface to the planner.

## Pipeline gotchas

`2>&1` collapses stderr into stdout — after that, "what went to stderr" is unanswerable. For multi-stage pipelines, `set -euo pipefail` makes any-stage failures propagate; without it, only the last stage's exit is consulted.

## Environment that warps output

CLIs check more than `isatty(1)`. Fix env-var levers for reproducibility:

| Variable | Effect when unset / default | What to set for deterministic capture |
|---|---|---|
| `TERM` | colour, cursor codes vary | `TERM=dumb` for plainest output |
| `NO_COLOR` | many tools respect this | `NO_COLOR=1` to disable colour even on TTYs |
| `CI` | many tools change output for CI | `CI=` (unset) or `CI=1` per intent |
| `LC_ALL` / `LANG` | number / date format vary | `LC_ALL=C` |
| `TZ` | local-time output drifts | `TZ=UTC` |
| `PAGER` | tools like `git log` paginate | `PAGER=cat` to disable |
| `COLUMNS` | line-wrapping width | `COLUMNS=80` for stable wraps |

`env -i HOME=$HOME PATH=$PATH TERM=dumb LC_ALL=C TZ=UTC ./bin/mytool ...` is the heavy form when the spec asserts byte-exact output.

## TTY-vs-pipe divergence

Many CLIs check `isatty(1)` and change output: colours, progress bars, pagination. The Oracle's redirected captures are pipes, so they record the non-TTY path. To assert TTY-only behaviour, force a PTY with `script`:

- `script` merges stdout and stderr under the PTY — separate channels are not recoverable. Pick TTY mode *or* separate channels.
- PTY transcripts carry `\r`, backspace, and ANSI control bytes; goldens need normalisation (`tr -d '\r'`, ANSI strip).
- Flag spelling differs by platform:

```bash
if [[ "$OSTYPE" == darwin* ]]; then
  script -q /dev/null ./bin/mytool subcommand > "$TXN/transcript.txt"
else
  script -q -c './bin/mytool subcommand' /dev/null > "$TXN/transcript.txt"
fi
echo "$?" > "$TXN/exit.txt"
```

## Signal-handling assertions

When the spec asserts shutdown behaviour ("shall exit cleanly on SIGINT"), drive the signal explicitly:

```bash
./bin/mytool serve >"$TXN/stdout.txt" 2>"$TXN/stderr.txt" &
PID=$!

# Poll for readiness — sleep is flaky for slow-starting binaries
for i in $(seq 1 60); do
  if grep -q 'listening' "$TXN/stderr.txt" 2>/dev/null; then break; fi
  sleep 0.1
done

kill -INT "$PID"
wait "$PID"; CODE=$?
echo "$CODE" > "$TXN/exit.txt"
```

If the binary handles SIGINT and exits with its own code, `wait` returns that code. If it doesn't and the kernel kills it, `wait` returns `130` (= 128 + SIGINT). The spec must say which case ("shall exit 0 on SIGINT" or "shall exit 130 uncaught") for the verdict to be unambiguous.

## Golden-file comparison

For large or structurally complex output, compare against a fixture. Redaction (replacing volatiles with stable placeholders) is what makes golden files non-flaky. See `references/golden-files.md`.

```bash
# Quick recipe: redact obvious volatiles, then diff
sed -E '
  s/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:.+-]+/[TIMESTAMP]/g;
  s/(pid )[0-9]+/\1[PID]/g;
  s/\/tmp\/[A-Za-z0-9._-]+/[TMPPATH]/g;
' "$TXN/stdout.txt" > "$TXN/stdout.redacted.txt"

diff -u testdata/expected.txt "$TXN/stdout.redacted.txt" > "$TXN/diff.txt" || true
```

## Evidence capture

Save artefacts under `.agent-history/oracle/<card-id>/<timestamp>/`:

- `command.txt` — the exact command line invoked, one line
- `stdin.txt` — input fed to the program, if any
- `stdout.txt` — raw stdout
- `stdout.redacted.txt` — stdout after volatile-field redaction (when comparing to a golden)
- `stderr.txt` — raw stderr
- `exit.txt` — single line: the exit code as a number
- `diff.txt` — output of `diff -u golden observed`, when applicable
- `verdict.md` — APPROVE / REJECT / ESCALATE with the spec table filled in

## Side-effect assertions

When the spec asserts effects beyond stdout/stderr/exit, delegate:

- Database → `db-state-qa` (snapshot before/after, structural diff).
- Log content → `log-tail-qa` (bounded wait for pattern).
- File-system → `find` + `stat`.

Link the side-effect transcript from `verdict.md`.

## Rules

- **Three channels, three files.** Never `2>&1` in the capture. Stderr carries information the spec may assert.
- **Capture exit code immediately.** `echo $?` on the next line. Pipelines and chained commands clobber it.
- **Build the binary the Oracle exercises.** `GOWORK=off` for Go in worktrees. A stale binary makes the verdict meaningless.
- **Force a PTY when the spec asserts TTY-mode output.** `script -q` on macOS, `script -q -c` on Linux. Capture both paths when both matter.
- **Reproduce signal behaviour explicitly.** Send the signal; wait; capture the code. Implicit shutdowns (SIGTERM at end of pipeline) are different from asserted SIGINT handling.
- **Redact before diff.** Timestamps, PIDs, temp paths, terminal widths, ANSI escapes. The redaction list belongs in `references/golden-files.md`.
- **Regenerate goldens deliberately.** A golden updated to match observed output is no longer a spec — it is a record. The Oracle does not regenerate; it reports mismatch and lets the worker decide.
## Report Format

```
## CLI QA Report

**Scope**: <which subcommands / flags verified>
**Verdict**: APPROVE | REJECT | ESCALATE

### Build
- [PASS/FAIL] `<build command>` — <notes>

### Channels
| Trial | stdout | stderr | exit |
|---|---|---|---|
| 1 | <one-line summary or "see file"> | <same> | <code> |

### Specifications Verified
| Spec # | Assertion | Verified by | Verdict |
|--------|-----------|-------------|---------|
| 1 | (paste from bl show) | (file path / diff) | satisfied / unsatisfied / could-not-determine |

### Findings
1. <description with reproduction command and evidence path>

### Transcript
Path: `.agent-history/oracle/<card-id>/<timestamp>/`
Contents: <brief listing>
```
Pliki w paczce (2)
  • SKILL.md 9 KB
  • references/golden-files.md 6 KB

Wersje

W bazie nie ma innych skilli o nazwie cli-qa.

Migawka bazy z 13.11.2026 · wersja paczki 957adf757ab4 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100build-cli-project
    Implementuje narzędzia wiersza poleceń w Rust po wyborze kształtu projektu: parsowanie argumentów, dyspozytor poleceń, stdin, stdout i stderr, kody wyjścia, konfiguracja, komunikaty błędów, testy i walidacja Cargo.
  • 100unix-cli
    Podaje wytyczne projektowania narzędzi wiersza poleceń w duchu UNIX: konwencje POSIX, kody wyjścia, strumienie stdout i stderr, sygnały oraz skrypty powłoki.
  • 100build-cli
    Pomaga tworzyć i rozbudowywać narzędzia wiersza poleceń w TypeScript, Go, Pythonie lub Rust: podpolecenia, typowane argumenty, priorytety konfiguracji, --json, kody wyjścia, --dry-run i testy uruchamiające binarkę.
  • 100effect-v4-cli
    Buduje i przegląda programy wiersza poleceń w Effect v4: kody wyjścia, czysty stdout, komunikaty na stderr, podpowiedzi i testy interfejsów.
  • 100effect-v4-cli
    Pomaga budować i przeglądać programy wiersza poleceń na Effect v4: kontrakt kodu wyjścia, czysty stdout, raportowanie błędów na stderr, prompty i widoki postępu TUI.
  • 86exploratory-run
    Uruchamia jednorazowy kontrolowany eksperyment w sprawdzonym środowisku, zapisuje kod wyjścia i przycięty stdout oraz stderr.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.