foreman-testing
/foreman-testing
Use when running, extending, or debugging the Foreman bats suite - covers the gate mutex, baseline and skip-budget ratchets, positive controls, and bats assertion semantics.
- Author
- charleshoskinson ↗
- Repository
- foreman ↗
- Licence
- Apache-2.0
- Last repository change
- 15.09.2026
Assessment and security
- Files in this package
- 3
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 15.09.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Use when running, extending, or debugging the Foreman bats suite - covers the gate mutex, baseline and skip-budget ratchets, positive controls, and bats assertion semantics.
Original author description
Use when running, extending, or debugging the Foreman bats suite - covers the gate mutex, baseline and skip-budget ratchets, positive controls, and bats assertion semantics.
How it works
- Running tests
- Assertion semantics -- the biggest trap
- Ratchets: baseline and skip-budget
- Gate mode: shadow vs enforce
- Controls: proving a check discriminates
- Style
SKILL.md preview
--- name: foreman-testing description: Use when running, extending, or debugging the Foreman bats suite - covers the gate mutex, baseline and skip-budget ratchets, positive controls, and bats assertion semantics. --- # foreman-testing Operational rules for running and extending the Foreman bats suite (`tests/*.bats`, driven by `tests/run.sh`). Every rule here was learned by something breaking; follow them exactly, do not rediscover them. ## Running tests Run everything through `tests/run.sh`, or take the mutex yourself: ```bash flock /root/.foreman/gate.lock bats --formatter tap tests/some.bats # or, for the full policy-aware run: tests/run.sh ``` - **A raw `flock ... bats ...` wrapper can deadlock** on files that shell out to `tests/run.sh` themselves (`tests/test-policy.bats` does): `tests/run.sh`'s own mutex is reentrant via an exported `FOREMAN_BATS_MUTEX_HELD`, but a bare outer `flock` never sets it, so the nested call blocks on a lock its own parent holds. Prefer `bash tests/run.sh <files>`. Full mechanism: `references/bats-traps.md`. - **Exactly one bats runner at a time, host-wide.** `tests/run.sh` takes `$HOME/.foreman/gate.lock` via `flock` before running anything (falls back to an atomic `mkdir` lock when `flock` is unavailable). Concurrent runs flake the timing-sensitive tests, notably `tests/watch.bats` and `tests/lane-run.bats`. - **Keep the host quiet during a gate run.** A concurrent build or agent spawn can flake the VTICK fake-clock timing assertions in `watch.bats` even while the mutex is held, because CPU contention stretches the real wall-clock scheduling slack those assertions bound. - **Per-file timeout is 600s** (`TEST_FILE_TIMEOUT_S`, `--kill-after=30`). A file that hangs is killed and recorded as `TIMEOUT` in the slice report rather than left to hang the host. See `references/bats-traps.md` for why this exists. ## Assertion semantics -- the biggest trap Bats runs every test body under `errexit`. A **bare command's exit status is a complete assertion** -- `grep -q pattern file` and a `jq -e` filter piped to `/dev/null` each fail the test on their own if they exit non-zero. No `assert_*` helper or bracket-test wrapper is required or implied. A test-quality scanner that only recognizes bracket tests or `assert_*` calls as "real checks" undercounts real checks. Full detail and a measured example: `references/bats-traps.md`. - **Never write a bare `skip`.** Always `skip "reason"`. Use the `require_*` helpers in `tests/lib/preconditions.bash` (`require_platform`, `require_tool`, `require_non_root`, `require_built`, `require_no_live_vendor`) -- each skips with an actionable reason instead of a bare one. - **Never put a literal `@test` at column 0 inside a heredoc** in a `.bats` file. Bats parses `.bats` files line-wise; a heredoc-embedded `@test` reads as a new test declaration and silently breaks the enclosing test. Build such fixtures with `printf` instead -- see `references/bats-traps.md`. ## Ratchets: baseline and skip-budget `tests/baseline.tsv` (one expected pass count per file) and `tests/skip-budget.tsv` (permitted skips per file x platform, platform is `linux|wsl|windows`) are edited **deliberately**, never regenerated from a run. Adding a new `.bats` file requires one `tests/baseline.tsv` row and three `tests/skip-budget.tsv` rows (one per platform) -- `tests/run.sh` errors on a file missing either. Full mechanics: `references/gate-and-ratchets.md`. - Raising a baseline **tightens** it -- correct when a file legitimately gained tests. - Lowering a baseline is a **regression** and needs justification. - Host-dependent counts are real (`tests/nats-bridge.bats` needs `nats-server`/`nats` on PATH; CI installs pinned versions). Do not "fix" a baseline to match an incomplete environment -- install the dependency instead. ## Gate mode: shadow vs enforce `tests/run.sh` defaults `TEST_GATE_MODE=shadow`. In shadow mode, skip-budget, pass-baseline, and bare-skip failures do **not** affect exit status -- a run can print `RESULT ERROR runner_errors=1` and still exit 0. **Never read the exit code alone; read the `RESULT` and `TOTAL` lines.** Enforcing the gate (`TEST_GATE_MODE=enforce`) is tag criterion 2 and requires the suite to complete on three consecutive runs first. ## Controls: proving a check discriminates A check that only passes on known-good input proves nothing alone -- a hard-wired `exit 0` would pass too. `tests/lib/positive-control.bash` proves a check answers differently on a known-bad and a known-good input in the same run; a check that classifies both arms identically is rejected. `tests/lib/check-inventory.sh` sweeps the whole tree (never a diff) into `tests/.check-inventory.tsv`; every `check_id` must appear in `tests/positive-control-registry.tsv` or the shrinking `tests/positive-control-todo.tsv`, or the build fails. `tests/inject-regressions.sh` seeds known defects into an isolated `git archive HEAD` copy -- the working tree is never mutated -- and asserts the owning slice fails for each. Full detail: `references/gate-and-ratchets.md`. ## Style Markdown here must satisfy `markdownlint-cli2` per the repo's `.markdownlint-cli2.jsonc`: no consecutive blank lines (MD012), fenced code blocks carry a language, headings are surrounded by blank lines, single trailing newline.
Files in this package (3)
- SKILL.md 5 KB
- references/bats-traps.md 6 KB
- references/gate-and-ratchets.md 8 KB
Versions
The database has no other skills named foreman-testing.
Database snapshot: 13.11.2026 · package version 408cfa707565 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100bats-testing-patterns
Master Bash Automated Testing System (Bats) for comprehensive shell script testing. Use when writing tests for shell scripts, CI/CD pipelines, or requiring test-driven development of shell utilities. - 95bats-testing-patterns
Master Bash Automated Testing System (Bats) for comprehensive shell script testing. Use when writing tests for shell scripts, CI/CD pipelines, or requiring test-driven development of shell utilities. - 100bats-testing-patterns
Master Bash Automated Testing System (Bats) for comprehensive shell script testing. Use when writing tests for shell scripts, CI/CD pipelines, or requiring test-driven development of shell utilities. - 100bats-testing-patterns
Master Bash Automated Testing System (Bats) for comprehensive shell script testing. Use when writing tests for shell scripts, CI/CD pipelines, or requiring test-driven development of shell utilities. - 100bats-testing-patterns
Master Bash Automated Testing System (Bats) for comprehensive shell script testing. Use when writing tests for shell scripts, CI/CD pipelines, or requiring test-driven development of shell utilities. - 100bats-testing
Configures Bats-core (Bash Automated Testing System) for testing CLI tools, shell scripts, and Unix programs - `.bats` test files with `@test` blocks, `run` to capture command exit + output, `[ \"$status\" -eq 0 ]` and `[ \"$output\" = ... ]` assertions, `setup`/`teardown`…