Skip to content
Skills Library
Programming · Planning

testing-strategy

/testing-strategy

Design a test suite for an open-source project that contributors can actually run and trust. Use when a project has no tests, when the suite is slow or flaky, when deciding what to test and at what level, when setting a coverage policy, or when contributors' PRs keep failing CI for unrelated reasons. Covers the test pyramid, fixtures, property-based and snapshot testing, cross-version matrices, flake elimination, and making tests runnable in one command. Also use for "how do I test this" and for reviewing test quality in a PR.

Repository
oss-skills ↗
Licence
Apache-2.0
Last repository change
08.08.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 08.08.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Design a test suite for an open-source project that contributors can actually run and trust. Use when a project has no tests, when the suite is slow or flaky, when deciding what to test and at what level, when setting a coverage policy, or when contributors' PRs keep failing CI for unrelated reasons. Covers the test pyramid, fixtures, property-based and snapshot testing, cross-version matrices, flake elimination, and making tests runnable in one command. Also use for "how do I test this" and for reviewing test quality in a PR.

Triggers from the description

how do I test this
Original author description

Design a test suite for an open-source project that contributors can actually run and trust. Use when a project has no tests, when the suite is slow or flaky, when deciding what to test and at what level, when setting a coverage policy, or when contributors' PRs keep failing CI for unrelated reasons. Covers the test pyramid, fixtures, property-based and snapshot testing, cross-version matrices, flake elimination, and making tests runnable in one command. Also use for "how do I test this" and for reviewing test quality in a PR.

How it works

  1. The one-command rule
  2. The pyramid, and where projects get it wrong
  3. What to actually test
  4. Techniques worth the investment
  5. Coverage policy
  6. Cross-version and cross-platform
  7. Killing flakes
  8. Contributor-facing test docs
  9. Anti-patterns
Original SKILL.md headings · 195 lines, 1,347 words
SKILL.md preview
---
name: testing-strategy
description: Design a test suite for an open-source project that contributors can actually run and trust. Use when a project has no tests, when the suite is slow or flaky, when deciding what to test and at what level, when setting a coverage policy, or when contributors' PRs keep failing CI for unrelated reasons. Covers the test pyramid, fixtures, property-based and snapshot testing, cross-version matrices, flake elimination, and making tests runnable in one command. Also use for "how do I test this" and for reviewing test quality in a PR.
---

# Testing Strategy

For an open-source project, the test suite has a second job beyond correctness: it is
the **contract that lets strangers change your code safely**. Without it, every
external PR is a risk you must personally verify, and you become the bottleneck.

## The one-command rule

A new contributor must be able to run the full suite in one command, from a fresh
clone, with no tribal knowledge:

```bash
make test          # or: npm test / pytest / cargo test / go test ./...
```

Requirements, all of them non-negotiable:

- **No manual database setup.** Use containers, or a file-backed engine in tests.
- **No secrets required.** Tests needing credentials are skipped by default with a
  clear message, and run only in CI with secrets present.
- **No network by default.** A suite that fails on a plane fails in a corporate proxy
  too, and looks like your bug.
- **Deterministic.** Seed randomness, freeze time, sort collections before comparing.
- **Fast enough to run before every commit.** Target under 60 seconds for the default
  suite; push the slow tests behind a flag.

If setup takes more than `git clone && make test`, contributions drop measurably. This
is the highest-leverage testing investment an OSS project can make.

## The pyramid, and where projects get it wrong

| Level | Share | Runtime | Tests |
|---|---|---|---|
| **Unit** | ~70% | ms | Pure logic, edge cases, error paths |
| **Integration** | ~20% | ~s | Component boundaries, real DB/filesystem |
| **End-to-end** | ~10% | ~10s+ | The two or three flows that must never break |

Two common failure shapes:

- **Ice cream cone** (mostly E2E) — slow, flaky, and when it fails it doesn't tell you
  where. Symptom: "just re-run CI" is normal team advice.
- **Hourglass** (units + E2E, no integration) — every component works, the assembly
  doesn't. Symptom: bugs are always at the seams.

Test the *public API*, not internals. Tests bound to private functions turn every
refactor into a test rewrite, which teaches contributors that tests are an obstacle.

## What to actually test

Prioritize by consequence-of-failure, not by ease:

1. **The documented behavior.** Every promise in the README should have a test. This
   catches the most damaging class of bug — the one that makes your docs a lie.
2. **Error paths.** Most projects test the happy path and ship broken error handling,
   which users hit precisely when they are already frustrated.
3. **Boundaries.** Empty, one, many, max. Zero, negative, off-by-one. Unicode,
   emoji, RTL text, embedded newlines. Paths with spaces. Windows line endings.
4. **Every bug you fix.** A regression test at fix time is the cheapest test you will
   ever write, and it names its own reason for existing.
5. **The public API shape itself** — snapshot the exported surface so accidental
   breaking changes fail CI (see `api-design`).

Do not test: framework behavior, third-party libraries, getters and setters, or
anything whose test is a restatement of the implementation.

## Techniques worth the investment

**Table-driven tests** — the highest density of coverage per line of test code, and
the easiest for contributors to extend when they find a new case.

```go
tests := []struct{ name, in, want string }{
    {"empty", "", ""},
    {"unicode", "café", "cafe"},
    {"emoji", "a👍b", "ab"},
}
for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) { /* ... */ })
}
```

**Property-based testing** (Hypothesis, fast-check, proptest, QuickCheck) — for
parsers, serializers, encoders, and anything with an inverse. One property replaces a
hundred examples and finds the case you would never have written:

```python
@given(st.text())
def test_roundtrip(s):
    assert decode(encode(s)) == s
```

**Snapshot tests** — good for CLI output, generated code, and rendered markup. Two
rules or they rot: review every snapshot diff as if it were source code, and never run
`--update-snapshots` without reading the diff. An auto-approved snapshot suite tests
nothing.

**Golden files** for compilers, formatters, and codegen: input file → expected output
file, both in the repo, regenerable with one flag.

**Mutation testing** (`mutmut`, `Stryker`, `cargo-mutants`) — the honest answer to "is
our coverage real". Expensive; run it quarterly or in a nightly job, not per-PR.

## Coverage policy

Coverage is a *diagnostic*, not a target. Enforcing 100% produces tests written to
satisfy the instrument.

A policy that works in practice:

- **Track coverage; do not gate on the absolute number.** Publish it, watch the trend.
- **Gate on coverage of the diff** — new code should come with tests. Codecov and
  similar report this natively; it is a much fairer bar for contributors than a global
  percentage they cannot influence.
- **Exempt generated code, `__main__` blocks, and platform-specific branches** in
  config, so the number means something.
- **Uncovered error paths are the signal that matters.** If your happy path is at 95%
  and error handling is at 20%, coverage tooling is telling you exactly what to do.

## Cross-version and cross-platform

Open-source code runs where you do not. Decide the support matrix explicitly, publish
it, and test it — an untested claim of Windows support is worse than no claim.

```yaml
strategy:
  fail-fast: false
  matrix:
    os: [ubuntu-latest, macos-latest, windows-latest]
    version: ['3.10', '3.13']          # oldest supported + newest
    include:
      - { os: ubuntu-latest, version: '3.11' }   # fill the middle on one OS only
```

`fail-fast: false` is important: you want to see *all* failing cells, not the first.
Full cartesian matrices waste CI minutes — test the corners exhaustively and the middle
on one platform. See `ci-pipelines`.

Windows deserves specific attention because it is where cross-platform bugs live: path
separators, `CRLF` vs `LF`, case-insensitive filesystems, `PATH` length limits, and
no `fork()`.

## Killing flakes

A flaky test is worse than no test: it trains everyone, including you, to ignore red
CI. Treat a flake as a P1 bug.

Root causes, in the order you should suspect them:

1. **Time** — `sleep()` as synchronization, tests that fail near midnight or DST
   boundaries, timeouts tuned to a fast laptop. Fix: inject a clock; poll for a
   condition with a generous timeout instead of sleeping a fixed duration.
2. **Order dependence** — shared global state, leaked singletons, a database not
   reset between tests. Fix: run with `--shuffle`/`-p` random ordering in CI so
   order dependence fails immediately rather than mysteriously.
3. **Concurrency** — real races in the code under test. These are the valuable ones;
   the test is telling you the truth.
4. **Network and external services** — never in unit tests; stub at the boundary.
5. **Resource contention** — ports, temp file names, CPU starvation on shared runners.
   Fix: bind to port 0 and read back the assigned port; use unique temp dirs.

Process: quarantine the flake into a separate job immediately so it stops blocking
contributors, open an issue with the failing logs, and fix or delete it within a
sprint. A permanently quarantined test is a deleted test with extra steps — be honest
and delete it.

## Contributor-facing test docs

In `CONTRIBUTING.md`, concretely:

```markdown
## Tests
make test            # full suite (~40s)
make test-unit       # fast, no containers
make test-e2e        # requires Docker
pytest tests/test_parser.py::test_unicode -x   # single test
```

Also state: what a new feature needs (tests + docs), what a bug fix needs (a
regression test), and what happens when CI is red for unrelated reasons — because a
contributor who cannot tell "my fault" from "not my fault" usually just leaves.

## Anti-patterns

- **Tests requiring undocumented local setup.** The contribution killer.
- **`sleep(2)` as synchronization.** Slow *and* flaky.
- **Asserting on log output** for behavior that has a return value.
- **One test asserting fifteen things.** When it fails you learn nothing.
- **Mocking the thing under test.** You are testing your mock.
- **Mocks so deep the test encodes the implementation.** Any refactor breaks it.
- **Skipped tests with no issue link.** Delete them or fix them.
- **`if (CI) skip()`.** The test is a lie in exactly the environment that matters.
- **Coverage gates that reject a one-line docs-adjacent fix.** You just taught a
  first-time contributor that this project is not worth the trouble.

Versions

RepositoryScoreLicence
This page · the-open-agent/oss-skills100Apache-2.0
komluk/scaffolding100MIT
oborchers/fractional-cto100MIT
anastasiyaw/codex-claude-code-config100MIT
andreibesleaga/gabbe100CC-BY-4.0
organizzolini/codebase100MIT
shinpr/agentic-code100MIT
sagargupta16/claude-code-recipes100MIT
officialdb/TES-SKILLS100Apache-2.0

Database snapshot: 13.11.2026 · package version f79caa9d8bb7 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100sota-testing
    State-of-the-art software testing strategy and practice (2026) for designing test strategy, writing unit/integration/e2e tests, or auditing test suites. Covers suite shape (pyramid/trophy/honeycomb), test design quality (behavior-first, AAA, determinism, smells), test doubles…
  • 100frontend-testing-strategy
    Reference for building a reliable frontend test suite - choosing what to test at unit, component, integration and end-to-end levels; the test pyramid and its trade-offs; testing user-visible behavior with React Testing Library; Vitest and Playwright usage; API mocking at the…
  • 90testing-code
    Use when writing tests ("write tests for X", "add a test for this", "write a failing test first", "red green refactor this"), or when a test is flaky or the suite is slow. Not for reviewing a diff (use reviewing-code). Not for deciding what a good test is (use…
  • 100testing-audit
    Audit the test suite for coverage gaps, test quality, flaky tests, and strategy alignment. Use when assessing test health before a release, or when tests feel weak, slow, or flaky.
  • 90qa-snapshot-tests
    Authors snapshot tests with governance — what to snapshot, granularity, update-review policy, and rot prevention. Output: a snapshot test suite plus a short update-review policy doc. Pulled by the `qa` specialist. TRIGGER: "add snapshot tests for X", "snapshot test the rendered…
  • 100eng-testing
    Use when writing tests, designing test strategy, or reviewing test coverage. Covers test pyramid, naming, mocking, and flaky test policy.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.