testing-principles

---
name: testing-principles
description: This skill provides language-agnostic testing principles including TDD, test quality, coverage standards, and test design patterns. Automatically loaded when writing tests, designing test strategies, reviewing test quality, or when the user mentions "TDD", "test coverage", "unit tests", or "test patterns".
---

# Language-Agnostic Testing Principles

## Core Testing Philosophy

1. **Tests are First-Class Code**: Maintain test quality equal to production code
2. **Fast Feedback**: Tests should run quickly and provide immediate feedback
3. **Reliability**: Tests should be deterministic and reproducible
4. **Independence**: Each test should run in isolation

## Test Selection

A test is code that has to be read, run, and kept in step with every change. It earns that cost only when it catches a defect nothing else catches. More tests on the same behaviour do not add safety; they add places to update and noise when one change breaks five tests at once.

**A test is written only when all four hold:**

1. **Unique defect.** It names the concrete bug it catches ("a whitespace-only title is saved"), and no other test — at any level — catches that bug. When two tests catch the same bug, keep the one at the lower level and delete the other.
2. **Lowest sufficient level.** A rule is proven once, at the lowest level that can prove it. A pure rule (validation, mapping, calculation) is one table-driven unit test with all its cases as rows. An integration test proves only what a unit test cannot see: the real database, authorization and ownership, persistence, wiring across components. One integration case is enough to prove that a unit-tested rule is wired in; it does not repeat the rule's variants.
3. **Behaviour, not restatement.** No test asserts a constant's value, an enum's member list, a type's shape, or a mock's call count as the outcome. These pass by construction and fail only when the code is edited on purpose.
4. **Our code, not the library's.** Do not test that zod trims, that the ORM saves a row, that the framework returns its default error body. Test the project's rule built on top of them — once.

**Traceability.** Every test links to an acceptance criterion or a named risk (in its title or a one-line comment). A test that links to neither and names no unique defect is deleted.

**Budget per part.** Integration and E2E tests: the per-part limits in `integration-e2e-testing` (Test Type Definition and Limits), table-driven cases encouraged; E2E only for the journeys whose failure the user would notice first. Unit tests: one per unit with branching logic, cases as rows. A plan that needs more states which defect each extra test catches that the budget's tests do not.

**Timing-sensitive checks** (latency, throughput) do not run in the regular suite: they live in a separate script run on demand, so a slow machine does not turn the suite red.

## Test-Driven Development (TDD)

TDD applies to behaviour that passes Test Selection. A task step that adds no behaviour of its own — a constant, a schema declaration, module wiring, a type — gets no test of its own; the behaviour that uses it is tested where it is observable.

### The RED-GREEN-REFACTOR Cycle

**Always follow this cycle:**

1. **RED**: Write a failing test first
   - Write the test before implementation
   - Ensure the test fails for the right reason
   - Verify test can actually fail

2. **GREEN**: Write minimal code to pass
   - Implement just enough to make the test pass
   - Don't optimize prematurely
   - Focus on making it work

3. **REFACTOR**: Improve code structure
   - Clean up implementation
   - Eliminate duplication
   - Improve naming and clarity
   - Keep all tests passing

4. **VERIFY**: Ensure the affected tests still pass
   - Run the tests you wrote and the unit test files of the modules you touched; an integration or e2e file only when you wrote or edited it
   - The full suite is the quality gate's run, not a step of this cycle
   - A passing run is not repeated until the code changes again

### TDD Benefits

- Better design through testability requirements
- Every behaviour that matters has a test written before it
- Living documentation of expected behavior
- Confidence to refactor

## Quality Requirements

### Coverage Standards

- Coverage is a map, not a target: read the report for untested branches in logic that matters, and add a test only when Test Selection passes for it
- This skill sets no percentage: a percentage is reached most cheaply with tautological tests. A threshold the project's own CI enforces is met, never with tests that fail Test Selection
- Critical paths and business rules are covered; boilerplate, wiring, and declarations are not chased

### Test Characteristics

All tests must be:

- **Independent**: No dependencies between tests
- **Reproducible**: Same input always produces same output
- **Fast**: Complete test suite runs in reasonable time
- **Self-checking**: Clear pass/fail without manual verification
- **Timely**: Written close to the code they test

## Test Types

### Unit Tests

**Purpose**: Test individual components in isolation

**Characteristics**:
- Test single function, method, or class
- Fast execution (milliseconds)
- No external dependencies
- Mock external services
- Majority of your test suite

**Example Scope**:
```
✓ Test calculateTotal() function
✓ Test UserValidator class
✓ Test parseDate() utility
```

### Integration Tests

**Purpose**: Test interactions between components

**Characteristics**:
- Test multiple components together
- May include database, file system, or APIs
- Slower than unit tests
- Verify contracts between modules
- Smaller portion of test suite

**Example Scope**:
```
✓ Test UserService with Database
✓ Test API endpoint with authentication
✓ Test file processing pipeline
```

### End-to-End (E2E) Tests

**Purpose**: Test complete workflows from user perspective

**Characteristics**:
- Test entire application stack
- Simulate real user interactions
- Slowest test type
- Fewest in number
- Highest confidence level

**Example Scope**:
```
✓ Test user registration flow
✓ Test checkout process
✓ Test complete report generation
```

### Test Pyramid

Follow the test pyramid structure:
```
    /\    ← Few E2E Tests (High confidence, slow)
   /  \
  /    \  ← Some Integration Tests (Medium confidence, medium speed)
 /      \
/________\ ← Many Unit Tests (Fast, foundational)
```

### Property-Based Testing

**Purpose**: Verify invariants that hold for ALL possible inputs, not just specific test cases.

**When to use:**
- Mathematical properties (commutativity, associativity, idempotency)
- Serialization/deserialization roundtrips (encode → decode = identity)
- Data transformation reversibility (sort stability, filter idempotency)
- Invariants across large input spaces (e.g., "output length <= input length")
- Boundary conditions that are hard to enumerate manually

**When NOT to use:**
- UI interaction testing (use integration/E2E tests instead)
- Specific business rules with known expected outputs (use example-based tests)
- External API integration (non-deterministic, use mocks)

**Structure:**
```
PROPERTY: "Description of the invariant that must always hold"
FOR ALL: generator description (e.g., "arbitrary strings of length 1-1000")
ASSERT: invariant expression (e.g., "decode(encode(input)) === input")
SHRINK: how to minimize failing cases to smallest reproducer
```

**Relationship to example-based tests**: Property tests complement (not replace) example-based tests. Use property tests for invariants, example-based tests for specific business scenarios with known expected outputs.

## Test Design Principles

### AAA Pattern (Arrange-Act-Assert)

Structure every test in three clear phases:

```
// Arrange: Setup test data and conditions
user = createTestUser()
validator = createValidator()

// Act: Execute the code under test
result = validator.validate(user)

// Assert: Verify expected outcome
assert(result.isValid == true)
```

**Adaptation**: Apply this structure using your language's idioms (methods, functions, procedures)

### One Assertion Per Concept

- Test one behavior per test case
- Multiple assertions OK if testing single concept
- Split unrelated assertions into separate tests

**Good** — one rule, its cases as rows:
```
test.each([
  ["missing email",  { age: 30 },                 "email is required"],
  ["malformed email", { email: "x", age: 30 },     "email is invalid"],
  ["negative age",   { email: "a@b.c", age: -1 }, "age must be positive"],
])("rejects %s", (_, input, message) => ...)
```

**Bad**:
```
test("validates user") // Asserts unrelated rules at once; one failure hides the others
```

### Descriptive Test Names

Test names should clearly describe:
- What is being tested
- Under what conditions
- What the expected outcome is

**Recommended format**: `"should [expected behavior] when [condition]"`

**Examples**:
```
test("should return error when email is invalid")
test("should calculate discount when user is premium")
test("should throw exception when file not found")
```

**Adaptation**: Follow your project's naming convention (camelCase, snake_case, describe/it blocks)

## Test Independence

### Isolation Requirements

- **No shared state**: Each test creates its own data
- **No execution order dependency**: Tests pass in any order
- **Clean up after tests**: Reset state, close connections
- **Avoid global variables**: Use local test data

### Setup and Teardown

- Use setup hooks to prepare test environment
- Use teardown hooks to clean up resources
- Keep setup minimal and focused
- Ensure teardown runs even if test fails

## Mocking and Test Doubles

### When to Use Mocks

- **Mock external dependencies in unit tests**: APIs, databases, file systems — an integration test uses the real database (Test Selection)
- **Mock slow operations**: Network calls, heavy computations
- **Mock unpredictable behavior**: Random values, current time
- **Mock unavailable services**: Third-party services

### Mocking Principles

- Mock at boundaries, not internally
- Keep mocks simple and focused
- Verify mock expectations when relevant
- Don't mock external libraries/frameworks you don't control (prefer adapters)

## Test Quality Practices

### Keep Tests Active

- **Fix failing tests' causes**: a failing existing test is resolved by fixing the implementation; deleting or weakening an existing test is escalated, never done to get green
- **No commented-out or skipped tests**: fix them, or escalate their removal
- **Keep tests running**: Broken tests lose value quickly
- **Maintain test suite**: Refactor tests as needed

### Test Code Quality

- Apply same standards as production code
- Use descriptive variable names
- Extract test helpers to reduce duplication
- Keep tests readable and maintainable
- Review test code thoroughly

### Test Helpers and Utilities

- Create reusable test data builders
- Extract common setup into helper functions
- Build test utilities for complex scenarios
- Share helpers across test files appropriately

## What to Test

### Focus on Behavior

**Test observable behavior, not implementation**:

✓ **Good**: Test that function returns expected output
✓ **Good**: Test that correct API endpoint is called
✗ **Bad**: Test that internal variable was set
✗ **Bad**: Test order of private method calls

### Test Public APIs

- Test through public interfaces
- Avoid testing private methods directly
- Test return values, outputs, exceptions
- Test side effects (database, files, logs)

### Test Edge Cases

Consider each of these, and keep the ones that pass Test Selection — as rows of the rule's table, not as separate tests:
- **Boundary conditions**: Min/max values, empty collections
- **Error cases**: Invalid input, null values, missing data
- **Edge cases**: Special characters, extreme values
- **Happy path**: Normal, expected usage

## Test Quality Criteria

These criteria ensure reliable, maintainable tests.

### Literal Expected Values

- Use hardcoded literal values in assertions
- Calculate expected values independently from the implementation
- If the implementation has a bug, the test catches it through independent verification
- If expected value equals mock return value unchanged, the test verifies nothing (no transformation occurred)

### Result-Based Verification

- Verify final results and observable outcomes
- Assert on return values, output data, or system state changes
- For mock verification, check that correct arguments were passed

### Meaningful Assertions

- Every test must include at least one assertion
- Assertions must validate observable behavior
- A test without assertions always passes and provides no value

### Appropriate Mock Scope

- Mock direct external I/O dependencies: databases, HTTP clients, file systems
- Use real implementations for internal utilities and business logic
- Over-mocking reduces test value by verifying wiring instead of behavior

### Boundary Value Testing

Test at boundaries of valid input ranges:
- Minimum valid value
- Maximum valid value
- Just below minimum (invalid)
- Just above maximum (invalid)
- Empty input (where applicable)

### Test Independence Verification

Each test must:
- Create its own test data
- Not depend on execution order
- Clean up its own state
- Pass when run in isolation

## Verification Requirements

### Before Commit

- ✓ All tests pass
- ✓ No tests skipped or commented
- ✓ No debug code left in tests
- ✓ Every test passes Test Selection: a unique defect, the lowest sufficient level, no tautology, no library behaviour
- ✓ Tests run in reasonable time

### Zero Tolerance Policy

- **Zero failing tests**: Fix immediately
- **Zero skipped tests**: Fix, or escalate the removal
- **Zero flaky tests**: Make deterministic
- **Zero slow tests**: Optimize or split

## Test Organization

### File Structure

- **Mirror production structure**: Tests follow code organization
- **Clear naming conventions**: Follow project's test file patterns
  - Examples: `UserService.test.*`, `user_service_test.*`, `test_user_service.*`, `UserServiceTests.*`
- **Logical grouping**: Group related tests together
- **Separate test types**: Unit, integration, e2e in separate directories

### Test Suite Organization

```
tests/
├── unit/           # Fast, isolated unit tests
├── integration/    # Integration tests
├── e2e/            # End-to-end tests
├── fixtures/       # Test data and fixtures
└── helpers/        # Shared test utilities
```

## Performance Considerations

### Test Speed

- **Unit tests**: < 100ms each
- **Integration tests**: < 1s each
- **Full suite**: < 10 minutes

### Optimization Strategies

- Run tests in parallel when possible
- Mock expensive operations
- Split slow test suites
- Profile and optimize slow tests

## Common Anti-Patterns to Avoid

### Test Smells

- ✗ Tests that test nothing (always pass)
- ✗ Tests that depend on execution order
- ✗ Tests that depend on external state
- ✗ Tests with complex logic (tests shouldn't need tests)
- ✗ Testing implementation details
- ✗ Excessive mocking (mocking everything)
- ✗ Test code duplication

### Flaky Tests

Eliminate tests that fail intermittently:
- Remove timing dependencies
- Avoid random data in tests
- Ensure proper cleanup
- Fix race conditions
- Make tests deterministic

## Regression Testing

### Prevent Regressions

- Every bug fix gets a test that reproduces it — a new row in an existing test when one already exercises that path
- Don't delete tests without good reason

### Legacy Code

- Add characterization tests before refactoring
- Test existing behavior first
- Add tests where a refactor needs a safety net, not to raise a percentage
- Refactor with confidence

## Testing Best Practices by Language Paradigm

### Type System Utilization

**For languages with static type systems:**
- Leverage compile-time verification for correctness
- Focus tests on business logic and runtime behavior
- Use language's type system to prevent invalid states

**For languages with dynamic typing:**
- Test runtime validation of the project's own contracts once per rule, as a table
- Explicitly test data contract validation
- Consider property-based testing for broader coverage

### Programming Paradigm Considerations

**Functional approach:**
- Test pure functions thoroughly (deterministic, no side effects)
- Test side effects at system boundaries
- Leverage property-based testing for invariants

**Object-oriented approach:**
- Test behavior through public interfaces
- Mock dependencies via abstraction layers
- Test polymorphic behavior carefully

**Common principle:** Adapt testing strategy to leverage language strengths while each test still passes Test Selection

## Documentation and Communication

### Tests as Documentation

- Tests document expected behavior
- Use clear, descriptive test names
- Include examples of usage
- Show edge cases and error handling

### Test Failure Messages

- Provide clear, actionable error messages
- Include actual vs expected values
- Add context about what was being tested
- Make debugging easier

## Expert References (Reasoning Calibration)

When facing testing strategy decisions, calibrate your reasoning against these established principles:

| Expert | Key Principle | Apply When |
|--------|--------------|------------|
| Kent C. Dodds | "The more your tests resemble the way your software is used, the more confidence they can give you." | Choosing between behavior-based vs. implementation-based tests |
| Michael Feathers | "Legacy code is simply code without tests." | Prioritizing test coverage for untested code areas |
| Gerard Meszaros | "Each test should have a single reason to fail." | Determining test granularity and assertion scope |
| James Shore | "TDD isn't about testing. TDD is about design." | Using TDD as a design feedback mechanism, not just validation |