test-evaluation
/test-evaluation
Ocenia istniejące zestawy testów i debuguje wolne lub niestabilne testy, sprawdzając jakość pokrycia i kod testów.
- Autor
- jcdendrite ↗
- Repozytorium
- claude-config ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 27.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 2
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 27.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Ocenia istniejące zestawy testów i debuguje wolne lub niestabilne testy, sprawdzając jakość pokrycia i kod testów.
Oryginalny opis (EN)
Evaluate and debug existing test suites. TRIGGER when: reviewing or evaluating an existing test suite, debugging slow or flaky tests, assessing test coverage quality, or reviewing test code. DO NOT TRIGGER when: writing new tests (use test-conventions), or the task is purely mechanical (running tests, fixing a single assertion, updating a snapshot).
Jak działa
- 1. Signs of an inverted pyramid
- 2. When to consider moving a test down the pyramid
- 3. Flaky test diagnosis
- 4. Anti-patterns
Podgląd SKILL.md
---
name: test-evaluation
description: >
Evaluate and debug existing test suites.
TRIGGER when: reviewing or evaluating an existing test suite, debugging slow
or flaky tests, assessing test coverage quality, or reviewing test code.
DO NOT TRIGGER when: writing new tests (use test-conventions), or the task is
purely mechanical (running tests, fixing a single assertion, updating a snapshot).
user-invocable: false
---
# Test Evaluation
## 1. Signs of an inverted pyramid
- Most tests call real services (DB, APIs) to test internal branching logic
- Tests are slow (>1s each) because of network round-trips
- Individual unit tests taking >50-100ms, indicating a hidden real dependency or overly complex setup
- External API rate limits constrain how often you can run the test suite
- Tests are flaky because of network timeouts or service availability
- You can't run the test suite offline or in CI without credentials
- Adding a new code branch requires setting up complex test state in a real database
## 2. When to consider moving a test down the pyramid
These are heuristics, not rules — evaluate each case against the specific codebase's constraints.
**Integration → unit when:**
- The test sets up complex DB state to exercise one `if` branch
- The test hits an external API just to test error handling
- The test is slow (>500ms) and tests pure logic, not wiring
- You're hitting rate limits on external APIs
**E2E → integration when:**
- The test calls a real third-party API to verify your code handles errors
- The test is flaky because of network conditions
- The test creates real resources (emails, contacts) as a side effect
Smoke tests verify deployment health, not behavior, so they never move down the pyramid — a failing one points to deployment/infra, not the test layer.
## 3. Flaky test diagnosis
When a test is intermittently failing:
1. **Root-cause first** — diagnose the specific cause:
- Test isolation failure (shared state, execution order dependency)
- External service timing (network, container startup, API availability)
- Time zone sensitivity (test passes in one TZ, fails in another — common with date boundary logic)
- Locale-dependent formatting (number/date formats vary by system locale)
- Floating point comparison (use approximate equality, not exact)
- Async timing (asserting on async state without proper synchronization)
- Non-deterministic iteration order (hash maps/sets with no guaranteed order)
- Port or resource conflicts in parallel execution
2. **Fix the layer** — if the test is flaky because it hits a real service to test logic, push it down the pyramid (see section 2)
3. **Quarantine in-test retries** — retry loops inside test code mask the root cause. If a fix isn't immediate, quarantine the test (skip + tracking issue) instead of letting retries erode trust in the suite; CI-level rerun-once policies are a separate, fine practice but track their frequency.
4. **Never delete without replacement** — a flaky test covering real risk (e.g. an auth boundary) must be replaced with a reliable test at the right layer, not simply deleted.
## 4. Anti-patterns
| Anti-pattern | Why it's wrong | Fix |
|---|---|---|
| All tests are integration tests | Slow, flaky, can't test every branch | Extract logic, add unit tests with stubs |
| Testing the test double | Assertions check values set up in the stub, not derived by code under test | Assert on values the code computed or transformed |
| No integration tests at all | Auth bugs, response shape regressions | Keep a few focused integration tests per endpoint |
| Smoke tests on every commit | Slow CI, rate limit exhaustion, flaky | Run smoke tests on deploy only |
| Testing implementation details | Tests break on refactor with no behavior change | Test inputs and outputs, not internal mechanics |
| Tautological assertions | `assert("error" in body or "data" in body)` passes on any response | Assert specific values |
| Duplicating production code in tests | Test passes with stale copy, drift | Import or call via integration |
| Reading source files to test behavior | Tests source text, not runtime behavior | Call function, assert output |
| Regex-parsing specific field values from runtime output in an assertion (log lines, JSON, XML) | Re-implements the production parser: passes when parsing is broken, breaks on format changes (validity-only assertions on the envelope are exempt). | Parse with the library; assert on the resulting object; or call the production parse/validate function |
| Unconditional global state deletion | Breaks subsequent tests that need the value | Save and restore in guaranteed cleanup |
| In-test retry loops for flaky tests | Masks root cause, inflates suite duration | Root-cause and fix or quarantine |
| Test interdependence | Test B depends on state from Test A; reorder breaks both | Each test sets up and tears down its own state |
| Assertion-free tests | Test runs code but never asserts; passes as long as nothing throws | Every test must assert on a specific expected outcome |
| Sleep-based synchronization | `sleep(2)` to wait for async work; slow and still flaky | Use polling with timeout, await, or synchronization primitives |
| Hardcoded colliding test data | Every test uses `id=1` or `email=test@example.com`; parallel runs collide | Generate unique identifiers per test |
| Over-mocking | So many mocks that the test encodes the implementation, not behavior; brittle to refactors | Mock only direct dependencies; let integration tests cover wiring |
| Mocking third-party internals | Mocking a library's internal API rather than its public interface | Use the library's test utilities or mock at your own abstraction boundary |
Pliki w paczce (2)
- SKILL.md 6 KB
- evals/trigger-cases.json 1 KB
Wersje
W bazie nie ma innych skilli o nazwie test-evaluation.
Migawka bazy z 13.11.2026 · wersja paczki 3d59fe35f814 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100test-conventions
Opisuje konwencje pisania testów w dowolnym kodzie: mocki, helpery, fixtury i strategię testów dla nowych funkcji. - 100test-engineer
Działa jako inżynier QA: projektuje strategię i zestawy testów, pisze testy dla istniejącego kodu oraz ocenia jakość testów i pokrycie. - 100am-agent-test-engineer
Działa jako inżynier QA: projektuje strategię testów i zestawy testów, pisze testy dla istniejącego kodu oraz ocenia pokrycie i jakość testów. - 100testing-audit
Audytuje zestaw testów: pokrycie, jakość, testy niestabilne i zgodność ze strategią przed wydaniem albo gdy testy słabną. - 90proof-audit
Audytuje kondycję zestawu testów: znajduje testy niestabilne i wolne, luki w pokryciu oraz antywzorce testowania. - 90proof-audit
Audytuje kondycję zestawu testów: znajduje testy niestabilne i wolne, luki w pokryciu oraz antywzorce testowania.