Przejdź do treści
Biblioteka skilli
Programowanie · Testowanie

fix-flaky

/fix-flaky

Triażuje i naprawia niestabilne testy w projekcie Semaphore: wskazuje najgorsze przypadki, diagnozuje przyczynę, pisze poprawkę lub uzasadnioną kwarantannę i weryfikuje ponownym uruchomieniem.

Repozytorium
sem-ai ↗
Licencja
Apache-2.0
Ostatnia zmiana repozytorium
06.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
1
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 06.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

Co robi i kiedy

Triażuje i naprawia niestabilne testy w projekcie Semaphore: wskazuje najgorsze przypadki, diagnozuje przyczynę, pisze poprawkę lub uzasadnioną kwarantannę i weryfikuje ponownym uruchomieniem.

Wyzwalacze z opisu

why is CI randomly red
Oryginalny opis (EN)

Triage and FIX flaky tests on a Semaphore project end-to-end — find the worst offenders, pull history, locate the test in the code, diagnose the root cause, write a fix or justified quarantine, and verify by re-running. Use whenever the user wants to fix/investigate flaky tests, de-flake CI, reduce intermittent failures, asks "why is CI randomly red", wants to quarantine a flake, or after `sem-ai flaky list` surfaces offenders. Goes BEYOND detection (test-intelligence) to root-cause + code change + verification.

Jak działa

  1. The loop
  2. Pull the actual failure (flaky failure)
  3. Composes with
  4. Gotchas
Sekcje z nagłówków SKILL.md, w oryginale · 121 linii, 1 101 słów
Podgląd SKILL.md
---
name: fix-flaky
description: Triage and FIX flaky tests on a Semaphore project end-to-end — find the worst offenders, pull history, locate the test in the code, diagnose the root cause, write a fix or justified quarantine, and verify by re-running. Use whenever the user wants to fix/investigate flaky tests, de-flake CI, reduce intermittent failures, asks "why is CI randomly red", wants to quarantine a flake, or after `sem-ai flaky list` surfaces offenders. Goes BEYOND detection (test-intelligence) to root-cause + code change + verification.
user-invocable: true
---

# Fix Flaky Tests

Detection tells you a test is flaky; this skill closes the loop to a fix. The
hard part is never "is it flaky" — it's tying the failure to the actual code.

## The loop

### 1. Discover + rank (ranked, denoised, compact — no jq needed)
```bash
sem-ai flaky list --project <name> \
  --sort-field total_disruptions_count --sort-dir desc \
  --disruptions ">1"
```
`--disruptions ">1"` drops one-off noise (single-failure `pass_rate:50` rows);
the sort ranks by recurrence. Output omits the per-test `disruption_history`
histogram by default (rarely needed — `--full` restores it; no diagnosis path
below requires it). Pick a test that recurs across many commits and whose
`test_file` you can read.

### 2. Get the per-context history
```bash
sem-ai flaky show <test_id> --project <name>     # POSITIONAL test_id (NOT --file). Returns per-context pass_rate, p95, disruptions_count.
```
For the real failure, run `flaky failure <test_id>` (see *Pull the actual
failure*) — don't hand-chase run ids. (`latest_disruption_run_id` is on the
`flaky list` row, not `show`.) Contexts whose stats are all-null simply have no
disruptions recorded on that branch; ignore them and read the non-null ones.

### 3. Locate in the code (paths are app-relative)
`flaky failure` (step 2) already hands you the failing `file`+`line` — no need to
derive them from the test name. But the reported path is **app-relative**, not
repo-root: in a monorepo `test/foo/bar_test.exs` lives under an app/service dir
(e.g. `apps/api/test/…`, `apps/web/test/…`, `services/worker/test/…`). Resolve
the on-disk path:
```bash
git -C <repo> ls-files | grep -F "$(echo <test_file> | sed 's/:[0-9]*$//')"
```
If that returns matches in several apps, disambiguate with the `test_group`/suite
from `flaky show` (e.g. a group like `MyApp.Web.WidgetTest` → the `web` app).
Read the test AND the code it exercises — flakes live in the seam.

### 4. Classify from the real failure — the table names the class, it does NOT hand you the fix
**Pull the real failure first** (see *Pull the actual failure* below): the
`left:`/`right:` + stacktrace are the diagnosis. The table only names the
**class** of nondeterminism and the **direction** a fix usually takes, so you
know what you're chasing — the actual fix comes from the test in front of you,
never from a cell.

| signal | likely class | fix direction |
|---|---|---|
| asserts strict `<`/`>` (or `compare == :lt`/`:gt`) on two timestamps taken close together; passes most runs | clock-tie / nondeterministic time | allow the tie (inclusive bound), or freeze/inject time so the values are deterministic |
| element acted on after an async re-render; `StaleReferenceError` | stale-element after async render | retry the lookup+action on stale — a presence-assert does NOT fix it (the node goes stale *after* lookup) |
| in-test wait/sleep budget shorter than the work's failure tail | timeout too short for async work | raise the wait budget to match a non-flaky sibling; make the predicate nil-safe |
| asserts order of a query/collection with no explicit ordering | nondeterministic ordering | add deterministic ordering at the source |
| passes alone but fails after other tests (leftover rows/keys/processes) | shared/global state | isolate setup/teardown; unique fixtures |
| asserts a count of live processes/children that's off by one+ | leaked process from a prior test (shared named supervisor / registered process) | terminate/drain the named process in `setup`/`on_exit`, not just the DB |
| calls a real external service | external dependency | stub/mock, or mark + isolate |

p95 (from `flaky show`) is the heuristic **only for the timeout row** — for
clock-tie/stale-element/ordering it's a red herring. For the timeout class,
compare the wait budget to the failure **tail**, not p95 (a ~95%-pass flake's
p95 sits under the budget); the real ceiling is **wait-helper fan-out ×
per-wait budget**. Two high-value moves: `grep` the repo for other callers of
that wait helper and **diff their budgets** (a non-flaky sibling is proof +
fix template); and before writing any retry/wait machinery, `grep` for an
existing helper (`retry_on_stale`, `assert_eventually`, a shared `Wait` util)
and **reuse it**.

### 5. Fix or quarantine
Smallest change that removes the nondeterminism. A justified quarantine
(skip/tag + linked ticket) is acceptable if a true fix is out of scope — say why.
Match repo conventions; no comments unless the repo uses them.

### 6. Verify by RE-RUNNING (one green proves nothing)
Use the **testbox** skill to run the single test many times against your change,
or a targeted rerun, and check the pass rate moved.
**Can't verify** (no local toolchain, can't push, or testbox unavailable — e.g.
an org that blocks debug sessions)? Say so and mark the fix **provisional** —
that's an acceptable outcome, not a failure.

## Pull the actual failure (`flaky failure`)
```bash
sem-ai flaky failure <test_id> --project <name>
```
One call resolves the latest disruption's job, fetches its log, and returns the
failing test's **real assertion** as JSON: `{test_name, run_id, framework,
summary, matched, failures:[{file, line, message}]}`. `message` is the actual
`code:`/`left:`/`right:`/`stacktrace` — not a guess. It works for ExUnit (which
`test report` can't parse), and filters to your test. Pin a specific occurrence
with `--run-id <job_id>`.
- `matched:false` → the failure block didn't match your test name (the job ran it
  but it may have passed that run, or the name differs); it returns all failures
  in that job — eyeball them.
- `log_unavailable` → the disruption's job log aged out (retention); diagnose from
  source + the playbook above.
- Timeout-class flakes show a raised exception (e.g. `Timeout: ...`), not an
  assertion diff — `message` carries the exception, not a failing `assert`.
- For ExUnit, `message` often includes the full process **Logger output** after the
  assertion — read all of it; the event ordering there is frequently the decisive
  evidence (e.g. an async consumer firing *after* the step you tested), not just `left/right`.

**Manual fallback** (older binaries without `flaky failure`): a `run_id` from
`flaky disruptions <test_id>` (`.run_id`; skip null-padding rows) is a **job id** →
`job log <run_id>` (takes NO `--project`) → grep the failure block:
```bash
sem-ai job log <run_id> | jq -r '.[].output // empty' \
  | grep -nE '[0-9]+\) (test|doctest)|match \(=\) failed|left:|right:|stacktrace:'
```

## Composes with
- **test-intelligence** — `sem-ai test report|summary` failure detail (when retrievable).
- **debug-pipeline** — `sem-ai diagnose <run-id>` for the broader failed run.
- **testbox** — step 6 verification.

## Gotchas
- `flaky show`/`disruptions`/`failure` take the `test_id` **positionally** (the `args` field via MCP); `--file` returns empty silently.
- `flaky disruptions` can return null-timestamp padding rows — ignore them.
- `flaky show` per-context `pass_rate`/`disruptions_count` can be `null` even when disruptions exist — trust the disruption rows.
- Don't `sem-ai context switch` mid-task if one is set; pass `--project`.

Wersje

W bazie nie ma innych skilli o nazwie fix-flaky.

Migawka bazy z 13.11.2026 · wersja paczki b8b1e1c3a7bf (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100qa-flaky-test-hunter
    Diagnozuje niestabilne testy: klasyfikuje przyczynę jako czasową, kolejnościową, zasobową lub zewnętrzną, izoluje je i opisuje plan naprawy.
  • 100flaky-test-management
    Pomaga wykrywać, triażować, szukać przyczyn, poddawać kwarantannie i eliminować niestabilne testy, w tym strategię ponowień i testy przechodzące lokalnie, ale padające w CI.
  • 95wicked-garden-qe-flaky-test-hunter
    Wykrywa niestabilne testy, analizuje ich historię i klasyfikuje przyczyny, nie traktując ponownych prób jako poprawki.
  • 100v1-e2e-testing
    Pisze i naprawia testy end-to-end w przeglądarce, zwłaszcza w Playwright, i diagnozuje niestabilne testy.
  • 90absolute-deflake
    Naprawia niestabilne testy: wykrywa je przez powtarzanie i przetasowanie uruchomień, diagnozuje przyczynę i weryfikuje poprawkę w wielu losowych przebiegach.
  • 100flaky-triage
    Znajduje niestabilne testy w ostatnich przebiegach i poddaje je kwarantannie, by przestały produkować fałszywe zgłoszenia błędów.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.