Skip to content
Skills Library
DevOps & cloud · Planning

repomatic-test-matrix

/repomatic-test-matrix

Choose what a repository's CI test matrix covers. Decide which Python versions, operating systems and runner images earn a cell. Mark which axes stay unstable probes, and pick where a one-off job runs. Use when you add or drop a Python version or OS, pin a dependency floor, or place a new job.

Repository
repomatic ↗
Licence
LGPL
Last repository change
27.10.2026

Assessment and security

95/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content30 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

LGPL, otwarta.

What it does and when to use it

Choose what a repository's CI test matrix covers. Decide which Python versions, operating systems and runner images earn a cell. Mark which axes stay unstable probes, and pick where a one-off job runs. Use when you add or drop a Python version or OS, pin a dependency floor, or place a new job.

Arguments

/repomatic-test-matrix [review|add <axis>|drop <axis>]
Original author description

Choose what a repository's CI test matrix covers. Decide which Python versions, operating systems and runner images earn a cell. Mark which axes stay unstable probes, and pick where a one-off job runs. Use when you add or drop a Python version or OS, pin a dependency floor, or place a new job.

How it works

  1. Context
  2. Instructions
Original SKILL.md headings · 115 lines, 1,681 words
SKILL.md preview
---
name: repomatic-test-matrix
description: Choose what a repository's CI test matrix covers. Decide which Python versions, operating systems and runner images earn a cell. Mark which axes stay unstable probes, and pick where a one-off job runs. Use when you add or drop a Python version or OS, pin a dependency floor, or place a new job.
compatibility: 'Designed for Claude Code. Recommended model: Sonnet.'
allowed-tools: Bash Read Grep Glob Edit
argument-hint: '[review|add <axis>|drop <axis>]'
---

## Context

!`grep -A20 '^\[tool.repomatic.test-matrix' pyproject.toml 2>/dev/null || echo "NO_TEST_MATRIX_CONFIG"`

!`grep -E '^\s*requires-python' pyproject.toml 2>/dev/null || echo "NO_REQUIRES_PYTHON"`

## Instructions

`repomatic show-metadata` builds the full and PR test matrices from `[tool.repomatic.test-matrix.*]`. Read the effective matrices before proposing anything:

```shell-session
$ repomatic show-metadata --output-format json test_matrix test_matrix_pr
```

A matrix cell costs a runner on every push, so each one has to earn its place. These are the selection conventions.

### Cover the shipped config broadly; probe unreleased axes narrowly; smoke-test released flavors

Released dependencies on stable Python get a broad spread, but *broad* means broad on the axis the product actually varies along, not on both at once. A tool whose behaviour changes per platform (it drives different system binaries, resolves different paths, ships a different feature set) earns a cell for every OS and architecture, because that spread **is** the product. Interpreter compatibility is OS-independent by construction, so the floor, the prerelease and the free-threaded build each need one runner rather than one per OS. Crossing the two axes multiplies cells to buy their *interaction*, which is worth paying for only where the failure history shows an interaction exists: measure it (see below) instead of assuming it, and put the floor on whichever OS its users actually run, which for a Python tool is usually Linux, where distribution packagers build against whatever interpreter their channel ships.

Unreleased dependency branches and prerelease Python run on one runner as `continue-on-error` probes (`test-matrix.unstable`), never across platforms: a probe that fails is information, and paying for it six times over buys none.

A *released* free-threaded build (`3.15t`) is a different case and runs **stable** on a single runner, as one `include` row pinned to that runner and left out of `unstable`. It is shipped software, so a failure there is a real failure.

### Ask what a cell has caught, not what it might

A cell justifies itself by having failed while its siblings passed. Anything less is a hypothesis, and the repository already holds the evidence to test it: walk recent runs of the workflow and, for every failing cell, check whether the *same OS* passed at its other Python version in that same run. A cell that never fails alone has never repaid its cost.

```shell-session
$ gh api 'repos/{owner}/{repo}/actions/workflows/tests.yaml/runs?branch=main&created=>={date}&per_page=40' \
    --jq '.workflow_runs[].id'
$ gh run view {run-id} --json jobs \
    --jq '.jobs[] | select(.name | test("py")) | "\(.name): \(.conclusion)"'
```

Keep the `created` filter, and do not use `gh run list` instead. Both an unfiltered page and that command can show runs weeks old as the newest ones.

Count cancelled runs too. A busy default branch cancels most of its runs through `cancel-in-progress`, and the cells that had already reported a verdict inside them are where most of the failure history lives; filtering to conclusive runs alone can shrink a real sample to nothing.

Two readings make the decision. A failure hitting **every** cell of an OS is an OS-level or universal bug, which one cell per OS would have caught. A failure hitting **one** cell while its twin passed is the only kind a per-OS version pair can catch, and if that count is zero across a real sample, the pairs are redundant. Expect the second reading to also surface environment artifacts rather than code bugs (one runner of a label carrying a different tool layout than another), and do not count those as a cell earning its keep.

State the sample size and both counts when proposing the change, and record in the config comment what would justify restoring what you cut, so the next reader inherits the measurement rather than the conclusion alone.

### Pin the dependency floor, and any release a workaround targets

Add the floor of a supported range as an explicit matrix value: the floor is what an install actually resolves for someone on an old environment, and no other cell runs it, unless the matrix holds the `lowest-direct` cell below.

Add any mid-range release a shim works around, too. That version is the one that catches the shim regressing, and it is invisible to a matrix that only spans the endpoints.

### Run every declared floor in one cell

One pinned value verifies one dependency. To run every declared floor at once, add one `full-include` row that carries a `resolution` key, on the floor Python of the fastest runner:

```toml
[tool.repomatic]
test-matrix.full-include = [
  { "os" = "ubuntu-26.04-arm", "python-version" = "3.11", "resolution" = "lowest-direct" },
]
```

The `tests` job of the bundled `tests.yaml` resolves the project again in a cell that carries that key: each direct dependency takes the lowest version its specifier allows. The cell fails when the code calls an API newer than a declared floor.

A `tests.yaml` whose jobs predate that step ignores the key, and the cell then repeats its twin. Copy the step that reads `matrix.resolution` from the upstream workflow first, at the tag the repository pins: `gh api 'repos/kdeldycke/repomatic/contents/.github/workflows/tests.yaml?ref=vX.Y.Z' -H 'Accept: application/vnd.github.raw'`. Then put `--frozen` on every `uv run` after it. A plain `uv run` finds the rewritten lockfile out of date and resolves it back to the newest releases.

Replay the cell before you add the row, in a scratch copy of the repository, since the resolution rewrites `uv.lock`. On the floor Python, run `uv sync --resolution lowest-direct --all-extras --group test`, then the suite with `uv run --frozen`. Run it several times and judge each run by its exit code, never by its summary line. Test plugins have floors too: one below its floor can abort the collection under xdist, or end a passing run on an `INTERNALERROR`.

A test that reads the matrix's `python-version` axis must also read its `include` rows. A `full-include` row flattens the full matrix into a job list with no axis.

The cell cannot see a dependency that another requirement holds above its floor, or a library that the tests replace with a mock. Check those two by hand: the `/repomatic-deps` skill, when present, holds the procedure.

### Select runners by measured speed and workload, not architecture

Measure, do not reason from the chip:

```shell-session
$ repomatic job-timings --workflow tests.yaml --limit 5
```

That reports median whole-job wall-clock per runner image from recent successful runs. Read it before proposing any runner change: the parallel `pytest --numprocesses=logical` suite favours `ubuntu-26.04-arm`, which is why that is the test PR Linux slot, but the ratio is workload-dependent and yours may differ.

Where one fast runner suffices, `ubuntu-26.04-arm` is the default: fastest and cheapest tier, against hosted macOS billing roughly ten times Linux. Reserve `macos-26` and Windows for the OS coverage only they add, and drop the slower twin of an OS pair with `test-matrix.remove.os`.

### Time whole jobs, not the tool pass

A measurement that times only tool execution misses checkout and install, which is where most of the difference between runner images lives. The lean `ubuntu-slim` image survived for a long time on exactly that mistake: measured end to end, the full image ran 27-32% *faster*.

`job-timings` reads the jobs API's start and end timestamps, so what it reports is whole-job by construction and this mistake is not expressible through it.

### Watch what is arriving and retiring

You are not the first to know an image is changing. `repomatic sync-runner-images` runs weekly from the `autofix.yaml` workflow, looks up every label this repository runs in GitHub's *Available Images* table, and proposes the mechanical half as a pull request. A retiring image has its literal `runs-on:` values moved onto a successor. A strictly newer *version* of an image in use gets the same rewrite, and also joins the full matrix as a `continue-on-error` probe unless the fleet already runs it. It opens a pull request only when something here is exposed, so its existence is the signal.

Deciding whether to merge is this skill's job, and the CI run that pull request triggers is the evidence for it. Closing the pull request alone brings the proposal back on the next run; declining one for good means naming the label in `[tool.repomatic.sync-runner-images] ignore`.

That table is the only source read, and it badges an image `deprecated` when deprecation *begins* rather than when it is announced, so a retirement surfaces here months after an announcement feed would have shown it. The runway is still ample, since the badge lands well before the image stops working. What the table cannot show at all is a change to the *contents* of an image already in use, like a default toolchain moving: the suite is what catches those.

### An image is stable once validated here, not once GitHub relabels it

GitHub's *preview* label chiefly gates `-latest` alias eligibility, and no workflow here uses a floating alias, so it says nothing about whether the image runs the suite green.

Never introduce a `-latest` alias to sidestep the question: GitHub repoints those with no commit to review, and `lint-repo` warns about them.

### Every job runs on a test axis

The images a job may run on are exactly those the test matrices use, and `lint-repo` warns about any other `runs-on:`. Read the effective set from `repomatic show-metadata` rather than from the package source, which a repository consuming repomatic does not have checked out. That keeps "where is the suite exercised" and "what may a job run on" a single question, because each extra image is one more to track, pin and migrate.

A job that genuinely needs something else widens the axes rather than naming a one-off image. This covers the Linux Nuitka hosts (a published binary is built on the image the suite is validated against, and its toolchain comes from a digest-pinned manylinux container regardless) and the light mechanical jobs.

### Reporting

Propose changes as a `[tool.repomatic.test-matrix.*]` diff, and for each added or removed cell say what it buys or costs: a version nothing else exercises, an OS-specific failure mode, a runner-minute saving. A cell nobody can justify in one sentence is a cell to drop.

Verify with `repomatic show-metadata` after editing, since the config is an input to a computed matrix rather than the matrix itself.

Versions

RepositoryScoreLicence
This page · kdeldycke/repomatic95LGPL
kdeldycke/dotfiles95BSD-2-Clause

Database snapshot: 13.11.2026 · package version 75403c745745 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100matrix-optimizer
    Optimize GitHub Actions matrix strategies for testing across multiple versions, platforms, and configurations. Use when configuring matrix builds, testing multiple versions, cross-platform testing, or optimizing CI resource usage. Trigger words include "matrix strategy", "test…
  • 100repomatic-ship
    Prepare a release. Reconcile the changelog, code and docs to the net release state. Then commit, push, and babysit CI until the release PR is built and `main` is green. Stop before the merge. Ask for review in normal use; run fully autonomous under `--dangerously-skip-permissions`.
  • 100repomatic-ship
    Prepare a release. Reconcile the changelog, code and docs to the net release state. Then commit, push, and babysit CI until the release PR is built and `main` is green. Stop before the merge. Ask for review in normal use; run fully autonomous under `--dangerously-skip-permissions`.
  • 100upgradelab
    Run the upgrade test harness in tools/upgradelab: build it, produce a snapshot of main, run a matrix cell (one old release upgraded to head under traffic), pick a deployment, tier and shape, read the checks (I*, N*, O1, B1, H*, E*, D*, A*) and the report, and keep the machine…
  • 100add-ci-platform
    Add a new OS platform to RediSearch CI. Use when adding a new distro version, OS, or container target to the build/test matrix.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.