Skip to content
Skills Library
Data & analytics · Testing

pair-testing

/pair-testing

Use when building counterfactual pair suites (two inputs identical except for one attribute) to measure fairness or robustness, including the flagship French CV screening fairness study.

Licence
Apache-2.0
Last repository change
01.11.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 01.11.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Use when building counterfactual pair suites (two inputs identical except for one attribute) to measure fairness or robustness, including the flagship French CV screening fairness study.

Original author description

Use when building counterfactual pair suites (two inputs identical except for one attribute) to measure fairness or robustness, including the flagship French CV screening fairness study.

How it works

  1. Building pairs
  2. Sources for variable values
  3. Metrics
  4. Interpretation and wording
Original SKILL.md headings · 28 lines, 298 words
SKILL.md preview
---
name: pair-testing
description: Use when building counterfactual pair suites (two inputs identical except for one attribute) to measure fairness or robustness, including the flagship French CV screening fairness study.
---

# Counterfactual pair testing

The method: build pairs of inputs identical except for one controlled variable, run both through the target, and measure how much the output changes. No ground truth is needed: a fair, robust system gives the same answer to both.

## Building pairs

1. Start from a base document (e.g. a synthetic CV) written once, realistic, with no attribute that hints at the tested variable.
2. Derive variant A and variant B by changing **only** the tested variable: name, gender markers (including grammatical agreement in French), age or birth year, address, career gap. Everything else must be byte-identical; write a test that diffs A and B and fails if more than the intended fields differ.
3. Randomise which variant is presented first when the target sees both, and run each variant in a separate call by default.
4. Balance: each value of the variable appears equally often as A and as B, across jobs and seniority levels.

## Sources for variable values

Use documented public sources (e.g. official first-name statistics) and record the source, date and selection rule in the suite README. Never invent lists ad hoc and never use real people's identities.

## Metrics

- `pair_gap`: mean absolute difference of the numeric score.
- `pair_flip`: share of pairs where the decision changes.
- Report both per variable value (`group_by`), with confidence intervals, and the direction of the gap.

## Interpretation and wording

- A gap is a measured difference on these synthetic inputs, not proof of how a deployed product behaves. Say so in the README and in any published study.
- Publish method, cases and code so anyone can reproduce the result.
- Before publishing results naming a vendor, have the `suite-reviewer` agent check methodology, and give the vendor the chance to reproduce.

Versions

The database has no other skills named pair-testing.

Database snapshot: 13.11.2026 · package version c70f378961d7 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

No similar skills in the database.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.