Pairwise (Fairness) Metrics
Pairwise Metrics use an LLM-as-Judge to compare two peer responses symmetrically — measuring how much their treatment diverges, with no notion of a "baseline" or "candidate" and no direction. They power counterfactual and fairness testing: the two responses were produced for inputs that are identical except for one attribute of the subject, and a fair system should treat both equivalently.
Instead of a signed win/loss, a pairwise metric outputs a non-negative magnitude:
| Output Score | Meaning |
|---|---|
0 | Equivalent treatment — the ideal outcome. |
| higher | Increasing divergence in how the subject was treated. |
Aggregated across the dataset, these become equivalence / divergence rates and a
mean divergence, which you gate in a target's pairwise block with
min_equivalence_rate, max_divergence_rate, or max_mean_divergence.
Pairwise is symmetric — a divergence favouring either side scores the same. Its sibling, Regression Metrics, is directional — it asks which of a baseline and candidate is better. They are separate axes with separate metric libraries and cannot be mixed in the same block.
How the pairing is formed
A pairwise block sources its two responses from within a single run, using two
strategies:
group— each variant is compared against its group's reference row.contrastive— the two opposing poles of each group are compared against each other.
Grouping is driven by a group_by column (default group_id) and a role column
(default role). See the manifest configuration for the
block syntax.
You rarely build this grouped data by hand — Counterfactual Adversarial
Generation produces exactly the group_id / role
/ dimension columns this axis consumes.
Available Pairwise Metrics
All pairwise metrics share the same variables: {{input}}, {{output_a}}, and
{{output_b}}.
pairwise-bias
Measures how much the treatment of the subject diverges between two peer responses that should be equivalent. Direction-agnostic; the ideal outcome is that both are treated the same.
pairwise-demographic-bias
A dimension-oriented fairness check: measures how much treatment diverges specifically because of the subject attribute under test (the attribute is supplied explicitly).
pairwise-refusal-disparity
A behavioral fairness check: measures divergence in willingness to answer between two peer responses. The ideal outcome is that both take the same action (both answer, or both refuse).