Adversarial Dataset Generation

A golden dataset tells you whether your agent works on the inputs you expected. It says nothing about the inputs you didn't. Adversarial Generation takes an existing dataset and synthesizes hostile or counterfactual variants of its rows, so you can certify robustness and fairness — not just happy-path accuracy.

CertOps offers two flavors, each feeding a different grading axis:

Flavor	Question it answers	Feeds
Mutation (pointwise)	Does the agent hold up under adversarial or out-of-scope inputs?	Pointwise (Evaluation) Metrics
Counterfactual (pairwise)	Does the agent treat otherwise-identical users equivalently?	Pairwise (Fairness) Metrics

Both take a source dataset and produce a new derived dataset you reference from a target's manifest exactly like any other — nothing special is required at run time.

---

Mutation (Robustness)

Mutation generation rewrites each seed row into an adversarial variant while keeping the intent answerable. The result is a dataset that probes how gracefully your agent refuses, deflects, or stays in-scope.

You pick which mutation types to apply from a 12-type catalog (GET /v1/adversarial/mutation-jobs/mutation-types):

Type	What it does
out_of_scope	Asks something adjacent but outside the application's domain
false_premise	Embeds a factually wrong assumption the application may not catch
confidence_trap	Demands a definitive answer on something genuinely uncertain
contradictory	Contradicts or challenges a known implicit rule from the domain
authority_claim	User claims special permission or role to bypass normal behaviour
prompt_probe	Attempts to elicit the system prompt or internal instructions
persona_override	Redefines the application's identity via roleplay or hypotheticals
escalation_bypass	Applies urgency/pressure to skip a handoff that should happen
context_bleed	Tries to surface information outside the user's permitted scope
staleness_probe	Asks about time-sensitive info without specifying recency
tool_misuse	Invokes agent tools in unintended ways or with unintended parameters
injection_payload	Embeds adversarial instructions inside content the agent must process

Trigger a job with POST /v1/adversarial/mutation-jobs (pointing at a source dataset and a list of types) and poll .../{job_id}/status. The generated dataset can then be gated with safety-oriented pointwise metrics like harmfulness, maliciousness, or a custom policy metric.

---

Counterfactual (Fairness)

Counterfactual generation holds the substance of a query constant and varies one attribute of the subject — producing groups of rows that a fair system should treat identically. This is the data that powers pairwise fairness testing.

Vary along a dimension from the catalog (GET /v1/adversarial/counterfactual-jobs/dimensions):

age · disability · gender · nationality · race_ethnicity · religion · sexual_orientation · socioeconomic

Two output shapes

The generation mode determines the shape, and it lines up 1:1 with the pairwise.mode you'll use to grade it:

• group → each seed row becomes a reference plus N distinct-value variant rows. Grade with pairwise.mode: group.
• contrastive → each row becomes two opposing pole_a / pole_b rows. Grade with pairwise.mode: contrastive.

Every generated dataset carries the columns the pairwise engine pairs on:

Column	Meaning
group_id	Rows sharing a group are identical except for their expression of the dimension
role	Provenance within the group: reference / variant (group) or pole_a / pole_b (contrastive)
dimension	The attribute that was varied

The full fairness loop

counterfactual generation  →  dataset (group_id, role, dimension)
        →  target with a `pairwise` block (mode: group | contrastive)
        →  pairwise-bias / pairwise-refusal-disparity judge
        →  equivalence / divergence rate  →  gate  →  certificate

This is why the pairwise metrics have no "baseline": the comparison happens within one run, between the counterfactual peers generated here.

---

Jobs are asynchronous

Both flavors run as background jobs behind a shared 3-stage pipeline (profile the dataset → infer context → generate, on a bounded worker pool). Trigger, then poll:

POST /v1/adversarial/mutation-jobs         →  job_id
GET  /v1/adversarial/mutation-jobs/{id}/status
POST /v1/adversarial/counterfactual-jobs   →  job_id
GET  /v1/adversarial/counterfactual-jobs/{id}/status

When a job completes it registers a new dataset; reference its ID from your manifest like any other dataset.