Evaluation, Regression & Pairwise Config

Within your target definition in the certops.yaml manifest, you must configure how CertOps actually grades the responses it receives from your AI component.

CertOps offers three grading blocks, each optional and independent:

1. evaluation (Pointwise): grading individual responses in isolation against absolute criteria.
2. regression: directional comparison of the current run against a historical known-good baseline (drift + win/loss/tie).
3. pairwise: symmetric comparison of peer responses within a run (counterfactual / fairness).

No comparison umbrella

Earlier versions nested comparison under a single comparison: block with a pairing.mode selector. That schema is retired — the backend now rejects it with a migration error. Use the sibling regression and pairwise blocks below instead. Gate keys are axis-specific: a max_loss_rate (regression) inside a pairwise block is an authoring error, not silently ignored.

1. Pointwise Evaluation Block

The evaluation block defines the absolute criteria your target must meet for a single run.

    evaluation:
      # Map dataset columns -> metric prompt variables
      metrics_mapping:
        input: "user_query"
        reference: "expected_document"

      # Deterministic local tier
      deterministic:
        - metric: "cosine-similarity"
          threshold: 0.80
          operator: "gte"
          blocking: true

      # LLM Judge per-sample tier
      llm:
        - metric: "hallucination"
          threshold: 1.0
          operator: "gte"
          blocking: true
        - metric: "answer-relevance"
          threshold: 0.8
          operator: "gte"
          blocking: false 

The blocking Flag (Hard vs Soft Gates)

• blocking: true: This metric is a Hard Gate. If the average score across your dataset falls below the threshold, the Component Verdict (and the Suite by extension) is REJECTED. The CLI will exit with a non-zero exit code.
• blocking: false: This metric is a Soft Gate. If it fails, CertOps will log a warning in the dashboard, but the overall run can still be marked as Certified.

The metrics_mapping Object

Because datasets are decoupled from metrics, you must explicitly tell CertOps which dataset columns should be piped into the Jinja2 variables required by your chosen metrics.

If your chosen metric (hallucination) requires an {{ input }} variable, but your CSV has a column named user_query, you map it here (input: "user_query"). (Note: The {{ output }} variable is automatically injected by CertOps using the parsed response_path from your target).

2. Regression Block (Directional)

The regression block mirrors the evaluation block, but instead of grading against an absolute threshold, it enforces comparisons against a Baseline (usually the version currently live in Production, resolved by tag).

    regression:
      baseline: "prod"

      # Average Drift Tier
      deterministic:
        - metric: "cosine-similarity"
          max_drift: 0.05      # Avg run score cannot drop more than 5% vs baseline
        - metric: "hallucination"
          max_drift: 0.1

      # Directional LLM judge tier (win/loss/tie vs baseline output)
      metrics_mapping:
        input: "question"
      llm:
        - metric: "general-quality"
          max_loss_rate: 0.3   # Candidate can't lose > 30% of individual matches
          blocking: true

• max_drift: In the deterministic tier, this prevents silent degradation. If your baseline cosine-similarity was 0.90, a max_drift of 0.05 means the new build cannot average below 0.85.
• max_loss_rate / min_win_rate: In the LLM tier, the judge compares the old prod output vs the new candidate output side-by-side per sample. If the candidate loses more than 30% of those head-to-head comparisons, the run is rejected. Use Regression Metrics here.

If no run carries the baseline tag yet, the comparison is skipped with a non-blocking notice and the run still passes on evaluation alone.

3. Pairwise Block (Symmetric / Fairness)

The pairwise block compares peer responses within a single run — no baseline, no direction. It powers counterfactual and fairness testing, grouping rows by a group_by column and pairing them by role.

    pairwise:
      mode: "group"            # group | contrastive
      group_by: "group_id"
      role: "role"
      metrics_mapping:
        input: "question"
      llm:
        - metric: "pairwise-bias"
          min_equivalence_rate: 0.9   # ≥ 90% of pairs must be treated equivalently
          blocking: true

• min_equivalence_rate: the fraction of pairs the judge must find equivalent.
• max_divergence_rate / max_mean_divergence: bound how often, or how much, treatment may diverge.

Use Pairwise (Fairness) Metrics here — regression metrics are not valid in this block.