Skip to main content

Metrics Overview

Metrics are the core grading criteria CertOps uses to evaluate the quality of your AI system's outputs.

When you configure a test in your certops.yaml, you map dataset columns to specific metrics. CertOps then runs these metrics against the output produced by your Target.

CertOps provides over 30 built-in metrics. They fall into two families: deterministic (local, code-based) and LLM-as-Judge. The judge family has three kinds, distinguished by how many responses they look at and whether the comparison has a direction.

Deterministic Metrics​

Computed locally using traditional rules-based programming or fast embedding models (like Cosine Similarity or ROUGE-L).

  • Cost: Free (Zero LLM Tokens).
  • Use Case: Checking strict structural rules (like JSON validity), exact lexical matching, or basic semantic similarity against a known reference.

👉 See Deterministic Metrics.

LLM-as-Judge Metrics​

1. Pointwise (Evaluation)​

Evaluate a single response in isolation against an absolute rubric.

  • Use Case: Subjective or complex criteria traditional code cannot handle — Hallucination, Coherence, Tone Consistency.

👉 See Pointwise (Evaluation) Metrics.

2. Regression (Directional)​

Compare a candidate response against a baseline response, resolved from a tagged run, and decide which is better (win / loss / tie).

  • Use Case: Determining if a new prompt or model swap improved or degraded the system over time.

👉 See Regression Metrics.

3. Pairwise (Symmetric / Fairness)​

Compare two peer responses for divergence, with no baseline and no direction.

  • Use Case: Counterfactual and fairness testing — does the system treat two otherwise-identical inputs equivalently?

👉 See Pairwise (Fairness) Metrics.