Skip to main content

Custom Metrics

While CertOps provides over 30 built-in "System" metrics, every AI use case is unique. You will frequently encounter scenarios where you need to evaluate highly specific organizational policies, complex routing rules, or deeply nuanced brand personas.

CertOps allows you to easily create your own Custom LLM-as-Judge Metrics via the Dashboard UI or the REST API. You can author any of the three judge kinds — pointwise (evaluation), regression (directional), or pairwise (symmetric fairness) — each with its own scoring rubric and variables.

Creating a Custom Metric​

To create a new custom metric in the Dashboard, navigate to the Metrics screen and click + New Metric.

You will need to configure three primary sections:

1. Basic Information​

  • Metric Name: A human-readable identifier (e.g., Brand Tone Check). The system will automatically generate a kebab-case ID (e.g., brand-tone-check) that you will use in your certops.yaml.
  • Description: A brief explanation of what the metric evaluates, helping your team understand its purpose.

2. Score Configuration​

Unlike testing frameworks with rigid Pass/Fail binaries, CertOps allows you to define the exact quantitative scale your metric operates on:

  • Minimum Score: The absolute lowest possible score (e.g., 0, 1, or -1).
  • Maximum Score: The absolute highest possible score (e.g., 1, 5, or 100).
  • Pass Threshold: (Optional) The cut-off score required for the evaluation to be considered "Successful" or "Passing" within a CI/CD Quality Gate.

3. Prompt Template (Jinja2)​

This is the core instructions sent to the LLM Judge. It contains your scoring rubric, edge cases, and the criteria the LLM must follow to grade the output securely.

CertOps uses Jinja2 templating to inject the evaluation data dynamically. You are not restricted to a fixed set of variables.

You can define any arbitrary variable names you need by wrapping them in double curly braces (e.g., {{user_persona}}, {{strict_rules}}, {{expected_length}}).

When you configure your test in the certops.yaml manifest, you will map these custom Jinja variables directly to the column names in your Dataset. This allows your metrics to be highly specialized without forcing your datasets into a rigid schema.

Tip! When writing your prompt template, always instruct the LLM to output a strictly structured response containing the numerical score. For example: Provide ONLY a JSON object: {"score": <number>, "reasoning": "<explanation>"}.

AI-Assisted Metric Generation​

Writing a good rubric from scratch is hard. CertOps can help you author metrics with an LLM, so you describe what you want to measure and the platform drafts the rubric and prompt for you. The flow (exposed both in the Dashboard and via the /v1/metrics API) has a few entry points:

  • Ideation (/v1/metrics/generate/ideas) — describe your use case and get a set of candidate metric ideas (name, description, what it would measure) to choose from.
  • Realization (/v1/metrics/generate/realize) — turn a chosen idea into a fully formed metric: scoring rubric, prompt template, and variables, ready to save.
  • Recommendation (/v1/metrics/recommend) — given a dataset, get suggestions for which existing or new metrics are the best fit for its columns.
  • Hybrid (/v1/metrics/generate/hybrid) — combine your own guidance with AI-drafted structure in a single pass.

The Calibration Playground​

Before you commit a metric to a blocking gate, you can calibrate it in the Playground (/v1/metrics/playground). You run the draft metric against a handful of sample inputs and see the scores and reasoning it produces — then tweak the rubric until the judge agrees with your own judgment. This prevents the classic failure mode of a custom metric that quietly mis-scores an entire certification run.