System Entities

The CertOps platform is built around three core entities that interact during an evaluation run: Models, Datasets, and Metrics.

While you can configure these directly in your certops.yaml manifest or via the CLI, they act as centralized, reusable objects within your organization's environment.

1. Datasets & Media Hub

The Dataset serves as your system's ground truth. Instead of hard-coding datasets to specific tests, CertOps allows a single centralized dataset to be mapped across many different system components. Utilizing the Media Hub, these datasets can even include multimodal assets like images or PDFs.

👉 Refer to the Datasets & Media Hub guide to learn about schema inference, relative asset paths, and dynamic column mapping.

2. Metrics Library

Metrics are the rules and criteria used to grade your system's outputs. CertOps provides a robust library of built-in deterministic, pointwise (evaluation), and pairwise (regression) metrics, but you can also design custom LLM-as-Judge metrics tailored to your strict business logic.

👉 Refer to the Metrics Library guide for the full list of available metrics and instructions on building custom grading criteria.

3. Model Configurations

When running LLM-as-Judge Metrics or generating formatting synthentic data, CertOps needs to know which LLM to use to perform the evaluation. Model Configs allow you to save API keys, routing URLs, and deployment details for providers like OpenAI, Azure, or local endpoints.

👉 Refer to the Model Configs guide to learn how to register evaluating models.