Defining the Test Suite

CertOps embraces Configuration as Code. The certops.yaml manifest is the single source of truth for auditing an AI System. It defines exactly what components to test, what datasets to challenge them with, and the exact grading criteria (metrics) those components must pass.

By committing certops.yaml to your source control alongside your application code, you ensure that your quality and evaluation standards are version-controlled, auditable, and reproducible.

Environment Agnosticism

A core philosophy of CertOps is that your tests should be completely decoupled from the physical environment they run in.

The Golden Rule

The manifest explicitly defines endpoints (e.g., /v1/chat), not hosts (e.g., https://api.production.com).

This means the exact same certops.yaml file can be used to independently certify Local Development, Staging, and Production environments without modifying a single line of code.

You provide the Host dynamically via the CertOps CLI at execution time:

# Testing locally
certops run --host "http://localhost:8000"

# Testing in production
certops run --host "https://api.mycompany.com"

Global Configuration

The top of your certops.yaml file defines the metadata and global settings for the entire test suite.

version: "1.0"

# Suite Metadata
suite:
  name: "Customer Support RAG Pipeline"
  owner: "cx-engineering-team"

# Global System Configuration
configuration:
  # The unified Model Config used by LLM-as-Judge metrics
  judge_model_config_id: "production-eval-gpt4o"
  
  # How many simultaneous requests to make to your API
  concurrency: 5
  
  # If true, stop testing Target B if Target A fails
  stop_on_failure: true