Quickstart

This guide takes you from zero to a signed certificate in about ten minutes. You'll upload a dataset, write a one-target manifest, certify a running agent, promote a baseline, and mint a Certificate of Conformity.

By the end you'll have exercised the whole loop:

upload  →  manifest  →  run (certify)  →  tag (baseline)  →  certificate

Prerequisites

• Python 3.11+
• An HTTP endpoint to test — any agent that takes a request and returns text. For this guide we'll assume a support chatbot at https://staging.acme.com exposing POST /chat.
• Your CertOps credentials, in tenant\email form.

1. Install & authenticate

pip install certops-cli

# Username is a tenant\email pair
certops login --username 'acme\alice@acme.com'

login caches your token (and, if configured, your Local Bridge relay) under ~/.certops. Confirm it with certops whoami.

Pointing at a different environment

By default the CLI talks to https://api.certops.ai. Override with --api-url or the CERTOPS_API_URL env var — e.g. a local backend: certops --api-url http://localhost:8000 login ....

2. Upload a dataset

Your dataset is the ground truth you'll challenge the agent with. A minimal CSV for a Q&A bot has a question and a known-good answer:

support_golden.csv

question,golden_answer
"How do I reset my password?","Go to Settings → Security → Reset Password and follow the email link."
"What are your support hours?","Our team is available 24/7 via chat and email."
"Can I get a refund?","Refunds are available within 30 days of purchase from your Orders page."

Upload it and note the returned Dataset ID:

certops upload support_golden.csv

✓ Dataset uploaded successfully
  ID     : ds_a1b2c3d4
  Rows   : 3
  Columns: question, golden_answer

3. Write the manifest

The certops.yaml manifest is your test-as-code. It defines what to call and how to grade it — but never where (the host is injected at run time, so one manifest certifies dev, staging, and prod alike).

certops.yaml

version: "1.0"

suite:
  name: "Support Chatbot Certification"
  owner: "cx-team"

targets:
  - id: "chatbot"
    name: "Support Chatbot"
    endpoint: "/chat"              # relative path only — no host
    method: "POST"

    request:
      format: "json"
      body: |
        { "message": "{{ question }}" }

    # Pull the agent's answer out of its JSON response
    response_path: "data.reply"

    dataset:
      id: "ds_a1b2c3d4"            # ← the ID from step 2

    evaluation:
      # Map dataset columns → the variables each metric expects
      metrics_mapping:
        input: "question"
        reference: "golden_answer"

      # Local, zero-cost checks
      deterministic:
        - metric: "cosine-similarity"
          threshold: 0.75
          operator: "gte"
          blocking: true
        # invocation-success is auto-injected — gate on it to fail on any HTTP error
        - metric: "invocation-success"
          threshold: 1.0
          operator: "gte"
          blocking: true

      # LLM-as-Judge checks
      llm:
        - metric: "answer-relevance"
          threshold: 0.8
          operator: "gte"
          blocking: true

configuration:
  judge_model_config_id: "your-judge-model-config"   # the LLM that runs the llm metrics
  concurrency: 5

A few things worth knowing:

• {{ question }} in the request body is a Jinja variable filled from your dataset, row by row. The agent's answer, extracted via response_path, is always available to metrics as {{ output }}.
• blocking: true makes a gate a hard gate — if it fails, the run is Rejected.
• judge_model_config_id points at a Model Config — the LLM used to run the llm metrics. Create one in the Dashboard first.

See Defining Targets and Evaluation config for the full schema.

4. Certify

Inject the host and run. The CLI triggers the run, streams live progress, prints a per-gate summary, and exits with a contract-defined code.

certops run -f certops.yaml --host chatbot=https://staging.acme.com

🚀 Triggering certification run...
  Run ID: 06a5-…-c480
  Run #  : 1
📡 Polling for results...

Support Chatbot (chatbot) — Certified
 🔧 Eval   cosine-similarity    ✅ PASS   0.8100 gte 0.75
 🔧 Eval   invocation-success   ✅ PASS   1.0000 gte 1.0
 🔧 Eval   answer-relevance     ✅ PASS   0.9000 gte 0.8

✅  CERTIFIED

The command exits 0. In CI, that green exit is your quality gate — the pipeline proceeds. A blocking failure would exit 1 (Rejected), and a timeout or outage would exit 2 (System Error), which fails the build without falsely blaming your model. See Automated CI/CD Quality Gates.

5. Promote a baseline

Right now you've proven absolute competence. To also catch drift — "we passed, but did we get worse than last week?" — mark this good run as the prod baseline:

certops tag <run_id> prod

Now add a regression block to the target so future runs compare against it directly:

    regression:
      baseline: "prod"
      deterministic:
        - metric: "cosine-similarity"
          max_drift: 0.05          # average can't drop >5% vs the prod baseline
      metrics_mapping:
        input: "question"
      llm:
        - metric: "general-quality"
          max_loss_rate: 0.3       # candidate can't lose >30% of head-to-heads
          blocking: true

The next certops run will resolve the latest prod-tagged run and gate on drift and win/loss/tie. (Testing for fairness instead of drift? That's the symmetric pairwise axis, fed by counterfactual generation.)

6. Mint the certificate

Finally, turn a certified run into a durable, verifiable artifact — addressed by its run number (#1 above):

certops certificate generate 1

📜 Certificate of Conformity
  Certificate : cert-9f3c…
  Suite       : Support Chatbot Certification
  Run #       : 1
  Verdict     : Certified
  PDF         : https://…   (expires ~1h)

Anyone can verify it — no login required — via the public Trust Badge:

certops certificate verify cert-9f3c...

Where to go next

• Test without deploying — point a target at local:8080 and the Local Bridge tunnels the SaaS to your laptop.
• Multi-target pipelines — certify a RAG retriever and generator in one run, and chain them with depends_on. See Defining Targets.
• Go beyond the happy path — synthesize adversarial and counterfactual datasets with Adversarial Generation.
• Full command reference — CLI Commands.