Defining Targets

A Suite acts as a container for ONE or MANY Targets. This structure allows CertOps to iterate and individually test a DAG (Directed Acyclic Graph) of independent components that make up your AI system.

For example, a RAG pipeline might be split into three targets:

1. The Query Rewriter API
2. The Document Retriever API
3. The final Generation LLM

Target Structure

Each target in your certops.yaml defines how to send a request to your API and how to extract the final answer from its response.

targets:
  - id: "support-agent-generator"
    name: "Final Answer Generator"
    
    # 1. Routing (Relative Path)
    endpoint: "/api/v1/generate"
    method: "POST"
    
    # 2. Authentication & Headers
    headers:
      Content-Type: "application/json"
      Authorization: "Bearer ${env.API_SECRET}"
      
    # 3. Dynamic Request Body
    request:
      format: "json"            # json (default) | raw | urlencoded | multipart
      body: |
        {
          "user_query": "{{ question }}",
          "session_id": "{{ session_id }}"
        }
      
    # 4. Response Extraction
    response_path: "data.choices[0].message.content"
    
    # 5. Dataset Binding
    dataset:
      id: "ds_customer_queries_v2"

    # 6. Resilience (optional) — retry a flaky endpoint before failing the sample
    retry_count: 2
    retry_delay: 1.0
      
    # 7. Evaluation Block (Covered in next section)
    evaluation: ...

Request Formats

CertOps is a generic API caller — the target dictates its own input format, declared in the request block:

format	Body source	Encoding
json (default)	body Jinja template → JSON (or the raw dataset row if body is omitted)	application/json
raw	body Jinja template → raw string (set your own Content-Type)	as-is
urlencoded	fields (each Jinja-rendered)	application/x-www-form-urlencoded
multipart	fields (text parts) + files (file-reference columns)	multipart/form-data

Multimodal uploads use multipart; each files entry names a dataset column whose cell is a file reference (an S3 key or URL), and CertOps attaches the bytes as a real file part.

Injecting Dynamic Variables

Never hardcode sensitive credentials (like Authorization tokens) into your certops.yaml.

You can securely inject environment variables directly from your CI/CD runner at execution time using the ${env.VARIABLE_NAME} syntax:

    headers:
      X-API-Key: "${env.INTERNAL_ROUTING_KEY}"

When the CertOps CLI runs, it will read INTERNAL_ROUTING_KEY from your local machine or CI/CD runner's environment variables and inject it before dispatching the request.

Request Templating

The request.body (or request.fields) is how you map columns from your CSV Dataset into the physical body of the HTTP request sent to your Target. It utilizes Jinja2 syntax.

If your dataset has a column named question, the template {{ question }} will be uniquely replaced with the row's actual question for every single evaluation in the run.

Inter-Target Variable Passing (depends_on)

When testing a pipeline of multiple AI components, downstream targets frequently need to consume the output of upstream targets.

CertOps allows you to define these relationships using the depends_on block. You can map variables to the full, raw HTTP JSON response of an upstream target, or explicitly extract specific fields using dot notation.

These mapped variables are then seamlessly available precisely like dataset columns in your request body and headers properties:

targets:
  - id: "classify-agent"
    endpoint: "/classify"
    # Suppose this API returns: {"species": {"id": "cat-001", "name": "Cats"}}

  - id: "query-agent"
    depends_on:
      species_data: "classify-agent"             # Passes the full JSON dict
      species_name: "classify-agent.species.name"# Passes just "Cats"
      species_id: "classify-agent.species.id"    # Passes just "cat-001"
    headers:
      X-Species-Id: "{{ species_id }}"
    request:
      body: |
        { "query": "{{ user_query }}", "focused_species": "{{ species_name }}" }

Note: The upstream source targets must be explicitly defined in your manifest before the downstream targets that depend on them.

Response Extraction Paths

AI APIs rarely return just a string. They return complex JSON objects containing usage metrics, citations, arrays, and metadata.

CertOps needs to isolate the exact string of text generated by the AI so it can be evaluated by the metrics. You define this using the response_path (using standard JSONPath syntax).

Given the following API response:

{
  "status": 200,
  "data": {
    "usage": { "tokens": 150 },
    "response_text": "Here is how to reset your password..."
  }
}

Your manifest should configure the response_path as:

    response_path: "data.response_text"