Skip to main content

Defining Targets

A Suite acts as a container for ONE or MANY Targets. This structure allows CertOps to iterate and individually test a DAG (Directed Acyclic Graph) of independent components that make up your AI system.

For example, a RAG pipeline might be split into three targets:

  1. The Query Rewriter API
  2. The Document Retriever API
  3. The final Generation LLM

Target Structure​

Each target in your certops.yaml defines how to send a request to your API and how to extract the final answer from its response.

targets:
  - id: "support-agent-generator"
    name: "Final Answer Generator"
    
    # 1. Routing (Relative Path)
    endpoint: "/api/v1/generate"
    method: "POST"
    
    # 2. Authentication & Headers
    headers:
      Content-Type: "application/json"
      Authorization: "Bearer ${env.API_SECRET}"
      
    # 3. Dynamic Request Body
    request_template: |
      {
        "user_query": "{{ question }}",
        "session_id": "{{ session_id }}"
      }
      
    # 4. Response Extraction
    response_path: "data.choices[0].message.content"
    
    # 5. Dataset Binding
    dataset:
      id: "ds_customer_queries_v2"
      
    # 6. Evaluation Block (Covered in next section)
    evaluation: ...

Injecting Dynamic Variables​

Never hardcode sensitive credentials (like Authorization tokens) into your certops.yaml.

You can securely inject environment variables directly from your CI/CD runner at execution time using the ${env.VARIABLE_NAME} syntax:

    headers:
      X-API-Key: "${env.INTERNAL_ROUTING_KEY}"

When the CertOps CLI runs, it will read INTERNAL_ROUTING_KEY from your local machine or CI/CD runner's environment variables and inject it before dispatching the request.

Request Templating​

The request_template is how you map columns from your CSV Dataset into the physical JSON body of the HTTP request sent to your Target. It utilizes Jinja2 syntax.

If your dataset has a column named question, the template {{ question }} will be uniquely replaced with the row's actual question for every single evaluation in the run.

Response Extraction Paths​

AI APIs rarely return just a string. They return complex JSON objects containing usage metrics, citations, arrays, and metadata.

CertOps needs to isolate the exact string of text generated by the AI so it can be evaluated by the metrics. You define this using the response_path (using standard JSONPath syntax).

Given the following API response:

{
  "status": 200,
  "data": {
    "usage": { "tokens": 150 },
    "response_text": "Here is how to reset your password..."
  }
}

Your manifest should configure the response_path as:

    response_path: "data.response_text"