Batch CLI Evaluation

Batch CLI evaluation handles tools that process multiple inputs at once — bulk classifiers, screening engines, or any runner that reads all tests and outputs results in one pass.

Overview

Use batch CLI evaluation when:

• An external tool processes multiple inputs in a single invocation (e.g., AML screening, bulk classification)
• The runner reads the eval YAML directly to extract all tests
• Output is JSONL with records keyed by test id
• Each test has its own grader to validate its corresponding output record

Execution Flow

1. AgentV invokes the batch runner once, passing --eval <yaml-path> and --output <jsonl-path>
2. Batch runner reads the eval YAML, extracts all tests, processes them, and writes JSONL output keyed by id
3. AgentV parses the JSONL and routes each record to its matching test by id
4. Per-test graders validate the output for each test independently

Eval File Structure

description: Batch CLI demo using structured input
providers:
  - batch_cli

prompts:
  - "{{ input }}"

tests:
  - id: case-001
    criteria: |-
      Batch runner returns JSON with decision=CLEAR.
    vars:
      input:
        - role: system
          content: You are a batch processor.
        - role: user
          content:
            request:
              type: screening_check
              jurisdiction: AU
            row:
              id: case-001
              name: Example A
              amount: 5000
      expected_output:
        - role: assistant
          content:
            decision: CLEAR

    assert:
      - name: decision-check
        type: script
        command: [bun, run, ./scripts/check-output.ts]
        cwd: .

  - id: case-002
    criteria: |-
      Batch runner returns JSON with decision=REVIEW.
    vars:
      input:
        - role: system
          content: You are a batch processor.
        - role: user
          content:
            request:
              type: screening_check
              jurisdiction: AU
            row:
              id: case-002
              name: Example B
              amount: 25000
      expected_output:
        - role: assistant
          content:
            decision: REVIEW

    assert:
      - name: decision-check
        type: script
        command: [bun, run, ./scripts/check-output.ts]
        cwd: .

Batch Runner Contract

The batch runner reads the eval YAML directly and processes all tests in one invocation.

Input

The runner receives the eval file path via --eval and an output path via --output:

bun run batch-runner.ts --eval ./my-eval.yaml --output ./output.jsonl

Output

JSONL where each line is a JSON object with an id matching a test:

{"id": "case-001", "text": "{\"decision\": \"CLEAR\", ...}"}
{"id": "case-002", "text": "{\"decision\": \"REVIEW\", ...}"}

The id field must match the test id for AgentV to route output to the correct grader.

Output with Trajectory Assertions

To enable trajectory:* assertions, include output with tool_calls:

{
  "id": "case-001",
  "text": "{\"decision\": \"CLEAR\", ...}",
  "output": [
    {
      "role": "assistant",
      "tool_calls": [
        {
          "tool": "screening_check",
          "input": { "origin_country": "NZ", "amount": 5000 },
          "output": { "decision": "CLEAR", "reasons": [] }
        }
      ]
    },
    {
      "role": "assistant",
      "content": { "decision": "CLEAR" }
    }
  ]
}

AgentV extracts tool calls directly from output[].tool_calls[] for trajectory assertions.

Grader Implementation

Each test has its own grader that validates the batch runner output. The grader receives the standard script input via stdin.

Input (stdin):

{
  "output": "{\"id\":\"case-001\",\"decision\":\"CLEAR\",...}",
  "expected_output": [{"role": "assistant", "content": {"decision": "CLEAR"}}],
  "input": [...]
}

Output (stdout):

{
  "pass": true,
  "score": 1.0,
  "reason": "Batch runner decision matches expected.",
  "checks": [
    { "text": "decision matches: CLEAR", "pass": true, "reason": "Expected and actual decisions are CLEAR." }
  ]
}

Example Grader

import fs from 'node:fs';

type EvalInput = {
  output?: string;
  expected_output?: Array<{ role: string; content: unknown }>;
};

function main() {
  const stdin = fs.readFileSync(0, 'utf8');
  const input = JSON.parse(stdin) as EvalInput;

  const expectedDecision = findExpectedDecision(input.expected_output);

  let candidateDecision: string | undefined;
  try {
    const parsed = JSON.parse(input.output ?? '');
    candidateDecision = parsed.decision;
  } catch {
    candidateDecision = undefined;
  }

  const checks: Array<{ text: string; pass: boolean; reason: string }> = [];

  if (expectedDecision === candidateDecision) {
    checks.push({ text: `decision matches: ${expectedDecision}`, pass: true, reason: 'Expected and actual decisions match.' });
  } else {
    checks.push({ text: `mismatch: expected=${expectedDecision} actual=${candidateDecision}`, pass: false, reason: 'Expected and actual decisions differ.' });
  }

  const passed = checks.every((check) => check.pass);

  process.stdout.write(JSON.stringify({
    pass: passed,
    score: passed ? 1 : 0,
    reason: passed
      ? 'Batch runner output matches expected.'
      : 'Batch runner output did not match expected.',
    checks,
  }));
}

function findExpectedDecision(messages?: Array<{ role: string; content: unknown }>) {
  if (!messages) return undefined;
  for (const msg of messages) {
    if (typeof msg.content === 'object' && msg.content !== null) {
      return (msg.content as Record<string, unknown>).decision as string;
    }
  }
  return undefined;
}

main();

Structured Content

Use structured objects in vars.expected_output to define expected output fields for easy validation:

vars:
  expected_output:
    - role: assistant
      content:
        decision: CLEAR
        confidence: high
        reasons: []

The grader extracts these fields and compares them against the parsed candidate output.

Target Configuration

Configure the batch CLI provider in your targets file or eval file:

# In agentv-providers.yaml or eval file
providers:
  batch_cli:
    provider: cli
    command: bun run ./scripts/batch-runner.ts --eval {EVAL_FILE} --output {OUTPUT_FILE}
    batch_requests: true

Key settings:

Setting	Description
provider: cli	Use the CLI provider
batch_requests: true	Run once for all tests instead of per-test
{EVAL_FILE}	Placeholder replaced with the eval file path
{OUTPUT_FILE}	Placeholder replaced with the JSONL output path

Best Practices

1. Use unique test IDs — the batch runner and AgentV use id to route outputs to the correct grader
2. Structured input — put structured data in user.content for the runner to extract
3. Structured expected_output — define expected output as objects for easy comparison
4. Deterministic runners — batch runners should produce consistent output for reliable testing
5. Healthcheck support — add a --healthcheck flag for runner validation:

   if (args.includes('--healthcheck')) {
     console.log('batch-runner: healthy');
     return;
   }