> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blackbox.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Benchmarks

> Launch a benchmark evaluation (SWE-bench, GAIA, AIME, GPQA, …) driven by the agent of your choice. Fire-and-forget: returns a benchmarkRunId immediately.

This endpoint starts a benchmark run that evaluates an agent across a dataset of tasks. The run is **fire-and-forget** — it returns a `benchmarkRunId` immediately and executes in the background on the managed sandbox. Like tasks, a benchmark can run on the **Claude**, **Codex**, or **Grok Build** [agent runtime](/api-reference/v1/agent-runtimes), selected from `model` or forced with `agent`.

<Tip>
  After launching, read the score and per-task traces with [Benchmark Results](/api-reference/v1/benchmark-results), and follow progress with [Benchmark Logs & Status](/api-reference/v1/benchmark-logs).
</Tip>

## Authentication

All requests require a BLACKBOX API key as a Bearer token (Pro plan required for `POST`). See [Authentication](/api-reference/v1/authentication).

## Headers

<ParamField header="Authorization" type="string" required>
  API Key of the form `Bearer <api_key>`.
</ParamField>

<ParamField header="Content-Type" type="string" required>
  Must be `application/json`.
</ParamField>

## Request Body

<ParamField body="benchmark" type="string" required>
  The benchmark to run. One of: `swe-bench`, `swe-bench-lite`, `swe-bench-multilingual`, `swe-gym`, `swe-bench-multimodal`, `hle`, `gaia`, `aime`, `gpqa`, `gpqa-main`, `omni-math`, `mmlu-pro`, `simpleqa`. An unknown value returns `400` listing the supported names.
</ParamField>

<ParamField body="model" type="string">
  Model that drives the agent. The id also selects the **agent runtime** — Anthropic/Claude ids run the Claude Agent SDK; OpenAI/Codex ids (e.g. `blackboxai/openai/gpt-5.3-codex`) run the Codex SDK; xAI Grok Build ids run the Grok CLI. See [Agent Runtimes](/api-reference/v1/agent-runtimes).

  <Warning>
    For the **Grok Build** runtime, pass the model's **full router id** — `blackboxai/x-ai/grok-build-0.1` — not the bare `x-ai/grok-build-0.1`. The Grok CLI validates `-m` against the router's model list and rejects an unprefixed id with `"unknown model id"`.
  </Warning>
</ParamField>

<ParamField body="agent" type="string">
  Explicit agent-runtime override — `"claude"`, `"codex"`, or `"grok"`. Overrides the runtime inferred from `model`. Omit to auto-select (default: **claude**). An invalid value returns `400` listing the supported agents.
</ParamField>

<ParamField body="apiKey" type="string">
  **Bring-your-own router** — your OpenAI-compatible router key (bearer token). Must be paired with `baseUrl`. When supplied, the agent runtime is pointed at **your** endpoint instead of the platform router, and `model` is passed through **verbatim** (no allowlist check). The key is used in-memory for the run only and is **never persisted**. See [Bring-your-own router](#bring-your-own-router).
</ParamField>

<ParamField body="baseUrl" type="string">
  **Bring-your-own router** — your router base URL (e.g. `https://my-router.example.com`). Must be an `http(s)` URL and paired with `apiKey`.
</ParamField>

<ParamField body="prompt" type="string">
  Optional extra instruction **appended to every task's auto-generated prompt** — a global steer applied across the whole run (e.g. `"Prefer minimal diffs; add a regression test"`). It does **not** replace the dataset-generated task instruction; it's added after it. Omit for the standard benchmark instruction.
</ParamField>

<ParamField body="nConcurrent" type="number" default="16">
  Maximum concurrent tasks within the run. Range `1`–`16` (defaults to `16`). What this concurrency *uses* depends on `env` — see [Concurrency & execution backends](#concurrency-execution-backends).
</ParamField>

<ParamField body="limit" type="number" default="10">
  Number of dataset instances to evaluate. Minimum `1`; there is **no fixed maximum** — it's clamped only to the benchmark's own dataset size (`totalInstances`, e.g. 12,032 for MMLU-Pro, 60 for AIME). Larger runs take proportionally longer and cost more.
</ParamField>

<ParamField body="timeout" type="number">
  Per-task agent timeout in seconds. Defaults to the benchmark's own default (e.g. `1800` for SWE-bench, `900` for AIME/MMLU-Pro/SimpleQA).
</ParamField>

<ParamField body="env" type="string" default="sandbox-per-task">
  Execution backend — how each task's environment is provisioned. See [Concurrency & execution backends](#concurrency-execution-backends).

  * `"sandbox-per-task"` (default) — each concurrent task runs in **its own isolated sandbox** (restored from the prepared snapshot). Strong isolation; concurrency is bounded by your account's **concurrent-sandbox quota**.
  * `"docker-in-parent"` — all tasks run as **concurrent Docker containers inside a single sandbox**. No per-task sandboxes, so concurrency is bounded by that one VM's **CPU / RAM** instead of the sandbox quota. Cheaper for small runs.
</ParamField>

## Response

<ResponseField name="benchmarkRunId" type="string">
  Unique id for the run. Use it to poll status, list tasks, or stream logs.
</ResponseField>

<ResponseField name="status" type="string">
  Initial status — `"queued"`.
</ResponseField>

<ResponseField name="benchmark" type="string">
  Resolved canonical benchmark name.
</ResponseField>

<ResponseField name="model" type="string">
  The model driving the agent (or `null` for the default).
</ResponseField>

<ResponseField name="agent" type="string">
  The resolved runtime that will actually run — `"claude"`, `"codex"`, or `"grok"` — after applying the override or model-based inference. This is also stored on the run, so [results](/api-reference/v1/benchmark-results) report the true agent.
</ResponseField>

<ResponseField name="byo" type="object | null">
  Echo of bring-your-own router usage (or `null`). The `apiKey` is masked: `{ "baseUrl": "...", "apiKey": "***", "model": "..." }`.
</ResponseField>

<ResponseField name="prompt" type="string | null">
  The extra instruction appended to each task (or `null` if none was provided).
</ResponseField>

<ResponseField name="nConcurrent" type="number">Resolved concurrency.</ResponseField>
<ResponseField name="limit" type="number">Resolved instance count.</ResponseField>
<ResponseField name="timeout" type="number">Resolved per-task timeout (seconds).</ResponseField>
<ResponseField name="env" type="string">Resolved execution backend.</ResponseField>

<RequestExample>
  ```bash Claude agent theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
  curl -X POST 'https://agent.blackbox.ai/api/v1/benchmarks/runs' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{
      "benchmark": "swe-bench-lite",
      "model": "blackboxai/anthropic/claude-sonnet-4.5",
      "limit": 20,
      "nConcurrent": 4
    }'
  ```

  ```bash Codex agent theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
  curl -X POST 'https://agent.blackbox.ai/api/v1/benchmarks/runs' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{
      "benchmark": "swe-bench-lite",
      "model": "blackboxai/openai/gpt-5.3-codex",
      "limit": 20
    }'
  ```

  ```bash Force Codex on a non-OpenAI model theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
  curl -X POST 'https://agent.blackbox.ai/api/v1/benchmarks/runs' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{
      "benchmark": "gpqa",
      "model": "blackboxai/openai/gpt-5.5",
      "agent": "codex",
      "limit": 50
    }'
  ```

  ```bash With an extra instruction (prompt) theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
  curl -X POST 'https://agent.blackbox.ai/api/v1/benchmarks/runs' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{
      "benchmark": "swe-bench-lite",
      "model": "blackboxai/openai/gpt-5.3-codex",
      "agent": "codex",
      "prompt": "Prefer minimal diffs; add a regression test.",
      "limit": 20
    }'
  ```

  ```bash Grok Build agent theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
  curl -X POST 'https://agent.blackbox.ai/api/v1/benchmarks/runs' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{
      "benchmark": "aime",
      "model": "blackboxai/x-ai/grok-build-0.1",
      "agent": "grok",
      "limit": 10,
      "nConcurrent": 5
    }'
  ```

  ```bash Bring-your-own router (your endpoint + key + model) theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
  curl -X POST 'https://agent.blackbox.ai/api/v1/benchmarks/runs' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{
      "benchmark": "aime",
      "limit": 10,
      "baseUrl": "https://my-router.example.com",
      "apiKey": "sk-my-router-key",
      "model": "my-org/my-model"
    }'
  ```
</RequestExample>

<ResponseExample>
  ```json Success theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
  {
    "benchmarkRunId": "b1c2d3e4-f5a6-7890-bcde-f12345678901",
    "status": "queued",
    "benchmark": "swe-bench-lite",
    "dataset": "princeton-nlp/SWE-bench_Lite",
    "model": "blackboxai/openai/gpt-5.3-codex",
    "agent": "codex",
    "prompt": "Prefer minimal diffs; add a regression test.",
    "nConcurrent": 4,
    "limit": 20,
    "timeout": 1800,
    "env": "sandbox-per-task"
  }
  ```

  ```json Error — unknown benchmark theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
  {
    "error": "Unknown benchmark \"swebenchx\". Supported: swe-bench, swe-bench-lite, ..."
  }
  ```
</ResponseExample>

## Bring-your-own router

Supply `apiKey` + `baseUrl` (always as a pair) to run the benchmark against **your own** OpenAI-compatible endpoint and key instead of the platform router. When BYO creds are present:

* the agent runtime (claude/codex/grok) is pointed at your `baseUrl` with your `apiKey`;
* `model` is passed through **verbatim** — no allowlist check — so you can evaluate self-hosted or third-party models;
* the key is used **in-memory for the run only** and is never stored;
* the response echoes a masked `byo` block (`apiKey` shown as `***`).

The same `apiKey` / `baseUrl` fields are also accepted by [Create Task](/api-reference/v1/create-task) to drive the interactive agent against your endpoint.

## Concurrency & execution backends

`nConcurrent` sets how many tasks run at once; `env` decides **what that concurrency consumes**.

|                  | `sandbox-per-task` (default)                                            | `docker-in-parent`                                                 |
| ---------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------ |
| Concurrency unit | One **isolated sandbox** per task (restored from the prepared snapshot) | One **Docker container** per task, all inside a **single** sandbox |
| Bounded by       | Your account's **concurrent-sandbox quota** (+ model rate limits)       | That one VM's **CPU / RAM / Docker** (+ model rate limits)         |
| Isolation        | Strong — separate VMs                                                   | Shared VM                                                          |
| Sandbox count    | `nConcurrent` sandboxes                                                 | **1** sandbox total                                                |
| Best for         | Heavy/long tasks (e.g. SWE-bench), strong isolation                     | Small/cheap runs, or high concurrency without using sandbox quota  |

To run tasks **concurrently inside a single sandbox** (rather than one sandbox each), use `env: "docker-in-parent"`:

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
curl -X POST 'https://agent.blackbox.ai/api/v1/benchmarks/runs' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "benchmark": "aime",
    "model": "blackboxai/x-ai/grok-build-0.1",
    "agent": "grok",
    "limit": 90,
    "nConcurrent": 16,
    "env": "docker-in-parent"
  }'
```

<Note>
  `nConcurrent` is capped at **16**. In `sandbox-per-task` the practical limit is your Vercel concurrent-sandbox quota; in `docker-in-parent` it's the single VM's resources (too many parallel containers + agent processes will saturate CPU/RAM). Raise the cap only alongside the matching infra headroom.
</Note>

## Listing runs & sub-resources

`GET /api/v1/benchmarks/runs` returns the authenticated user's benchmark runs (optionally filtered by `?status=`). Per-run sub-resources live under `/api/v1/benchmarks/runs/{benchmarkRunId}/…`:

| Sub-resource        | Purpose                                                                                                            |
| ------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `GET …/results`     | Score + percentage + per-task traces + tracking log — see [Benchmark Results](/api-reference/v1/benchmark-results) |
| `GET …/status`      | Lightweight progress poll                                                                                          |
| `GET …/tasks`       | Per-instance task results                                                                                          |
| `GET …/logs`        | Log snapshot (live or persisted)                                                                                   |
| `GET …/logs/stream` | SSE log stream (live, or DB replay after the run ends)                                                             |
| `POST …/cancel`     | Cancel an in-flight run                                                                                            |

See [Benchmark Logs & Status](/api-reference/v1/benchmark-logs) for the status/logs/stream details.

## Error Codes

| Status | Description                                               |
| ------ | --------------------------------------------------------- |
| 200    | Run queued                                                |
| 400    | Missing/invalid `benchmark`, invalid `agent`, or bad JSON |
| 401    | Invalid or missing API key                                |
| 403    | Pro subscription required                                 |
| 429    | Too many concurrent benchmark runs                        |
| 500    | Failed to launch the run                                  |

<CardGroup cols={2}>
  <Card title="Benchmark Results" icon="chart-simple" href="/api-reference/v1/benchmark-results">
    Score, percentage, per-task traces, and the tracking log.
  </Card>

  <Card title="Benchmark Logs & Status" icon="signal-stream" href="/api-reference/v1/benchmark-logs">
    Poll status and stream logs (live, then from the DB).
  </Card>

  <Card title="Agent Runtimes" icon="robot" href="/api-reference/v1/agent-runtimes">
    How `model` / `agent` choose Claude vs Codex vs Grok.
  </Card>

  <Card title="Models" icon="layer-group" href="/api-reference/v1/models">
    Model ids and their runtime mapping.
  </Card>
</CardGroup>