--- order: 1 subtitle: Public HTTP API. Resources, auth, conventions, errors. --- # Benchwright API Base URL: `{{site}}/v1/`. Every request and response is JSON, UTF-8, `Content-Type: application/json` (or `application/problem+json` for errors, or `text/event-stream` for SSE streams). --- ## 1. Versioning and headers Version lives in the path: `/v1/...`. New fields may be added to responses without a version bump. Removing or renaming fields, changing semantics, or tightening validation requires `/v2`. Clients should always send: ``` User-Agent: / () Accept: application/json Accept-Encoding: gzip, br ``` Responses include: ``` X-Benchwright-Request-Id: req_01HZ… X-Benchwright-Api-Version: 1 ``` **Treat API responses as uncacheable; the server mostly does not say so.** No `Cache-Control` header is emitted on ordinary `/v1` responses at all, so an intermediary applying heuristic freshness could serve you a stale run. The two places it is stated explicitly are registry reads (`Cache-Control: no-store`) and SSE streams (`Cache-Control: no-cache`). Use the ETags in §3.8 for revalidation rather than relying on cache headers. ## 2. Authentication and scopes ### 2.0 Two prefixes: `/v1` and `/api` There are two HTTP surfaces and they authenticate differently. Knowing which one you are on explains most unexpected 401s. | Prefix | Who it is for | How you authenticate | Stability | |---|---|---|---| | `/v1/...` | programs: this API, the CLI, the SDKs | `Authorization: Bearer `, scoped | versioned, and covered by the deprecation policy in §17 | | `/api/...` | the web app in your browser | your signed-in session cookie | internal, may change without notice | **`/api` is not a public API.** It backs the pages you see when signed in, so it expects a session and will reject a token. Calling `/api/...` with a PAT returns `401`, and that is the endpoint working as intended rather than a fault. Every `/api` route that is meant for programs has a `/v1` equivalent, and those are the ones to use. For example, read a run's routing at `GET /v1/runs/{id}/egress-routes` rather than `/api/runs/{id}/egress-routes`. The reverse also holds. A few operations require a browser session and refuse PATs on purpose, token creation among them, for the reason in §2.4. ### 2.1 Personal access tokens (PATs) A user provisions tokens under `/manage` (API Keys) or `POST /v1/tokens` (API; browser session required, see §2.4). Tokens look like: ``` bw_live_pat_8f7a9…e3 ``` The token is shown once at creation. Subsequent reads surface only the prefix + last 4, a label, scopes, and last-used timestamp. Auth header: ``` Authorization: Bearer bw_live_pat_8f7a9…e3 ``` Storage: the server keeps only `SHA-256(token)` — a DB dump cannot be replayed. `last_used_at` is bumped on every authenticated request so the web UI can surface dormant tokens for cleanup. Revoked tokens are soft-deleted (`revoked_at` stamped) so audit queries retain "this token was live between X and Y". Rotate produces a fresh value under the same id; the previous hash is overwritten atomically. For CI, provision a PAT under a shared service user account. ### 2.2 Scopes Tokens declare the minimum privilege they need. `benchwright token create` defaults `--scope` to `runs:read,runs:write,artifacts:read,secrets:read,billing:read,registry:read` — `registry:read` is included so the default token can browse the catalog (`bench ls` and friends) under the §8 access rules. (`benchwright login` mints nothing; it stores a PAT you paste in, with whatever scopes that token already has.) `POST /v1/tokens` validates scopes against the known list and rejects wildcard grants (e.g. `runs:*`) with `400 invalid_argument`; a token that already carries a wildcard is still honored at enforcement time, but wildcards cannot be minted through the API. | Scope | Lets the holder… | |----------------------|-----------------------------------------------------| | `runs:read` | List and inspect runs, events, tasks | | `runs:write` | Launch, fork, halt, pause/resume, update visibility | | `artifacts:read` | Download report files, atomize code, results | | `secrets:read` | List secret names + rules (NEVER the value); read host policy + blocked-host log | | `secrets:write` | Create, rotate, delete secrets and rules | | `host_policy:write` | Add/remove allowed and blocked hosts | | `billing:read` | View balance, history, usage trend, export, statement | | `billing:write` | Create top-ups, manage the subscription, change auto-refill + monthly cap | | `registry:read` | Browse the public benchmark catalog | | `registry:publish` | Flip visibility on your own runs + benchmarks | | `webhooks:write` | Reserved for the upcoming webhooks API | | `operators:read` | List operators + read conversation turns / pending | | `operators:write` | Create, start and shut down operators; send a chat message; stop a turn; clear pending | Registry reads (§8.1–§8.3) work **unauthenticated** — the public catalog is open to anyone, and a browser (JWT) session is full-access as usual. `registry:read` only gates **PATs**: a token that authenticates without it is rejected (`403 scope_insufficient`) even for public rows, so a CI token scoped to runs can't quietly enumerate the catalog. Anonymous access also extends to the per-run read surface of **public and unlisted runs**: run detail, events (poll and SSE), tasks, task payloads, artifacts and their content, fingerprint, narrate, and report all serve without credentials when the run's visibility allows it. A `403 scope_insufficient` response carries a structured `required_scope` field naming the exact scope the call needs, so a client can render an actionable "mint a token with this scope" hint without parsing `detail`. ### 2.3 Provider credentials Keys needed by the pipeline itself (OpenRouter, OpenAI, Anthropic, Google, or arbitrary HTTP services the Driver reaches) live in the Secrets API (§9), not in the PAT. Rotating your OpenRouter key does not invalidate your PAT and vice versa. ### 2.4 Token lifecycle ``` GET /v1/tokens/me # what am I? identity + the scopes I hold GET /v1/tokens # list the caller's PATs POST /v1/tokens # create (returns `token` once) PATCH /v1/tokens/{id} # set or clear a per-token spend cap GET /v1/tokens/{id}/spending # per-token spend rollup POST /v1/tokens/{id}:rotate # mint a new value, keep id+label+scopes DELETE /v1/tokens/{id} # revoke (soft-delete, audit-preserving) ``` `GET /v1/tokens/me` answers "who is this credential, and what may it do": auth type, user id, email, role, whether the caller is full-access, and the scope list. It is the only way a PAT can enumerate its **own** scopes (the list endpoint shows tokens, and a token cannot see itself listed with certainty), so it is the right preflight for a client that wants to fail with "this token is missing `runs:write`" instead of a 403 halfway through a workflow. Token **mutation** (create, spend-cap patch, rotate, delete) is browser-session-only: a PAT calling any of these gets `403 scope_insufficient`, with the explanation in `title` and the bare operation name in `detail`. `GET /v1/tokens`, `GET /v1/tokens/me` and `GET /v1/tokens/{id}/spending` work with a PAT. **This limit is deliberate, and it should stay.** A token that could mint tokens would let anyone who obtained one quietly issue themselves a second, longer-lived, wider-scoped credential, and revoking the original would not take that away. Signing in is the one step that proves a person is present, so it is the step that gates minting. The reason worth writing down is what makes the limit affordable. Recovering from trouble must never require minting a token, because the moment it does, this boundary is the thing standing between you and your own account. Every recovery path is therefore reachable with an ordinary PAT: `operators:write` creates, starts, and shuts down operators ([§12a](#12a-resource-operators)), `runs:write` halts runs. If you ever find yourself needing a new token purely to get out of a hole, that is a missing endpoint rather than a reason to loosen this. `GET /v1/tokens` returns metadata only — the secret is never echoed on list responses. The full secret is returned exactly once, in the `token` field of the create response and the rotate response. ```json GET /v1/tokens { "data": [ { "id": "", "label": "ci", "scopes": ["runs:read", "runs:write", "artifacts:read"], "prefix": "bw_live_pat_8f7a", "last4": "3c91", "created_at": "2026-02-14T…Z", "last_used_at": "2026-04-23T14:00:04Z", "spend_cap_cents": 5000, "cap_period": "month", "spent_cents_period": 312, "spend_period": "month", "expires_at": "2026-12-31T00:00:00Z", "expired": false } ], "meta": { "payer": { /* whose wallet this caller bills to, §2.5 */ } } } ``` Token ids are bare UUIDs. Revoked tokens are filtered out of the list entirely (there is no `revoked_at` field on responses). **Spend caps.** `POST /v1/tokens` and `PATCH /v1/tokens/{id}` both take `{"spend_cap_cents": 5000, "cap_period": "month"}`, which caps what runs launched with that token can spend. Set it at creation when you can: a cap added afterwards leaves a window the token spends without one. `cap_period` is one of `day`, `month`, `lifetime` (default `month`); on `PATCH`, `{"spend_cap_cents": null}` clears both the cap and its period. `GET /v1/tokens/{id}/spending?period=month` returns `{token_id, period, spent_cents, spend_cap_cents, cap_period}` — the same number the list response carries inline as `spent_cents_period`. **Expiry.** `PATCH /v1/tokens/{id}` with `{"expires_at": "2026-12-31"}` gives a token a deadline, and `POST /v1/tokens` accepts the same field at creation. A bare date means the END of that day UTC, so a token you set to expire today still works for the rest of today; a full ISO-8601 timestamp is taken literally. `{"expires_at": null}` clears the deadline. The value must be in the future: to retire a token now, revoke it. Expiry is checked when the token authenticates, not by a background sweep, so it takes effect on the very next request. An expired token keeps its row, so it stays visible in `GET /v1/tokens` with `expired: true` rather than vanishing the way a revoked one does. That distinction is the point: "this lapsed on Tuesday" and "somebody revoked this" are different incidents. A request carrying an expired token gets its own error rather than the generic one, because the fix is different (the value is fine, the date is not) and `auth_required` would send you to re-check your copy-paste: ```json 401 { "code": "token_expired", "title": "API token expired", "detail": "This token expired on 2026-08-11. It has not been revoked, so …", "expired_at": "2026-08-11T00:00:00+00:00" } ``` This short-circuits: presenting an expired token fails even if the request also carries a session cookie that would have worked. You named a credential, so we answer about that credential rather than quietly serving you under another one. A string that was never a token of ours is not a credential and still falls through to whatever else you have. Rotation is for compromise recovery: it overwrites the stored hash in place, so the previous value stops authenticating immediately. It replaces the *secret*, not the *grant* — scopes, spend cap and expiry all ride through untouched, so rotating an expired token returns a value that still does not authenticate (the response says so with `expired: true`). Revoke (`DELETE`) soft-deletes the row for audit; the next authenticated request with that token gets `401 auth_required`. ### 2.5 Teams A team is a shared workspace: members see each other's runs in `GET /v1/runs` (team-scope widening), and usage bills to the team owner's wallet. Billing-, token-, and secret-shaped responses carry a `meta.payer` object naming whose wallet the call is scoped to. A member who may read the team's resources but not change them gets `403 team_readonly` on a write. That is the general guard, and the code to handle. (`403 forbidden_team_member` exists at exactly one endpoint, the billing-controls PATCH, where only the owner may set the cap and refill.) `409 already_on_team` comes from `POST /v1/teams`: you cannot create a team while you already belong to one. An invite you are not allowed to send or accept is `403 forbidden`. ``` GET /v1/teams/me # my team: team, members, invites, balance, is_owner POST /v1/teams # create a team (caller becomes owner) POST /v1/teams/me/invites # invite by email to my team POST /v1/teams/{id}/invites # invite by email to a specific team POST /v1/team-invites/{id}:accept # accept a pending invite POST /v1/team-invites/{id}:revoke # revoke a pending invite POST /v1/teams/me:leave # member leaves (owners cannot leave) POST /v1/teams/me:delete # owner dissolves the team ``` `GET /v1/teams/me` returns `{team, members, invites, balance, is_owner}`; `team` is `null` when the caller has no team. Owners cannot `:leave` (they are the payer); `me:delete` dissolves the workspace: all members are removed, pending invites are cancelled, and shared runs revert to being private to whoever launched them. ## 3. Conventions ### 3.1 Identifiers Every resource id is a bare string. The route segment names the resource kind, so the id itself doesn't have to. | Resource | Shape | Example | |--------------------|----------------------------------------------------|------------------------------------------| | Run / trace | UUID | `608721fe-54f1-43e3-ab9b-7e0269783f70` | | Benchmark | `bm_` + UUID (or `bm_` + slug, on projected rows) | `bm_c3f2a8b0-…`, `bm_mmlu_pro` | | Subset | `sub_:<12 hex>` | `sub_bm_c3f2a8b0-…:7281bbc31736` | | Impl | `imp_:<12 hex>` | `imp_bm_c3f2a8b0-…:c8a1b2d3e4f5` | | Task (per run) | base64url(`::`) | `NjA4Nzlu…` | | Event | base64url(`:`) | `NjA4Nzlu…` | | Artifact | base64url(`:`) | `NjA4Nzlu…` | | Secret | UUID | … | | Secret rule | UUID | … | | Transaction (ledger row) | `txn_` + UUID | `txn_608721fe-…` | | Top-up | UUID | … | | Bug report | UUID | … | Subset and impl ids are **content-addressed**, and the trailing 12 hex characters are the first 12 of the content hash itself: of the task set for a subset, of the code for an impl. The `fingerprint` field on an impl repeats those same 12 characters, and the full hash is carried separately as `impl_fingerprint`. So two ids that share a prefix and differ in the hash are two versions of the same thing under one benchmark, which is what makes them comparable. Task / event / artifact ids encode the natural key as base64url so they round-trip through URLs and the server can decode them back into `(run, seq)` or `(run, invocation, index)` without a separate lookup table. UUID-shaped ids are validated for shape; bad shapes 400. ### 3.2 Timestamps RFC 3339 strings in UTC: ```json "started_at": "2026-04-23T14:02:11.371738+00:00" ``` **Parse them as RFC 3339, don't string-match on `Z`.** The UTC offset is normally written `+00:00` rather than `Z`, and the fractional part is typically microseconds (6 digits) rather than milliseconds. Both spellings are valid RFC 3339 and both may appear, so anything that slices the string or compares suffixes will break on one of them. Duration fields are `_ms` integers (`duration_ms`). Never floats. ### 3.3 Money Cents as integers, currency as ISO-4217, and (on run costs) an exact `amount_usd` beside them: ```json "cost": { "amount_cents": 4137, "currency": "USD", "amount_usd": 41.3719 } ``` `amount_cents` is a whole-cent integer and never mixes units. It is also lossy in one direction on purpose: **any non-zero cost rounds up to at least 1¢**, so real spend never reports as free. That makes it the wrong field to reconcile with, because a sub-cent figure is indistinguishable from a nearly-1¢ one once it is rounded. **`amount_usd` is the exact figure.** It is a float carrying the underlying fractional-cent value, and it is what to reconcile a run against its `billing_history` rows. Every cost entry on a run (§4.1) carries it. Money objects on the billing endpoints (§10) are whole cents only, rounded half-up. ### 3.4 Pagination Run, event, task, and artifact lists use cursor pagination: ``` GET /v1/runs?limit=50&page_token= ``` Response: ```json { "data": [ /* up to 50 items */ ], "pagination": { "has_more": true, "next_page_token": "" } } ``` `limit` defaults to **50** and caps at **100** on these lists (the event stream is the exception, §5.1: 500 and 1000). A larger value is **silently clamped**, not rejected, so a client asking for 500 gets 100 and a `next_page_token`. Follow the cursor rather than trusting that you asked for everything. Registry lists (§8) clamp at 200. `next_page_token` is opaque. Treat it as a blob to echo back, not as something to construct or parse: what it encodes (a sequence position, or a timestamp plus an id, depending on the list's ordering) is an implementation detail that can change without a version bump. Cursor-paginated lists never report a total. Offset-paginated lists — `GET /v1/billing/transactions` — take `limit` / `offset` instead and include a best-effort `total_estimate` (PostgREST `count=estimated`); clients must not rely on it for correctness. Registry lists (§8) also use `limit`/`offset` where they paginate at all. ### 3.5 Filtering and sorting Operators on endpoints that declare them: ``` ?status=running # exact match ?status=in:running,ok # set membership ?started_at=gte:2026-04-01 # comparison ?sort=-started_at,cost_billed # comma list, `-` reverses ``` The comparison prefixes are `gte:`, `lte:`, `gt:` and `lt:`; `like:` and `ilike:` do pattern matching (case-sensitive and case-insensitive). A value with no recognized prefix is an exact match. The set of filterable and sortable fields is documented per endpoint. Unknown filter keys are silently ignored, not rejected. Text search exists only where an endpoint declares it — `?q=` on the registry catalog (§8.1); there is no general `search` operator. ### 3.6 Partial responses and expansion `?fields=` (partial responses) is **not yet available**; responses always return the full shape. `?expand=` exists on exactly two endpoints: ``` GET /v1/runs/?expand=phase_breakdown,child_forks GET /v1/billing/balance?expand=topups,usage_trend,counters,tiers,subscription ``` An unknown `?expand=` key on `GET /v1/billing/balance` is rejected with `400 unknown_filter_field`. Expansion never recurses more than one level. ### 3.7 Idempotency Any `POST`, `PATCH`, `PUT`, or `DELETE` that creates or mutates a resource accepts: ``` Idempotency-Key: 4f1c7a1a-3b2c-4f6b-9b8a-123456789abc ``` The server caches the response status + body for 24 hours, keyed by `(token_id, key)`. Retrying with the same key returns the original result with an `Idempotent-Replayed: true` header on the reply. Conflicting retries (same key, different request body or path) get `409 idempotency_conflict`. A retry that arrives while the first request is still in flight also 409s rather than risk running the handler twice. **Launch caveat (`POST /v1/runs`):** a successful acceptance (`202`) is cached even when the run later dies at boot (`status=error`). A bare retry with the same key therefore returns that failed run's id with `Idempotent-Replayed: true` and does **not** launch a new run — which is dangerous right after an infra fix. The `benchwright` CLI heals this: on replay it GETs the run and, if the run is missing or terminal-failed (`error` / `failed` / `cancelled`), relaunches once under a derived key and prints a stderr note. Raw HTTP clients should do the same, or pass a fresh `Idempotency-Key` / omit the header after a failed attempt. Still- `running` or `ok` replays stay as intentional dedupe. Keys longer than 256 chars are rejected with `400 invalid_argument`. JWT (browser) callers share the cache keyed by `user_id` instead of `token_id` — rotating a PAT invalidates any outstanding keys tied to it. ### 3.8 Concurrency tokens `GET` on mutable resources returns a weak ETag: ``` ETag: W/":1761234800123" ``` The digits after the colon are the row's `updated_at` in epoch milliseconds. ETags are weak (`W/` prefix) because response JSON isn't byte-stable — two GETs match semantically as long as the row hasn't moved. `PATCH` accepts `If-Match` to gate the write on the row still being at the expected revision: ``` If-Match: W/":1761234800123" ``` On mismatch the server responds `412 precondition_failed` and sets the response `ETag` to the current value so the client can re-fetch and retry. `If-Match` is optional — omit it to opt out of the concurrency check. `If-Match: *` also bypasses the check and matches any current row. Enforced today on `/v1/runs/{id}` (PATCH), `/v1/benchmarks/{id}` (PATCH), and both build-subset PATCHes (`/v1/build/subsets/{id}/recipe` and `/tasks`, §8.9), which also serve the matching ETag on their GET. Other mutable resources (tokens, secrets, host policies, catalog subsets, impls) accept `If-Match` permissively: the header is read and ignored, so a stale write there succeeds rather than answering `412`. They will begin enforcing as their handlers migrate; clients that always send the ETag they last saw will not regress. ### 3.9 Errors All 4xx/5xx responses follow RFC 9457 (problem+json): ```json { "type": "https://docs.benchwright.ai/errors/insufficient_balance", "title": "Balance below the $1.00 launch floor", "status": 402, "code": "insufficient_balance", "detail": "Top up to at least $1.00 before launching. The pipeline halts a run the moment a live debit would drop balance under $1.", "request_id": "req_01HZ…", "retryable": false } ``` Clients should key off `code`, not `detail`. Two error codes add structured fields beside the standard ones: `403 scope_insufficient` carries `required_scope`, and `409 delete_blocked` carries `id`, `deleted`, `halt_reason` and `sandboxes_killed` (§4.7). Exceptions, where the body is a plain `{"error": "…"}` rather than problem+json: the billing endpoints that delegate to the legacy billing handlers (top-ups, subscription, Stripe webhook), and the whole `/v1/operators/*` surface, which bridges to the same handlers the web app uses. Read `error` as the message on those, and see §14 for what a bare `500` looks like. ### 3.10 Null vs. missing `null` means "server knows this field, value is absent". Missing keys mean "not applicable in this response shape". Clients that deserialize into typed structs treat both as `None`. --- ## 4. Resource: Runs A run is the top-level unit of work: one pipeline execution from an initial request to a final report. Runs are the most important resource in the API; everything else is either a child of a run or a setting that configures future runs. ### 4.1 The Run object ```json { "id": "608721fe-54f1-43e3-ab9b-7e0269783f70", "object": "run", "user_id": "", "status": "running", "display_status": "running", "current_phase": "materialize", "halted": false, "halt_reason": null, "request": "Run MMLU Abstract Algebra on qwen3-32b via OpenRouter", "driver": "openrouter://deepseek/deepseek-v4-flash", "benchmark": null, "benchmark_name": "MMLU", "benchmark_id": "", "subset": null, "subset_id": "", "impl": null, "impl_id": "", "model": { "identifier": "qwen/qwen3-32b", "source": "openrouter", "provenance": "specified" }, "sut_config": { "temperature": 0.0, "max_tokens": 2048 }, "backend": "sandbox0", "harness": "benchwright", "harness_config": null, "fingerprint_hash": "c8a1b2d3e4f5", "score": null, "task_count": 100, "cache_hits": { "count": 2, "phases": { "fetch": "", "analyze": "" } }, "materialize_indices": "0-99", "canonical_indices": null, "cost": { "incurred": { "amount_cents": 412, "currency": "USD", "amount_usd": 4.1174 }, "driver": { "amount_cents": 291, "currency": "USD", "amount_usd": 2.9083 }, "model": { "amount_cents": 38, "currency": "USD", "amount_usd": 0.3791 }, "sandbox": { "amount_cents": 121, "currency": "USD", "amount_usd": 1.2062 }, "billed": { "amount_cents": 495, "currency": "USD", "amount_usd": 4.9412 }, "proxy_linger": { "amount_cents": 1, "currency": "USD", "amount_usd": 0.0027 }, "narration": { "amount_cents": 3, "currency": "USD", "amount_usd": 0.0284, "calls": 7, "input_tokens": 42000, "output_tokens": 180 } }, "tokens": { "input": 38211, "output": 12402, "total": 50613, "driver": { "input": 4712000, "output": 12000, "total": 4724000 } }, "duration_ms": 421003, "visibility": "private", "forked_from": null, "forked_from_deleted": null, "fork_description": null, "relaunch_count": 0, "relaunch_budget_sec": null, "port_version": "0.5.15", "storage": { "event_count": 1204, "event_bytes": 882133, "payload_count": 100, "payload_bytes": 4118220, "total_bytes": 5000353 }, "created_at": "2026-04-23T14:00:02.003Z", "started_at": "2026-04-23T14:00:04.118Z", "finished_at": null, "updated_at": "2026-04-23T14:07:05.889Z", "error": null, "error_type": null, "review_resolution": null, "inference": { "calls": 100, "ok": 98, "completion_tokens": 50000, "duration_ms": 120000, "by_model": [ { "model": "qwen/qwen3-32b", "calls": 100, "ok": 98, "completion_tokens": 50000, "duration_ms": 120000, "status_codes": { "200": 98, "429": 2 } } ] }, "links": { "self": "/v1/runs/", "events": "/v1/runs//events", "artifacts": "/v1/runs//artifacts", "tasks": "/v1/runs//tasks", "report": "/v1/runs//report", "dashboard": "https://benchwright.ai/run/" } } ``` Every run has the same shape whether it's `running`, `ok`, or `error`. Costs are live: polling a running row shows the debit counter climbing in cents without any JSON reshaping. `cost.model` is the SUT's own inference spend, distinct from `cost.driver`, the Driver LLM's spend (§8 of [concepts](./concepts.md#8-driver-vs-sut-vs-operator)). **`driver` and `cost.driver` are the only names for the Driver's model id and the Driver's spend.** Both values previously shipped under `operator` / `cost.operator`, and before that under `agent` / `cost.agent`. None of the older keys are emitted any more; a client reading them gets an absent field, not a zero. See [§17](#17-deprecation-and-stability) for the removal itself, which was a breaking change made deliberately. The `operator` spelling predated the split that gave Operator its current meaning (the goal-level agent that launches runs, §12a). Once that split landed, the key named a different actor than the one whose identity and spend it carried, which made every reading of a cost breakdown a small guess about which agent was meant. It was removed rather than kept, because a wrong name that still works is read as correct. Every cost entry carries an exact `amount_usd` beside the whole-cent `amount_cents`. **Reconcile against `amount_usd`.** `amount_cents` rounds any non-zero cost *up* to at least 1¢, so a run that spent $0.0007 reports `1`, and no consumer reading cents can tell that apart from $0.0099 (§3.3). **`cost.billed` is what you are charged. `cost.incurred` is not.** `incurred` is everything the run consumed *including* `cost.model`, and the SUT's spend is deliberately never charged here: on a cloud-inference run it is metered on the minted OpenRouter key's own stream, and on a BYOK run the user's provider already charged it. So the two legitimately differ, often by an order of magnitude, and the gap between them is `cost.model` by design rather than a billing discrepancy. This field was called `total` until it had misled readers in both directions — quoted as the price of a run, and read as a platform *subtotal* that `cost.model` then got added back onto, double-counting the SUT. Neither reader was careless; `total` is simply the key you reach for when you want to know what something cost. `incurred` is named to make you ask. **`cost.proxy_linger` is not part of `cost.incurred`.** It meters the LiteLLM proxy window kept alive *after* the run's own ledger closed (`metadata.proxy_linger_seconds`), so it is accrued by a different writer and is always reported separately. A client that sums the entries under `cost` to get an all-in figure must add it explicitly; one that reads `cost.incurred` alone will under-report a run that lingered. `cache_hits` is an **object**, not a count: `{"count": N, "phases": {…}}`, where `phases` maps each reused phase to the id of the run its output came from. Use `cache_hits.count` where you want the number. `storage` is always present, with every counter defaulting to `0` rather than the block being `null`. `relaunch_budget_sec` is `null` when unset, never `0`, because `0` would mean "no time allowed", which is a different claim. `tokens` mirrors that same split, and the two halves are **separate streams — don't read one as the run total**. `tokens.input` / `tokens.output` / `tokens.total` count the **model under test**: its own inference, metered off the run's proxy. `tokens.driver` counts the **Driver** (the LLM that executes the run). A driver-heavy run can bill orders of magnitude more driver tokens than model tokens, so a client showing a single "tokens" figure should say which one it means, or add them explicitly. `model` is the run's frozen model pin. `model.identifier` is the model string (e.g. `qwen/qwen3-32b`); `model.source` is the provider id (e.g. `openrouter`, `together`); `model.provenance` records how the pin was chosen — `"specified"` when the launch body carried an explicit `model`, other values when the pipeline resolved it from the request text or account defaults. `sut_config` is the sampling configuration the harness actually applied to the model under test (temperature, seed, max tokens, …), captured once per run. `{}` means provider defaults with no overrides; `null` means it was not captured. `benchmark` is `null` on list rows; the flat `benchmark_name` is always present, so key off `benchmark_name` / `benchmark_id`. The nested `subset` and `impl` objects are always `null` today — use `subset_id` / `impl_id`. **`benchmark_id`, `subset_id` and `impl_id` are absent, not null, until the run finalizes.** They come from the catalog projection, which is written when the run reaches a terminal status, so an in-flight run simply has no such keys. Per §3.10 a typed client sees `None` either way, but code that checks for key *presence* must not read the absence as "this run has no benchmark". `forked_from_deleted` names the run that was removed when this one was adopted out of a deleted parent (§4.7); it is `null` on runs that never lost a parent. Launch `metadata` is **not** part of the Run object: it is behavioral launch configuration only (§4.2.1), and with the single exception of `publish_on_complete` it is neither persisted nor echoed back. `backend` records which compute backend ran the sandboxes for this run: `"e2b"` (E2B managed microVMs) or `"sandbox0"` (Kubernetes pods, either ours or your own). Set by the launcher from the `compute` request field (or the account default, which is sandbox0 unless you configured your own provider) and is immutable after launch. #### 4.1.1 `status`, `display_status`, and `halt_reason` `status` is the pipeline's own state (`running`, `ok`, `error`). `display_status` is what a UI should show: it folds `halted` and `halt_reason` back into the status so that "stopped on purpose" does not render as a failure. | `display_status` | When | |---|---| | `running` / `ok` / `error` | pass-through, nothing halted | | `killed` | halted by an operator kill, whether the run ended or is still winding down | | `waiting` | halted mid-run and waiting on a person or a retry decision | | `paused` | halted mid-run by an operator pause, resumable | | `halted` | halted for any other reason (a ceiling was hit, balance ran out) | A finished run that was killed reads `killed` rather than `error`, so gate CI on `status` and label the UI from `display_status`. `halt_reason` says *why*. **These are prefixed strings, not bare enum values**, and several carry a payload after the prefix, so match them with a prefix test and never with equality: | Prefix | Meaning | |---|---| | `agent_review: ` | the Driver asked a human a question and is waiting (§4.8 `review_resolution`) | | `limit_exceeded:.` | a `limits` ceiling reached, e.g. `limit_exceeded:run.cost` | | `max_cost_exceeded` | pre-`limits` spelling of `limit_exceeded:run.cost` | | `max_model_cost_exceeded` | pre-`limits` spelling of `limit_exceeded:run.model_cost` | | `max_duration_exceeded` | pre-`limits` spelling of `limit_exceeded:run.duration` | | `restart_cap_exceeded: ` | a phase asked to restart once too often | | `operator_kill by [: ]` | `POST /halt` or `DELETE` (§4.6, §4.7) | | `operator_pause: ` | `POST /pause` (§4.11) | | `insufficient_balance` | balance fell below the launch floor mid-run | `inference` is LiteLLM proxy telemetry, present on the detail response only (not on list rows): totals for `calls`, `ok`, `completion_tokens`, and `duration_ms`, plus a `by_model` array with the same counters per model id. Used to compute tok/s, mean latency, and success rate. `null` on older runs. ### 4.2 POST /v1/runs: launch a run Runs have three launch modes, distinguished by which top-level field the body carries. Exactly one must be present. | Mode | Field | Phases executed | Typical use | |-------------------|--------------|-------------------------------------------------------|--------------------------------------| | Natural-language | `request` | Find → Analyze → Setup → Atomize → Materialize → Report | exploratory, or no catalog entry yet | | Pinned benchmark | `benchmark` | as natural-language, seeded from the catalog entry | rerun a known catalog benchmark | | Replay | `replay.from_run` | Attach → Materialize → Report | noise check, model A/B, regression | All modes share `model`, `constraints`, and `metadata`. Idempotency comes from the `Idempotency-Key` **header** (§3.7); there is no `idempotency_key` body field. **Pinned mode** takes a catalog benchmark id or exact name alone — `{ "benchmark": "", "model": {…} }`. Do **not** also send `request` (that is a second launch mode and returns `400`). Optional `subset` / `impl` further pin a split or scorer when you have them; they are not required. The launcher resolves the catalog row, writes a synthetic `request` (`"Run "`), and — for benchwright-built (product) benchmarks — seeds `metadata.product_subset_id` so ProductBridge runs the stored recipe. Unknown id → `404 benchmark_not_found`. Wire prefix `bm_` is accepted and stripped. #### 4.2.1 Natural-language launch The pipeline decides everything from a description string. ```json POST /v1/runs Authorization: Bearer bw_live_pat_… Idempotency-Key: 4f1c7a1a-3b2c-4f6b-9b8a-123456789abc Content-Type: application/json { "request": "Run MMLU Abstract Algebra on openai/gpt-4o-mini via OpenRouter", "driver": "openrouter://deepseek/deepseek-v4-flash", "model": { "model": "openai/gpt-4o-mini", "provider": "openrouter" }, "compute": "e2b", "constraints": { "max_cost_cents": 2000, "max_duration_ms": 900000, "max_tasks": 100 }, "trials": { "plan": { "k": 3 }, "aggregator": { "name": "mean" } }, "metadata": { "autonomous": true, "limit": 25 } } ``` - `model` replaces the natural-language model parsing the Find phase does. - `driver` pins the **Driver**, the LLM that *drives* the pipeline (Find, Analyze, Atomize and friends), as distinct from `model`, which is the model being evaluated. **`driver` is the only key read for this.** `operator` and `agent` were the previous names and are no longer accepted: a body sending either pins no Driver at all and the run uses the default below, so a script still on an old key launches on a model it did not choose. `operator` was the more misleading of the two, since it set the Driver while naming the Operator, which is a different actor (§12a). Omit `driver` entirely and the run uses your configured Operator's own model, falling back to the platform default. An **explicit** pin must name a priced Gemini model or an `openrouter://` spec; anything else is refused with `403 admin_required`. A driver arrived at through your own operator settings is always allowed, since that one runs on your own credit. - `phase_models` (object) overrides the driver for individual phases, `{"analyze": "openrouter://…"}`. Each spec must satisfy the same rule as `driver`, with the same `403 admin_required` when it does not. - `compute` overrides which backend runs **this run's** sandboxes — `"e2b"` or `"sandbox0"` — without changing your saved default (Settings → Compute). Omit it (or send `"default"` / `"auto"`) to use your account default. An explicit `"sandbox0"` uses your configured sandbox0 endpoint even if it's currently *disabled* as your default, so a one-off run can target it. Unknown value → `400 invalid_argument`. The chosen backend is stamped on the run as `backend`. - `model.provider` pins the inference provider. Only `"openrouter"` and `"together"` are currently validated as pinnable providers; any other value returns `400 unknown_provider` before any sandbox spawns. - `hints.benchmark` is read only in combination with `metadata.canonical_indices` (below), to resolve canonical task identity against that benchmark; it does not skip the catalog search on its own. `hints.subset` is ignored. - `trials` runs each task multiple times: `plan.k` (1–20) is the trial count and `aggregator` names how per-trial scores fold into the row score — `any_pass` (default), `all_pass`, `mean`, or the parameterized `majority_on` / `weighted_mean` / `min_above`. An unrecognized aggregator name fails `400 unknown_aggregator`. - `limits` declares every ceiling, spelled `.`: ```json { "limits": { "run": { "cost": 5.0, "duration": 3600, "tokens": 2000000 }, "task": { "attempts": 1 }, "attempt": { "duration": 600 } } } ``` **Money is in whole dollars and time is in seconds.** The unit belongs to the quantity and never to the name, so there is no `_cents` or `_ms` suffix to get backwards. | Level | Quantity | Bounds | |---|---|---| | `run` | `cost` | total spend: sandbox + driver + model | | `run` | `model_cost` | SUT spend only — your own provider's bill | | `run` | `duration` | wall clock for the whole run | | `run` | `tokens` | prompt + completion tokens | | `run` | `tasks` | how many dataset rows run at all | | `task` | `cost` | model spend on one task, **across all its attempts** | | `task` | `tokens` | tokens on one task, across all its attempts | | `task` | `duration` | wall clock for one task | | `task` | `attempts` | tries before a task is failed | | `attempt` | `duration` | wall clock for ONE try of one task | | `phase` | `calls` | LLM calls one phase may spend | | `phase` | `duration` | wall clock for one phase | **`run` and `phase` ceilings stop the run. `task` ceilings stop only that task** — it is recorded as failed and the remaining tasks carry on, because a task over its own budget is a failed task, not a failed run. A task stopped this way is never retried: retrying would spend the ceiling again, up to `task.attempts` times. Task ceilings bound the whole retry chain, which is what makes them the useful shape — a per-attempt money cap still allows N attempts x the number you wrote down. `phase` is the only level that may name its target: `"phase:atomize": {"calls": 250}` overrides `"phase": {"calls": 100}` for that phase alone. Use it to give a build-heavy phase more room, not just less. Reaching a run-level ceiling halts the run with `halt_reason="limit_exceeded:."`. `run.cost` compares `cost.incurred` (`cost_total`), which includes model spend, so it fires on a BYOK run where `cost_billed` would barely move. A limit the platform does not enforce is **refused with 400**, never stored and ignored: a ceiling that silently does nothing is worse than no ceiling. `attempt.cost` is the current example — the proxy tags spend per task, so retries share one tally and an attempt has no cost of its own to measure. The pre-`limits` spellings still work and mean the same thing: `constraints.max_cost_cents` (cents), `constraints.max_duration_ms` (ms), `constraints.max_tasks`, `metadata.limit`, `metadata.task_attempts`, and `metadata.task_timeout_sec` — which maps to `attempt.duration`, because that is the level it always bounded. Halts are acted on between phases, so a single long-running phase — notably a harness bridge run (`inspect`/`lmeval`/`lighteval`/ `harbor`), where the whole eval is one phase — can overrun a ceiling until the phase ends. For a hard pre-launch bound on harness runs, pair a ceiling with `max_tasks` (sample count). - `metadata.autonomous: true` runs the pipeline **without the `requestHumanReview` tool** registered on any phase, so the run can never halt at `awaiting_review` — it succeeds or fails on its own (`requestPhaseRestart` and normal error paths still apply). For unattended runs — CI, catalog/smoke sweeps — where no operator is watching to resolve a review. Requires an explicit model pin (`400 autonomous_model_required` otherwise). - `metadata.limit: ` caps Materialize to the first **N** samples for **any** run — ad-hoc (Atomize→Materialize) or inspect-harness. The Driver is told the dataset total is N (so it submits indices `0..N-1`) and the `materializeStart` tool hard-clamps any submitted index `>= N`, so a "run everything" request can't exceed it. Used for cheap smoke runs and bounded ad-hoc runs. (The legacy `metadata.inspect_harness.limit` is still honored as a fallback for harness runs.) - `metadata.indices: ""` restricts Materialize to specific dataset indices, as an index-spec string (`"0-99"`, `"0,2,4"`). - `metadata.canonical_indices: ""` pairs with `hints.benchmark`: the given canonical benchmark positions are resolved to task identities so the run scores against the catalog's canonical tasks. - `metadata.proxy_linger_seconds: ` keeps the run's LiteLLM proxy sandbox alive for **N** seconds after the run finishes, instead of tearing it down at completion. Useful for inspecting the proxy post-run. Clamped to `[0, 86400]` (24h); `0` (default) tears it down immediately. CLI: `--proxy-linger-seconds N`. - `metadata.local_egress: true` routes the run's outbound traffic through a machine you control, and `metadata.egress_endpoint` names a specific one. See §9a. - `metadata.harbor_task: ""` pins a single Harbor task by name on a harbor-harness run. - `materialize.orchestrator.storage: "8Gi"` (a top-level field, not under `metadata`) asks for a work sandbox with more ephemeral disk for the orchestrator, the sandbox that downloads the dataset and drives the harness. Reach for it when a run dies on disk pressure with a large dataset. - `metadata` is otherwise behavioral launch configuration only — `limit`, `autonomous`, `indices`, `canonical_indices`, and the per-harness config objects. It is **not persisted**, not returned on the run resource or its events, and not filterable. To correlate runs to CI jobs, record the returned run id on the CI side. - The one exception is `metadata.publish_on_complete: true`, which **is** persisted on the run. It is an owner opt-in: set it at launch and the run flips to `visibility=public` the moment it finalizes `status=ok`, saving a publish call (§8.4) on a run you already intend to share. Only a successful run is flipped: one that errors or is halted stays private, so this cannot publish a failure you would not have published by hand. ##### Choosing the eval harness `harness` selects **how** the benchmark is evaluated. It's an enum — an unknown value is rejected with `400 unknown_harness` before any sandbox spawns. | `harness` | What runs the eval | |---------------|---------------------------------------------------------------------------| | `benchwright` | **Default.** Our own harness — the Driver authors the eval impl from scratch per task (Atomize → Materialize). No external runner. | | `inspect` | [`inspect_ai`](https://inspect.ai-safety-institute.org.uk/) via `inspect_evals`, pre-baked in the work-sandbox template. | | `lmeval` | EleutherAI `lm-evaluation-harness`. | | `lighteval` | Hugging Face `lighteval`. | | `harbor` | Harbor agentic terminal-bench harness (own orchestrator sandbox). | `scratch`, `native`, `autonomous`, `adhoc`, `agent`, `none` are accepted as synonyms for `benchwright`; `lm-eval` / `lm_eval` for `lmeval`. A **third-party** harness (`inspect` / `lmeval` / `lighteval` / `harbor`) takes its per-run config — the task id and its args — under `metadata._harness`: ```json { "request": "Run MMLU abstract_algebra on qwen3-32b via Inspect", "harness": "inspect", "model": { "model": "qwen/qwen3-32b", "provider": "openrouter" }, "metadata": { "inspect_harness": { "task": "inspect_evals/mmlu_0_shot", "args": { "subjects": ["abstract_algebra"] } } } } ``` An explicit third-party `harness` with no task is valid: the Find phase resolves the task id from the request text. `harness` itself is optional too — omit it and the harness is inferred from which `metadata._harness` object is present, defaulting to `benchwright`. > **Renamed from `bridge`.** This field and its `metadata._harness` > config keys were previously `bridge` / `metadata._bridge`. The old > names are rejected with `400 bridge_renamed_to_harness` (rather than > silently ignored) so a stale recipe fails loudly instead of running on > the wrong harness. > **Reasoning models + harness exact-match scorers.** A harness task > that caps generation for a non-CoT multiple-choice scorer (e.g. > `inspect_evals/mmlu_0_shot`, which applies `max_non_cot_tokens`) > truncates a *reasoning* SUT (qwen3, etc.) inside its thinking block, so > the final answer never reaches the scorer and the run reads near the > random floor. For reasoning SUTs prefer a CoT variant (`cot=True`), a > larger token budget, or the `benchwright` harness (which captures the > model's answer `content` directly). #### 4.2.2 Pinned benchmark launch Rerun a catalog entry without inventing a natural-language description. `benchmark` is the mode discriminator; `request` must be omitted. ```json POST /v1/runs { "benchmark": "example-landing-screenshot", "model": { "model": "qwen/qwen3-32b", "provider": "openrouter" }, "metadata": { "autonomous": true, "limit": 1 } } ``` | Key | Type | Meaning | |-------------|--------|---------| | `benchmark` | string | required. Catalog id (with or without `bm_`) or exact name. | | `subset` | string | optional. Product `build_subsets` id or classic subset id. | | `impl` | string | optional. Recorded on launch metadata; full classic-triple seed is still evolving — prefer `replay` when you need a prior impl byte-for-byte. | | `model` | object | same as natural-language (required when `metadata.autonomous`). | #### 4.2.3 Replay launch Re-materialize a prior run's subset + impl against a different model (or the same one, as a noise check). The pipeline rewinds to Materialize and inherits Find / Fetch / Analyze / Setup / Atomize from the parent. Attach is deliberately **not** inherited when Materialize needs the model proxy: a replay exists to point the same eval at a different model, so re-running Attach is what makes the new pin take effect instead of silently reusing the parent's proxy config. ```json POST /v1/runs { "replay": { "from_run": "" }, "model": { "model": "qwen/qwen3-235b-a22b-thinking", "provider": "openrouter" } } ``` Accepted keys under `replay`: | Key | Type | Meaning | |------------|--------|--------------------------------------------------------------| | `from_run` | string | required. Parent run id. Inherits benchmark + subset + impl. | **The replay does not necessarily run from the id you passed.** If that run is itself a fork, the server walks `forked_from` up to the first ancestor that is not one (up to 16 levels) and replays from *that* root, since the root is where the inherited phase output actually lives. So the new run's `forked_from` may name a different run than the one you sent. Read it back rather than assuming. Omitting `model` reuses the parent's model (noise check). Providing a new `model` is the cross-model A/B. The resulting run shares the parent's `impl_id`, so the registry can render a same-impl comparison without needing any extra correlation. Replay over a rolling subset pins to the parent's `sample_count` snapshot, not the subset's current size, so repeated replays produce stable comparisons even as the subset accretes new tasks. #### 4.2.4 Response Every launch mode returns the same `202 Accepted` shape: ```json { "id": "", "status": "launching", "mode": "replay", "sandbox_id": "", "links": { "self": "/v1/runs/", "events": "/v1/runs//events", "dashboard": "https://benchwright.ai/run/" } } ``` `sandbox_id` and `links.dashboard` are present when known at launch time. There is no cost estimate on the launch response. #### 4.2.5 Preflight errors Returned synchronously, no run created: - `400 invalid_argument`: multiple launch modes present, or none. Combining `request` with `benchmark` is rejected — pinned mode is `benchmark` alone. An empty description after mode checks names the three accepted shapes in the detail. - `404 benchmark_not_found`: pinned `benchmark` id/name is not in the catalog. - `404 subset_not_found`: pinned `subset` does not exist at all. - `400 invalid_argument`: pinned `subset` exists but belongs to a different benchmark than the one you pinned. The two cases are kept apart because they mean different things: a typo versus a mismatched pair. - `403 admin_required`: a non-admin caller explicitly pinned a non-Gemini driver, or sent a `phase_models` override naming one. - `409 replay_unsupported`: `replay.from_run` resolves to a parent that ran the `benchwright` harness. That harness authors its impl per run, so there is nothing fixed to re-materialize against. - `402 insufficient_balance`: balance < $1 launch floor. - `404 parent_not_found`: `replay.from_run` does not exist or is not accessible to the caller. - `400 parent_still_running`: replay/fork of a run that has not finished. - `403 forbidden`: fork of a run the caller does not own. - `409 idempotency_conflict`. - `400 no_provider_secret`: no provider API key exists in the user's vault at all. - `412 no_provider_secret`: a specific pinned provider's key is missing (the user has vault secrets, but none match the pinned model's required `provider`). Both variants carry `code` + `detail` only — add the key under `/manage/vault`. - `400 autonomous_model_required`: `metadata.autonomous` without an explicit model pin. - `400 unknown_aggregator`: unrecognized `trials.aggregator` name. - `429 tier_concurrency_exceeded`: the account is already at its tier's concurrent-run cap. - `501 e2b_not_configured` / `400 compute_not_configured`: the requested (or default) compute backend has no credentials configured on this deploy. ### 4.3 POST /v1/runs/{id}/fork Same shape as launch, but seeds from the parent run: ```json POST /v1/runs//fork Idempotency-Key: … { "description": "Retry Atomize with a stricter scoring function", "from_phase": "atomize", "model": { "model": "qwen/qwen3-32b", "provider": "openrouter" }, "indices": "0-49" } ``` **`description` is required** (`400` without it), and it is not just a label: with no `from_phase`, the server's ForkPhase reads it to choose the rewind point. Say what you want changed, not what the fork is called. `from_phase` is optional. `model` (object) overrides the parent's model pin; `indices` (index-spec string) restricts which dataset indices the fork materializes. Only your own runs can be forked (admins excepted); forking someone else's run fails `403 forbidden`. Forking a run of the `benchwright` harness is refused `409 fork_unsupported`. The parent preflight is inherited from launch: `404 parent_not_found` for an unknown or inaccessible parent, and `400 parent_still_running` for one that has not finished. ### 4.4 GET /v1/runs/{id} Returns the Run object. `?expand=phase_breakdown` adds: ```json { "phase_breakdown": [ { "phase": "find", "invocation": 1, "duration_ms": 8200, "tokens": { "input": 1200, "output": 340 }, "cost": { "llm_cents": 3 }, "llm_calls": 1, "model": "gemini-3-flash-preview" }, … ] } ``` `?expand=child_forks` adds the list of traces forked from this one (not recursive). ### 4.5 GET /v1/runs: list ``` GET /v1/runs ?status=in:running,error &benchmark=MMLU &visibility=private &started_at=gte:2026-04-01T00:00:00Z &sort=-started_at &limit=50 ``` Returns `{ data: [Run...], pagination }`. The filterable fields are exactly `status`, `benchmark` (matches `benchmark_name`), `visibility`, `started_at`, and `finished_at`; unknown filter keys are silently ignored (§3.5). Sortable keys: `started_at`, `finished_at`, `created_at`, `cost_total`, `cost_billed`. There are no `model.*` or `metadata.*` filters. Team members see teammates' runs in this list alongside their own (§2.5). Admins can pass `?scope=all` to list every user's runs. ### 4.6 POST /v1/runs/{id}/halt Stops the run. Halt is an immediate synchronous teardown (the same kill core as DELETE, §4.7), not a deferred graceful stop. ``` POST /v1/runs//halt { "reason": "superseded by build 8813" } ``` The body is optional, but a `reason` is **recorded**, not ignored: the run's `halt_reason` becomes `operator_kill by : `, with whitespace collapsed and the text truncated to 200 characters. Send one: six months later it is the only thing that distinguishes a deliberate stop from a failure. Returns the updated Run object with `halted=true`. For a non-destructive stop that preserves the work sandbox and can be resumed, use `POST /v1/runs/{id}/pause` (§4.11) instead. ### 4.7 DELETE /v1/runs/{id}: stop it, then destroy it DELETE means delete. It first issues the eager teardown path — every live sandbox tagged with this run is killed, then `status=error` + `halt_reason=operator_kill by ` — and then removes the run record itself. The event stream, the per-task payloads and the registry summary go with it. This cannot be undone. To stop a run and keep it readable, use `POST /v1/runs/{id}/halt` (§4.6). To keep a run but hide it, PATCH `visibility` to `private` (§4.8). Teardown finds those sandboxes by asking the compute backend which live sandboxes carry this run's id, rather than by replaying the event stream for `SandboxCreated` without a matching `SandboxKilled`. The event walk misses sandboxes that came from a snapshot-and-restore cycle, which do not always emit a kill for the handle they replaced, so it would leave real machines running while reporting a clean teardown. Returns `200 OK`: ```json { "id": "", "deleted": true, "halt_reason": "operator_kill by ", "sandboxes_killed": 2, "snapshots_freed": 1, "forks_adopted": [""], "events_adopted": 6, "reparented_to": "" } ``` **It deletes one run, never a chain.** A fork does not copy its parent's work, it inherits it by reference, so removing a run that others forked from would otherwise strand them. Instead the fork children are *adopted*: everything they were inheriting across the removed link — the latest phase output per phase, plus the fetch, atomize and model artifacts — is copied onto each child before the parent goes, and each child is re-pointed at the deleted run's own parent so the rest of its ancestry still resolves. A child's own event always wins over an inherited one, exactly as it did before the delete, so nothing a fork produced itself is overwritten. `forks_adopted` lists the runs kept this way (empty when there were none), `events_adopted` counts the events copied down across all of them, and `reparented_to` is the run they now point at (`null` if the deleted run was itself a root). On each adopted child, `forked_from_deleted` records the run that was removed, and every copied event carries `data._adopted_from` naming its origin — the provenance moves, it is not erased. **`deleted: false` never accompanies a 2xx.** If the record cannot be removed the response is `409` with `code: "delete_blocked"` and `retryable: false`; the run is halted but still present, and repeating the request will not change that. Treat any non-2xx as "the run is still there". That `409` carries structured fields beside the standard problem+json ones (`id`, `deleted`, `halt_reason` and `sandboxes_killed`), so a client can report what the attempt *did* accomplish (the run was stopped, its machines were released) without a second request. ### 4.8 PATCH /v1/runs/{id} Exactly two fields are patchable: ```json { "visibility": "public", "review_resolution": "Scoring was too strict, reran with expected=normalize(str)." } ``` `metadata` is not patchable. `visibility` takes `private`, `unlisted` or `public`. `unlisted` is a real state and not a synonym for either neighbour: an unlisted run is readable by anyone who has its id, including anonymously, but it is not projected into the public catalog. `visibility=public` requires only `status == "ok"`; otherwise the server returns `409 visibility_constraint_violated`. Making a run non-private also recomputes and restamps its `fingerprint_hash` from the run's stored artifact events, so a published run always carries a hash a reader can re-derive for themselves from `GET /v1/runs/{id}/fingerprint`. This route honours `If-Match` (§3.8) against the run's `updated_at`, which is worth sending here: publishing is the one run write where two clients racing can disagree about what state you meant to leave it in. Setting `review_resolution` has a side effect: it clears `halted`/`halt_reason` and wakes the driver sandbox, so a run parked at `awaiting_review` resumes with the resolution text as its instructions. ### 4.9 GET /v1/runs/{id}/narrate Live one-sentence narration of the run. Only generated while a dashboard has this endpoint open — if nobody's polling, no narration work runs and no narration cost accrues. Responses are cached per `(run, last_seq)` so concurrent viewers share one generation. ``` GET /v1/runs//narrate?slice=recent ``` - `slice=recent`: present tense, what the Driver is doing right now. - `slice=full`: past tense, what the run did end to end. Response: ```json { "sentence": "Rerunning Atomize after the score function threw on task 12.", "generated_at": "2026-04-23T14:07:05.889Z", "slice": "recent", "from_cache": false, "first_seq": 102, "last_seq": 158 } ``` `first_seq` / `last_seq` bound the event window the sentence was generated from. - `400 invalid_argument` for a `slice` that is neither `recent` nor `full`. - `204 No Content` when the caller has not enabled `run_narration_enabled`. Narration is **opt-in**, so an account that never touched the setting gets `204` exactly like one that turned it off. Clients should stop polling — no work was done, nothing will change until the flag is set. (An anonymous request against a public run has no settings to consult and skips the check entirely.) - `503 narration_unavailable` when the narration model is offline; callers fall back to phase-derived labels. ### 4.10 Aggregate stats One non-admin read endpoint summarizes the caller's runs without paginating through them: ``` GET /v1/runs:phase-stats # per-phase duration + attempt aggregates ``` The response is keyed by phase: ```json { "find": { "avg_ms": 8200, "attempts": 41, "success": 40, "failed": 1 }, … } ``` It pairs `PhaseStarted` with `PhaseCompleted` server-side so a client can render "where does my time go" charts without pulling event rows. There are no cost or token aggregates on this endpoint. The scope is narrower than "your runs", and the counters are worth reading literally: - Only your **most recent 100 finalized runs** (`status` of `ok` or `error`) are considered. This is a recent-history sample, not a lifetime total, and it does not move while runs are in flight. - `attempts` counts phases that started; `success` counts those that completed. - `failed` counts a phase that started and never completed **on a run that errored**. The same shape on a run that finished `ok` counts toward neither `success` nor `failed`, so `success + failed` can be less than `attempts`. - With no finalized runs yet the response is a bare `{}` rather than a zero-filled skeleton. Render "no data" from an empty object. ### 4.11 POST /v1/runs/{id}/pause and /resume Operator pause: a non-destructive stop that preserves the work sandbox. Requires `runs:write` and ownership (admins can pause any run). ``` POST /v1/runs//pause # halted=true, halt_reason=operator_pause: POST /v1/runs//resume # clears the pause and relaunches the Driver ``` Pause sets `halted=true` with an `operator_pause` reason; the Driver suspends at the next phase boundary, keeping the work sandbox intact. Returns the updated Run object. Errors: `409 invalid_state` when the run is not `running`; `409 already_halted` when it is already halted (including `awaiting_review` — do not clobber a Driver question with a pause). Resume only lifts an `operator_pause` halt — a run waiting at `awaiting_review` is unblocked via `PATCH review_resolution` (§4.8), not `/resume`; that case returns `409 not_paused`. If the halt was cleared but the Driver could not be relaunched on this host, resume returns `501 resume_not_supported` — fork the run to continue from that point. --- ## 5. Resource: Events and streaming Pipeline events are the primary reason a run resource is event-sourced. Every phase start, every LLM call, every sandbox creation, every materialize-task result is a typed event. Two consumption modes: polling and SSE. ### 5.1 GET /v1/runs/{id}/events Polling / backfill. Returns an array in seq order. ``` GET /v1/runs//events?after_seq=42&limit=500&types=MaterializeTaskResult ``` Additional query parameters: `before_seq` (upper bound, pairs with `after_seq` for a window), `order=desc` (newest first; default is seq-ascending), `exclude_types` (comma list, the complement of `types`), and `limit` (default **500**, capped at **1000**). **`types` and `exclude_types` do not compose.** They write the same underlying filter, so sending both is not an intersection: `exclude_types` wins and `types` is discarded. Pick one per request. Response: ```json { "data": [ { "id": "", "seq": 43, "type": "MaterializeTaskResult", "created_at": "2026-04-23T14:05:01.003Z", "data": { "task_id": "abstract_algebra_0", "index": 0, "score": 1.0, "duration_ms": 822, "output_preview": "The answer is (D). …" } } ], "pagination": { "has_more": true, "next_page_token": "" } } ``` `after_seq` is preferred over `page_token` for event streams because seq is monotonic and stable across reconnects. **`has_more` is not a liveness signal.** It means only "this page came back full", so a running run that happens to have no further events answers `has_more: false`, and a finished run whose last page filled exactly answers `has_more: true`. Decide whether to keep polling from the run's `status` (§4.1) or from the SSE `run.finished` event, never from `has_more`. ### 5.2 GET /v1/runs/{id}/events:stream (SSE) Server-sent events for live dashboards and CLIs. ``` GET /v1/runs//events:stream Accept: text/event-stream Last-Event-ID: 42 ``` Each event is emitted as: ``` id: 43 event: MaterializeTaskResult data: {"type":"MaterializeTaskResult","task_id":"abstract_algebra_0","index":0,"score":1.0,…} ``` The event type appears twice on purpose: in the SSE `event:` line and folded into the `data` object as `"type"`, so a consumer that only ever parses `data` still knows what it is holding. Keep-alives are `event: ping` every 20 seconds. The stream closes cleanly with a final `event: run.finished` when `status` becomes terminal. Opening a stream on a run that has **already** finished is well defined and cheap: you get the backfill, then `run.finished` immediately. Browsers and SDKs use the native `Last-Event-ID` semantics for resume; clients that cannot set headers (e.g. a bare `EventSource`) can pass `?last_event_id=42` as a query parameter instead. ### 5.3 Event identity and ordering - `(run_id, seq)` is globally unique and monotonic per run. - Event IDs are globally unique but not monotonic across runs. - Clients dedupe by event id; both modes can deliver the same event twice (SSE reconnect, poll overlap). **The two modes use different identifiers, and mixing them fails quietly.** `GET …/events` gives each row an opaque `id` that encodes `(run, seq)`. SSE emits the bare `seq` as its `id:`, and reads `Last-Event-ID` as a number. Feed a polling `id` into `Last-Event-ID` and it does not parse, so the stream resumes from the beginning and replays the whole run instead of erroring. The rule that works for both: **resume from `seq`, dedupe by `id`.** Track the highest `seq` you have processed and hand that to `Last-Event-ID` or `after_seq`; use the `id` only for deduplication. --- ## 6. Resource: Artifacts Everything the pipeline emits that is larger than a routine event becomes an artifact: the Driver's `task_impl.py`, each pending save during Atomize, report files, fingerprint breakdowns. One endpoint covers all of them, filtered by `type`. ### 6.1 GET /v1/runs/{id}/artifacts ``` GET /v1/runs//artifacts ?type=in:atomize,report,fetch &include_versions=true ``` Response: ```json { "data": [ { "id": "", "type": "atomize", "name": "task_impl.py", "mime": "text/x-python", "size_bytes": 2841, "version": 3, "latest_version": 3, "seq": 117, "invocation": 2, "created_at": "2026-04-23T14:03:42.000Z", "download_url": "/v1/runs//artifacts//content" }, { "id": "", "type": "report", "name": "report.html", "mime": "text/html", "size_bytes": 48129, "seq": 402, "invocation": 1, "download_url": "/v1/runs//artifacts//content", "created_at": "2026-04-23T14:09:14.000Z" } ], "pagination": { "has_more": false } } ``` Every row carries `seq` and `invocation`. `size_bytes` is **approximate**: it counts characters in the stored JSON string rather than bytes on the wire, so anything non-ASCII reads low. Size a progress bar with it, not a buffer. `type` values: | `type` | Source event | Versions? | |-----------------|--------------------------------|----------------------| | `atomize` | `AtomizeArtifact` | yes, per invocation | | `atomize_draft` | `PendingAtomizeCode` | yes, per save | | `analyze` | `AnalyzeArtifact` | no | | `fetch` | `FetchArtifact` | no | | `model_proxy` | `ModelProxyArtifact` | no (config only) | | `report` | `ReportFile` | no | `type=atomize*` is accepted as a wildcard covering `atomize` + `atomize_draft`. There is no `materialize` artifact type — per-task results are served by `/tasks/{id}/payload` (§7.2). ### 6.2 GET /v1/runs/{id}/artifacts/{art_id} Artifact metadata, as one row of the §6.1 list, with the exception that `version` and `latest_version` are **not** included here even for a versioned type. If you need the version of a specific artifact, read it from the list response rather than from this one. ### 6.3 GET /v1/runs/{id}/artifacts/{art_id}/content Raw bytes with the artifact's `mime` as Content-Type. `Range` headers are not supported — content is always served whole. Active-content mime types (`text/html`, SVG, XML, JavaScript) are downgraded to `text/plain` with `X-Content-Type-Options: nosniff` so an artifact can never script against the API origin. ### 6.4 GET /v1/runs/{id}/fingerprint A run's identity hash and the step-by-step derivation: ```json { "hash": "c8a1b2d3e4f5", "components": [ { "type": "ModelProxyArtifact", "seq": 42, "data": { /* … */ }, "canonical": "{\"model\":\"…\"}", "cumulative_hash": "a1…" }, { "type": "AtomizeArtifact", "seq": 117, "data": { /* … */ }, "canonical": "{\"content\":\"…\"}", "cumulative_hash": "c8…" } ] } ``` Each component carries `data` (the filtered event payload the `canonical` string was derived from) alongside the hash chain, so an audit can see *what* went into the digest and not only that it changed. A run that emitted no artifact events has no fingerprint to break down and answers `404 not_found`. Useful for reproducibility audits ("did two runs use the same code and model config?") without downloading the artifact bodies. ### 6.5 GET /v1/runs/{id}/report Responds `302` with a redirect to the content URL of the run's **first** `ReportFile` (lowest `seq`), so clients don't have to list artifacts first. A run that wrote several report files is not choosing a canonical one here; list `?type=report` (§6.1) if you want the others. The content serves under the §6.3 rules (HTML downgrades to `text/plain`). The route is gated by run visibility only — no artifact scope is required. --- ## 7. Resource: Tasks and results A task is one row of the benchmark dataset plus the per-run result for that row. ### 7.1 GET /v1/runs/{id}/tasks ``` GET /v1/runs//tasks ?version=2 &passed=false &error_class=timeout &include_retries=true &slim=1 ``` Query parameters: `passed` (true/false), `version` (one materialize invocation; default latest, and an out-of-range number is clamped to the latest rather than rejected), `error_class` (exact match on the derived bucket below), `include_retries=true` (keep superseded attempts instead of only the final one per index), `slim=1` (a lighter row, see below). `sort` and `limit` are **not** implemented — the full task set is always returned in one response and `pagination.has_more` is always `false`. `include_retries` is matched against the literal string `true`; `include_retries=1` is not an error and not an enablement, it simply does nothing. `slim=1` drops six fields, not two: `id`, `run_id`, `output_preview`, `passed`, `meta` and `links`. Losing `id` and `links.payload` means you cannot follow a slim row to its payload (§7.2), so use it for aggregate views and not as the first half of a drill-down. Response: ```json { "data": [ { "id": "", "run_id": "", "invocation": 2, "index": 0, "trial": 0, "task_id": "abstract_algebra_0", "output_preview": "The answer is (D). …", "score": 1.0, "passed": true, "duration_ms": 822, "error": null, "error_class": "ok", "timestamp": 1777039501003, "meta": { }, "links": { "payload": "/v1/runs//tasks//payload" } } ], "pagination": { "has_more": false }, "summary": { "total": 100, "passed": 82, "failed": 18, "pass_rate": 0.82, "invocation": 2, "invocation_count": 2, "error_class_counts": { "timeout": 3 }, "duration_ms_p50": 744, "duration_ms_p95": 1801 } } ``` Rows do not carry `input` / `expected` / full `output` — those live behind `links.payload` (§7.2). `timestamp` is **epoch milliseconds**, not an ISO string, and defaults to `0` when the row carries none. `passed` is `score >= 1.0`. `error_class` is **derived, never null**, and always one of a fixed set of lowercase buckets: a row with no error reads `"ok"`, so filter on `"ok"` rather than on a null. The buckets: ``` ok · truncation · disk_full · rate_limited · timeout · code_error sandbox_died · missing_result_file · task_no_result · spawn_failed cancelled · cancelled_pre_launch · inference_rejected · other ``` **`summary` is computed from a different row set than `data`, so `summary.total` normally does not equal `data.length`.** `summary` reports the run's result: it prefers the aggregated per-task rows when the run produced them (trials folded into one score per task), and otherwise uses the per-trial rows with errored attempts filtered out. `data` reports the rows themselves, under whatever filters you passed. Read `summary` for "how did this run do", and do not recompute it from `data` expecting the same number. `duration_ms_p50` / `p95` are omitted when no rows carry durations, and runs launched with `trials` add `trials`, `attempts_per_task`, and `aggregator`. ### 7.2 GET /v1/runs/{id}/tasks/{tsk_id}/payload The full record for one task: `input`, `expected`, `output`, plus `raw_output`, `atif`, `meta`, `score`, `passed`, `duration_ms`, `error`, `error_class`, `size_bytes`, and `created_at`. It also carries the addressing fields `id`, `run_id`, `invocation`, `index`, `trial` and `task_id`. Several of those are computed at read time rather than stored: `passed`, `error_class` (the §7.1 bucket) and the `atif` timing breakdown are all derived per request, so do not expect this response to match a raw row of the underlying table field for field. Two query parameters worth knowing: - `?trial=N` pins one trial of a multi-trial task. **Omit it and you get the best-scoring trial**, not the first or the last. Convenient for "show me what this task can do", misleading if you assumed it was trial 0. - `?raw=0` drops `raw_output`. That field holds the model's untrimmed output and dominates the response on agentic harnesses (a single harbor task measured 5.3 MB with it and 67 kB without), so pass it whenever you only need the scored fields. --- ## 8. Resource: Benchmarks, subsets, impls The public registry of benchmarks, their subsets (splits), their impls (distinct `task_impl.py` fingerprints), and their aggregated leaderboard rows. **Access.** Registry reads are part of the public catalog: an **unauthenticated** request returns every benchmark/subset/impl that has at least one public run. A browser (JWT) session additionally folds in the caller's own private runs, each row carrying a `visibility` field. A **PAT** must declare `registry:read` — a token that authenticates without it gets `403 scope_insufficient`. The authoring endpoints (§8.4–§8.8) require `registry:publish`. Two id spaces coexist. Projected catalog rows (runs the pipeline finalized) use composite ids — a benchmark slug (`MMLU`) and a `:` subset id (`MMLU:7281bbc31736`). Rows created through the authoring endpoints carry the `bm_`/`sub_`/`imp_` prefixes. Both shapes are accepted on the read routes. Only the **benchmark** id is opaque there (`bm_`). Subsets and impls are **content-addressed**: `sub_:<12 hex>` fingerprints the task set, `imp_:<12 hex>` fingerprints the code. So the same tasks or the same scorer under the same benchmark always resolve to the same id, and reading an id tells you which benchmark it belongs to without a lookup. ### 8.1 GET /v1/registry/benchmarks The catalog list. By default it returns the full nested tree (benchmarks → subsets → impls/runs), wrapped in a `data` envelope: ```json { "data": [ /* benchmark tree elements */ ] } ``` For catalog rendering pass `?summary=1` to get the lightweight shape — per-benchmark counts plus per-subset top-score, no nested run arrays — which scales in benchmark count rather than run count. The summary mode returns its own envelope: ```json { "benchmarks": [ … ], "total": 42, "offset": 0, "limit": 50, "facets": { … }, "agg": { … } } ``` `facets` and `agg` are present only when you pass `?counts=1`; without it the keys are absent, so ask for them rather than expecting them. Summary mode also takes list/facet parameters — `limit` / `offset` paging (clamped to 200 here), `?q=` text search, and category/facet filters — so the catalog page can search and paginate server-side. ``` GET /v1/registry/benchmarks?summary=1 &hide_private=1 # drop the caller's own private rows &hide_public=1 # drop public rows (show only my private) ``` Each element of `benchmarks` (abbreviated; the shape carries more counters than are shown): ```json { "id": "MMLU", "name": "MMLU", "category": "knowledge", "visibility": "public", "tags": [], "highlights": ["Tests multi-subject knowledge…"], "subsetCount": 5, "implCount": 5, "runCount": 13, "modelCount": 1, "hasOfficialImpl": false, "lastRunAt": "2026-06-17T18:07:43.819Z", "subsets": [ { "id": "MMLU:7281bbc31736", "slug": "subset-7281bbc31736", "name": "mmlu_first_1500", "sampleCount": 1500, "runCount": 1, "implCount": 1, "topScore": 0.859, "topModel": "qwen/qwen3-32b" } ] } ``` ### 8.2 GET /v1/registry/benchmarks/{id} A single benchmark with its subsets, impls, native + derived leaderboards, covering-run counts, per-subset index spans, and any per-user range names. This is the page-load payload for `/registry/{id}`; per-task content and the difficulty distribution are fetched separately (§8.3) so the overview stays fast. ### 8.3 First-class read endpoints Stable URLs for the pieces a dashboard links to directly, so it doesn't have to fetch the whole benchmark tree. All inherit the §8 access rules (anon-readable; `registry:read` for PATs). ``` GET /v1/registry/stats ?hide_private=1 &hide_public=1 → catalog counters: { benchmarks, subsets, impls, runs, models, tasks } GET /v1/registry/benchmarks/{id}/coverage → per-subset { covering_runs, derived_runs } for the family GET /v1/registry/benchmarks/{id}/range?indices=0-5 → ad-hoc virtual range: every run that ran the tasks at those canonical indices (0-5, 0,2,4 — ≤2000 positions), scored on just them, + a best-per-model leaderboard. No stored subset needed; coverage is by task content, so only runs that actually ran those tasks appear. GET /v1/registry/benchmarks/{id}/range/dataset?indices=0-5 → the canonical task tuples for any set of canonical positions (full benchmark, a slice, or a named range — ≤50000 positions) as JSONL, served as an attachment GET /v1/registry/benchmarks/{id}/range/dataset/info?indices=0-5 → { rows, bytes } for the JSONL above (cheap badge metadata) GET /v1/registry/subsets/{id} → one subset: its impls, runs, leaderboard GET /v1/registry/subsets/{id}/leaderboard → best score per model across impls (public runs only) GET /v1/registry/subsets/{id}/derived → models that ran a SUPERSET of this subset, re-scored on this subset's tasks (no re-run); excludes models with a native run GET /v1/registry/subsets/{id}/covering-runs → every run on a superset of this subset, one row per run, scored on this subset's tasks GET /v1/registry/subsets/{id}/distribution → difficulty histogram + score/run summary + slim per-task rows (index · difficulty · pass-rate, NO content) GET /v1/registry/subsets/{id}/tasks/{index} → one dataset index's full content + per-model results GET /v1/registry/subsets/{id}/dataset → the canonical task tuples as JSONL (one {index, input, expected} per line) — feed straight into an eval harness GET /v1/registry/subsets/{id}/dataset/info → { rows, bytes } for the JSONL above (cheap badge metadata) GET /v1/registry/impls/{id} → one impl: label, fingerprint, official flag, its runs ``` `distribution` → `tasks/{index}` is a three-tier load: the overview (§8.2) ships precomputed aggregates, `distribution` adds per-task difficulty rows with no content, and `tasks/{index}` resolves a single task's full input/expected/output on drill-in — so opening a 1500-task subset never serializes 1500 prompts up front. Per-user range naming (a viewer labels an index range as a named subset) is a `registry:publish` write: ``` POST /v1/registry/benchmarks/{id}/names { subset_id, name, visibility? } DELETE /v1/registry/benchmarks/{id}/names/{slug} ``` ### 8.4 POST /v1/runs/{id}/publish Shortcut for the "publish this run" flow. The body carries one field: ```json POST /v1/runs//publish { "visibility": "public" } ``` The run's fingerprint is recomputed if needed, run visibility flips, and the registry projection updates. Benchmark visibility is derived from run visibility — a benchmark is public when it has at least one public run, so there is no separate "publish the benchmark" flag. `visibility` is `private`, `unlisted` or `public`, and defaults to `public` when you omit it: this endpoint exists to publish, so the empty body does the obvious thing. `unlisted` makes the run readable by anyone holding its URL without listing it in the catalog. Response: ```json { "id": "", "visibility": "public", "fingerprint_hash": "c8a1b2d3e4f5" } ``` `fingerprint_hash` is present when the run has one. Publishing is refused `409 visibility_constraint_violated` when the run did not finish `ok`, or when it has no benchmark to publish under. Both are about the registry rather than about permission: a leaderboard row needs a score and something to hang it on. ### 8.5 POST /v1/benchmarks: create a benchmark The catalog is a first-class resource, not just a side effect of running things. Creating a benchmark requires `registry:publish` on PATs; it starts private and only your runs reference it until you flip visibility. ```json POST /v1/benchmarks { "name": "BFCL v4 (my mirror)", "description": "Mirror of Berkeley Function Calling Leaderboard v4.", "category": "tool-use", "tags": ["function-calling", "parallel"] } ``` Response: the Benchmark object with `id: "bm_"`. Two things about that response are worth knowing before you build on it. The `"visibility": "private"` it reports is a **fixed literal, not a stored field**: benchmarks have no visibility column, PATCH ignores `visibility` if you send it, and a benchmark's real visibility is derived from its runs (§8.4). And while `source`, `sources`, `paper` and `dataset` do exist as provenance fields on a benchmark, they are not settable through this endpoint. ### 8.6 POST /v1/benchmarks/{bm_id}/subsets: create a subset Subsets hold the actual task rows. Non-rolling subsets freeze after creation; rolling subsets accept appends indefinitely. Pass a `grader` to make the subset **runnable**: the recipe (`setup`/`boot`/`run`/`capture`, judge settings) the pipeline executes your samples with. Without one this is a catalog-only publish, which is the right shape for a harness-backed benchmark that brings its own scorer. Your samples are stored either way; the response reports which you got in `runnable`, and a run pinned to a subset that has none fails immediately with `product_benchmark_no_recipe` rather than part-way through setup. ```json POST /v1/benchmarks/bm_01J…/subsets { "slug": "parallel-function-v4", "name": "Parallel Function v4", "description": "200 parallel function calling questions from BFCL v4.", "rolling": false, "tasks": [ { "task_id": "parallel_function_001", "input": { "question": "…", "functions": [ /* OpenAI tool shape */ ] }, "expected": [ { "name": "search_flights", "args": { /* … */ } } ] } ] } ``` `slug` and `name` are both required. The response is a short acknowledgement, not the full Subset object: ```json { "id": "sub_bm_01J…:7281bbc31736", "slug": "parallel-function-v4", "name": "Parallel Function v4", "created": true, "rolling": false, "sample_count": 200, "benchmark_id": "bm_01J…", "runnable": true } ``` Like an impl id, a subset id is content-addressed: `sub_:`. **The fingerprint covers the TASKS, not the slug**, so posting the same task set under a different slug is an *update* of the existing subset, not a second one: same id, and your `slug` / `name` / recipe replace the ones already on that row. That is what makes re-authoring idempotent, but it means the slug cannot be used to keep two variants of one task set apart. The response says which happened: `201` + `"created": true` for a new row, `200` + `"created": false` for an update, and `renamed_from_slug` appears only when your slug displaced a different one. Watch for that field: the row you renamed may already be referenced by finished runs. Shape constraints: - `input` + `expected` must be JSON-serializable. - 1000 tasks per request (the only enforced size cap). Chunk larger datasets and use `Idempotency-Key` per chunk to make resumable uploads safe. Keep `task_id` unique within the subset. Nothing rejects a duplicate, but `task_id` is how §8.9's `upsert` / `delete` address a sample, so a repeated one makes those edits ambiguous. Rolling subsets accept appends: ``` POST /v1/subsets//tasks:append Idempotency-Key: bugs-append-2026-04-23 { "tasks": [ /* same shape */ ] } ``` The server appends the rows and bumps `sample_count`. **Appends are not versioned**: there is no snapshot marker and no way to address the subset as it stood at an earlier size, so a run resolves the task set as it stands at the moment it runs. If you need a set that cannot move under a comparison, publish it as a non-rolling subset. Appending to a frozen (non-rolling) subset fails `409 subset_frozen`. ### 8.7 POST /v1/impls: register a scorer An impl is a `task_impl.py` (or future equivalents) bound to a subset. Impls are immutable: the code defines the fingerprint, and changing the code means registering a new impl. Required fields are `subset`, `label`, and `code`; `benchmark` and `language` are ignored if sent (the subset already implies the benchmark, and Python is the only impl language). ```json POST /v1/impls { "subset": "", "label": "strict-set scorer v1", "code": "import json\nfrom bench_task import BenchTask\n\nclass Task(BenchTask): …" } ``` Returns the Impl object: ```json { "id": "imp_bm_01J…:c8a1b2d3e4f5", "subset": { "id": "sub_bm_01J…:7281bbc31736" }, "label": "strict-set scorer v1", "fingerprint": "c8a1b2d3e4f5", "created_at": "2026-04-23T14:00:00.412+00:00" } ``` The id is not opaque: it is `imp_:`, and `fingerprint` repeats those same 12 characters. Register the same code twice and you get the same id back, which is what "impls are immutable" means in practice. `GET /v1/impls/{id}/code` (fetching the stored code back) is **not yet available**. ### 8.8 Updating and deleting ``` PATCH /v1/benchmarks/{id} PATCH /v1/subsets/{id} PATCH /v1/impls/{id} ``` | Resource | PATCH fields | |------------|-------------------------------------------------| | Benchmark | `name`, `description`, `category` | | Subset | `name`, `description`, `rolling→false` (freeze) | | Impl | `label`, `description` | DELETE routes for benchmarks, subsets, and impls are **not yet available**. **A PATCH does not respond with the updated object.** Benchmark and subset answer `{"id": "…", "ok": true}`; impl answers the same plus an echo of whichever of `label` / `description` / `official` it changed. Re-read the resource if you need its full state back. `If-Match` (§3.8) is **enforced on `PATCH /v1/benchmarks/{id}` only**. The subset and impl handlers accept the header and ignore it, so a stale write there succeeds silently rather than answering `412`. Send it anyway, since it costs nothing and starts working as those handlers migrate, but do not rely on it for read-modify-write on subsets or impls yet. `official=true` on an impl is a privileged write. A non-admin PATCH that touches `official` fails `403 admin_required` before the rest of the patch is considered, so a mixed patch changes nothing. A PATCH naming no recognized field at all is `400 invalid_argument`, and the detail lists what is patchable. The `rolling→false` transition is one-way: once you freeze a subset, you can't reopen it. Create a new subset if you need to keep accreting. Task rows are never patched in place. On a rolling subset you can append; on a frozen subset you can't. If a task is wrong, register a new subset with the correction, and let the registry show both so consumers can see which runs used which. ### 8.9 Editing a BUILT benchmark (recipe + samples) The routes above edit catalog prose. A benchwright-BUILT product benchmark also has a **recipe** (the sandbox script set that boots the product, drives it, captures its output and judges the result) and its **samples**. Those live in the authored definition, and the owner edits them here: ``` GET /v1/build/subsets/{id} PATCH /v1/build/subsets/{id}/recipe PATCH /v1/build/subsets/{id}/tasks ``` `{id}` is a subset id, a benchmark id, or a benchmark **name** — the last two resolve to that benchmark's live subset, so the handle the registry printed is enough and you do not have to look an id up first. A subset id from `/v1/registry/benchmarks` also works: catalog ids and authored ids come from different fingerprints, and a catalog one is mapped to its benchmark's authored subset for you. A name matching more than one benchmark answers `409 ambiguous` listing the ids rather than picking one, since this route changes how a benchmark scores. A subset that reached the catalog through a **run** has no authored recipe at all, and says so (`404 not_authored`) rather than implying the id was wrong. All three are gated on the benchmark's **owner** (admins too); anyone else gets `404`. The GET needs `registry:read` (diffing a recipe before proposing a fix is a read); both PATCHes need `registry:publish`. ```bash # broaden a mint that was too narrow, raise a timeout curl -X PATCH https://benchwright.ai/v1/build/subsets/sqlbot:9f2c1a7b0e44/recipe \ -H "Authorization: Bearer $BW_PAT" -H 'Content-Type: application/json' \ -d '{"config":{"boot":"mint --match \"^clip-[0-9]+$\"","run_timeout":900}, "note":"mint missed multi-digit clips"}' # retarget one sample curl -X PATCH https://benchwright.ai/v1/build/subsets/sqlbot:9f2c1a7b0e44/tasks \ -H "Authorization: Bearer $BW_PAT" -H 'Content-Type: application/json' \ -d '{"upsert":[{"task_id":"q2","input":{"at":"00:04:40","clip":"c3"}}]}' ``` **How each one versions.** A recipe patch is a merge: keys you send replace theirs, an explicit `null` deletes one, the rest is untouched. The subset id does not move (it addresses the samples, which did not change) and the *scorer* version moves instead: the recipe is content-addressed into `task_impl.py`, so the next run mints a new impl id and every run stays stamped with the recipe that graded it. The response carries `recipe_fingerprint` and `changed_keys`; an edit that changes nothing answers `changed: false`. A tasks patch (`tasks` to replace the set, or `upsert`/`delete` by `task_id`) **forks**: the edited sample set gets its own content-addressed subset carrying the recipe forward, the response names both ids, and the old subset is retired so runs stop resolving to it. Editing a retired subset answers `409 superseded` naming its successor; editing samples back to a retired set revives that subset rather than minting a third. The GET returns an `ETag`, and both PATCHes honor `If-Match` (§3.8) — worth using here, since editing a recipe is read-modify-write and a mismatch answers `412 precondition_failed` instead of overwriting someone else's fix. Other failures use the standard codes: `404 not_found` (unknown subset, or not yours), `403 scope_insufficient`, `400 invalid_argument` (an empty sample set, over 2000 tasks, or a recipe patch carrying neither `config` nor `type`; either one alone is fine, so `{"type": "…"}` with no `config` is a valid retype). The same three, from the CLI: ```bash benchwright bench recipe show my-bench benchwright bench recipe set my-bench --set run_timeout=900 --set trials=3 --note 'why' benchwright bench samples set my-bench --upsert-file fix.jsonl --note 'why' ``` --- ## 9. Resource: Secrets and host policy Two related collections: - **Secrets**: named blobs of credential value (an API key, an HTTP bearer token, a basic-auth pair). - **Rules**: per-secret host-pattern entries that tell the server-side injection proxy how to attach the secret to outbound requests. - **Host policy**: per-user allow/block list for outbound HTTP from the pipeline. ### 9.1 GET /v1/secrets ```json { "data": [ { "id": "", "name": "OPENROUTER_API_KEY", "description": "personal workspace key, sandboxed org", "created_at": "2026-02-14T…Z", "updated_at": "2026-04-10T…Z", "rules": [ { "id": "", "secret_id": "", "host_pattern": "openrouter.ai", "inject_kind": "bearer", "inject_target": "Authorization", "created_at": "2026-02-14T…Z" } ], "injections": ["product_sandbox"] } ], "meta": { "payer": { /* whose vault this call is scoped to, §2.5 */ } } } ``` Secret values never surface — there is no masked preview and no last-used timestamp on responses. Rule ids are bare UUIDs. **`rules` and `injections` are two different mechanisms.** A rule tells the injection proxy to attach the secret to *outbound HTTP* matching a host pattern. An injection exports the secret as an *environment variable, under its own name*, when a machine boots. `injections` is a flat array of surfaces, each one `product_sandbox` (the sandbox your benchmark runs in) or `operator` (your operator's machine); an empty array means the secret is only ever used through its rules. ### 9.2 POST /v1/secrets ```json { "name": "OPENROUTER_API_KEY", "description": "...", "value": "sk-or-...", "rules": [ { "host_pattern": "openrouter.ai", "inject_kind": "bearer", "inject_target": "Authorization" } ], "injections": ["product_sandbox"] } ``` `inject_kind` must be one of `bearer`, `header`, `basic`, `query`; anything else is `400 invalid_argument`. `inject_target` is the header name or query parameter name. Creating a rule with `host_pattern=foo.example.com` also upserts `user_host_policy(mode=allow)` for that host, so the proxy doesn't block it. **A rule sent inline here is skipped silently when any of `host_pattern`, `inject_kind` or `inject_target` is blank**: the secret is still created, minus that rule. Send complete rules, or add them afterwards through §9.5, which validates each one on its own. `injections` (optional) is the boot-time env-var surface list from §9.1: each entry must be `product_sandbox` or `operator`. Because an injected secret becomes an environment variable under its own name, a non-empty `injections` also requires the secret **name** to be a valid identifier (`^[A-Za-z_][A-Za-z0-9_]*$`); otherwise the create fails `400 invalid_argument` rather than quietly renaming your variable. Renaming a secret through PATCH is held to the same rule while it has injections. `POST /v1/secrets` returns `201` with `{"id": ""}`. Two conflict cases: `409 name_conflict` when a secret with that name already exists, and `409 cloud_managed` when the name is `OPENROUTER_API_KEY` while the account is enrolled in Cloud inference (that key is platform-managed). ### 9.3 Secret rotation ``` PATCH /v1/secrets/ Idempotency-Key: … { "value": "sk-or-new-key-value" } ``` Returns `200` with `{"id": ""}`. A run already in flight keeps using the previous value (the proxy caches per-run); new runs get the new one. One qualifier on that: a run **resuming** from a halt or a pause re-reads the vault as it comes back, so a run that was paused across a rotation resumes on the new value. Only a run that never stops keeps the old one for its whole life. `PATCH` without a `value` field updates only the description / name — the stored secret is not touched. Use this to relabel without invalidating in-flight runs. ### 9.4 Deleting a secret ``` DELETE /v1/secrets/ ``` Returns `204` and cascades into the secret's rule rows. Active runs are unaffected (the proxy already has its per-run copy); the next launch that needs a matching credential will fail `412 no_provider_secret` until you create a replacement. Deleting also withdraws the host-policy entries the secret's rules added for it (the ones §9.6 reports with `source=auto_from_secret`), so removing a credential does not leave its hosts allowed. An entry you added yourself (`source=user`) survives, since you asked for that one independently. ### 9.5 Managing rules independently For day-to-day allowlist tuning the rule collection has its own URLs, so you don't need to PATCH the whole secret to add or drop one host pattern: ``` POST /v1/secrets//rules { "host_pattern": "openrouter.ai", "inject_kind": "bearer", "inject_target": "Authorization" } DELETE /v1/secrets//rules/ ``` `POST` returns `201` with the new SecretRule wrapped in a **single-element array** (the database's representation is passed through verbatim), so read `[0]` rather than the object directly. `inject_kind` must be one of `bearer` / `header` / `basic` / `query` (anything else is `400 invalid_argument`); `inject_target` is the header name or query parameter. Creating a rule auto-allows the host pattern in `user_host_policy` (same behavior as inline rules under §9.2). `DELETE` is `204`. ### 9.6 Host policy ``` GET /v1/host-policy POST /v1/host-policy { host_pattern, mode } DELETE /v1/host-policy/{id} POST /v1/host-policy:starter-pack # adds github, pypi, huggingface… PATCH /v1/host-policy/default { mode: "allow" | "block" } ``` ```json GET /v1/host-policy { "data": [ { "id": "", "host_pattern": "openrouter.ai", "mode": "allow", "source": "auto_from_secret", "created_at": "2026-02-14T…Z" } ], "default_mode": "allow" } ``` **`default_mode` is `allow`.** The list is not an allowlist that everything else falls outside of: by default a run may reach any host, and the entries refine that. Flip the default with `PATCH /v1/host-policy/default {"mode": "block"}` when you want the opposite posture, and read `default_mode` before you interpret the list, because the same rows mean different things under each. `source` records who added an entry: `user` (you did), `auto_from_secret` (a secret rule added it, §9.2) or `starter_pack`. It matters on delete (§9.4). `mode` on POST is `allow` or `block`, and the reply is `201 {"ok": true}`. `POST /v1/host-policy:starter-pack` adds the five hosts almost every run wants (`github.com`, `raw.githubusercontent.com`, `pypi.org`, `files.pythonhosted.org`, `huggingface.co`) and answers `{"ok": true, "added": 5}`. `GET /v1/host-policy` requires the `secrets:read` scope on PATs; writes require `host_policy:write`. ### 9.7 Blocked-host log ``` GET /v1/runs/{id}/blocked-hosts GET /v1/blocked-hosts DELETE /v1/blocked-hosts?host= ``` Returns hosts that the injection proxy refused during the run, with `seen_count`, `first_seen`, `last_seen`. Makes "trust on observation" a first-class workflow: run once, see what got blocked, decide which to allow, rerun. The two reads take **different scopes**, because they answer different questions. `GET /v1/runs/{id}/blocked-hosts` is part of the run, so it needs `runs:read` (same as `/v1/runs/{id}/egress-routes`, §9a.4). The account-wide `GET /v1/blocked-hosts` is a vault-shaped read and needs `secrets:read`. `DELETE /v1/blocked-hosts?host=` clears the log rows for one host (requires `host_policy:write`; returns `204`) — used after allowing a host so it stops appearing in the recently-blocked list. --- ## 9a. Resource: Local egress Runs reach the internet from the datacenter by default. Local egress lets a run exit from a machine you control instead, which is what you want when a site will not serve a datacenter address or a page only renders correctly from a real home connection. You register the machine once, then name it on a run. CLI: `benchwright egress up|ls|rm` and `benchwright run --egress` (see [cli.md](./cli.md#egress)). ### 9a.1 Registering a machine ``` POST /v1/egress/endpoints # register, and heartbeat while serving GET /v1/egress/endpoints # your registered machines DELETE /v1/egress/endpoints/{id} # forget one ``` | Method | Path | Scope | Notes | |--------|------|-------|-------| | POST | `/v1/egress/endpoints` | `runs:write` | Register a machine, or refresh it while it serves. Body `{id, public_key, nonce, signature, label?, mode?, allow_hosts?}`. Registration is signature-checked, so knowing a fingerprint is not enough to claim the machine it names. **409** when that fingerprint is already registered to a different key. | | GET | `/v1/egress/endpoints` | `runs:read` | Your machines: `{id, label, mode, allow_hosts, last_seen_at, created_at, instance_id, claimed_at, contended_at, contended_detail}` plus a computed `online`. | | DELETE | `/v1/egress/endpoints/{id}` | `runs:write` | Forget one. Runs that named it will no longer find it. | These sit on `runs:*` rather than a scope of their own. A machine only ever exists to serve your own runs, so a token that can launch runs can offer a machine to run them from, and no existing token needs re-minting. **You choose the fingerprint, the server does not mint it.** Send it as `id` in the shape `eg_` plus 10 characters from `a-z2-7` (`^eg_[a-z2-7]{10}$`); anything else is a `400`, and the response echoes the id you sent. It is public by design (it travels in run commands and shared recipes), which is exactly why the registration is signed: the request must also carry `public_key`, a `nonce`, and a `signature` over them, and a fingerprint already held by a different key is refused `409` rather than re-pointed. `benchwright egress up` generates and stores the keypair for you. **The declared policy is `mode` plus `allow_hosts`, and the two are read together.** `allow_hosts` carries the same list in every mode; `mode` decides what the list means, so neither field is meaningful alone: | `mode` | `allow_hosts` means | Everything else | |---|---|---| | `hosts` (default for a new endpoint) | route only these through the machine | takes the normal datacenter path | | `except` | route everything except these | routes through the machine | | `all` | not used, send an empty list | routes through the machine | A host that is not routed through your machine still reaches the internet by the normal path. This is a routing choice, not an access rule. To restrict what a run may reach at all, use host policy (§9). **Registering is also the heartbeat, and a heartbeat cannot change policy.** The agent repeats this call while it serves. A request that omits `mode` keeps whatever is already stored rather than falling back to a default, so an older client that does not send the field cannot quietly rewrite a policy simply by staying alive. Only a request that states `mode` changes it, and the response echoes the policy now in force so a caller can see what it is actually serving instead of assuming its own defaults applied. If a different agent process claims a fingerprint whose current holder is still live, the claim is allowed, since the usual cause is a restart, and it is recorded. `GET /v1/egress/endpoints` then returns `contended_at` and `contended_detail` on that endpoint. This matters because two agents on one fingerprint each declare their own policy, and a run freezes whichever was in force at launch. ### 9a.2 Asking for it on a run Launch with `metadata.local_egress: true` to use whichever of your machines is serving, or name one with `metadata.egress_endpoint: "eg_…"`. **`metadata.wait_for_egress` applies to the pinned form only.** The launch-time liveness check runs when you named a specific machine: without `wait_for_egress` the launch is refused `412 egress_endpoint_unavailable` if that machine is not serving right now, and with it the run is held until the machine comes online. A bare `local_egress: true` names nothing to check, so it gets no launch-time check at all: the run starts either way and picks up whichever machine is serving when it needs one. Pin the endpoint when you want the launch to fail fast on a machine that is down. ### 9a.3 Serving work ``` GET /v1/local-egress/pending # the next run waiting for this machine GET /v1/local-egress/{id} # liveness while serving POST /v1/local-egress/{id}/attach # connected, the run may proceed POST /v1/local-egress/{id}/fail # could not connect, fail the run POST /v1/local-egress/{id}/denials # hosts your own allowlist refused ``` Your machine is behind your router, so nothing can call it. It asks for work instead: poll `pending` (`runs:read`), connect, then `attach` (`runs:write`) to release the run. `GET /v1/local-egress/{id}` is status only and returns no connection details, so a long-running session can check liveness without re-fetching credentials. `benchwright egress up` does all of this for you. `POST …/denials` (`runs:write`, body `{"denials": [{"host": "…", "count": N}]}`) is how your machine reports what **it** refused. Those hosts land on the run's blocked-hosts card (§9.7) tagged `source=home`, alongside the ones the platform proxy blocked. Report them: a run that died because your own allowlist said no otherwise shows an empty card, and the reason it stopped is invisible from our side. The session id scopes the report to that run, so a stray call cannot land on someone else's. ### 9a.4 Where the traffic actually went ``` GET /v1/runs/{id}/egress-routes ``` Requires `runs:read`. One row per host the run reached: ```json { "data": [ { "host": "example.com", "route": "tunnel", "seen_count": 34, "first_seen": "2026-08-08T01:12:03Z", "last_seen": "2026-08-08T01:14:57Z" }, { "host": "image.tmdb.org", "route": "direct", "seen_count": 8, "first_seen": "2026-08-08T01:12:11Z", "last_seen": "2026-08-08T01:13:40Z" } ] } ``` `route` is `tunnel` when the host left through your machine and `direct` when it took the normal datacenter path. This is the run's own record of each decision as it made it, so you can confirm that only what you intended crossed your connection. The same list is on the run page. If you were also watching from your own machine, the two should agree, and a disagreement is worth a look. --- ## 10. Resource: Balance and top-ups ### 10.1 GET /v1/billing/balance ```json { "object": "balance", "balance": { "amount_cents": 42155, "currency": "USD" }, "month_spent": { "amount_cents": 1230, "currency": "USD" }, "monthly_cap": { "amount_cents": 10000, "currency": "USD" }, "auto_refill": { "enabled": true, "amount": { "amount_cents": 2500, "currency": "USD" }, "threshold": { "amount_cents": 500, "currency": "USD" } }, "alert": { "enabled": true, "at_cap_pct": 80 }, "updated_at": "2026-04-23T14:07:05.889Z" } ``` `?expand=topups,usage_trend,counters,tiers,subscription` folds those blocks into the same response (§3.6); an unknown expand key fails `400 unknown_filter_field`. The payer block (§2.5) is **top-level `payer` here**, not `meta.payer`. This one endpoint differs from the token, secret, and settings responses, which nest it under `meta`. Read both spellings if your client is generic over the payer. Every money object on this endpoint is whole cents rounded HALF_UP, with no `amount_usd` beside it; that exact field exists on run costs (§3.3, §4.1), where sub-cent figures are routine. ### 10.2 POST /v1/billing/topups Browser-facing hosted Stripe Checkout (requires an interactive user; PATs need `billing:write`): ```json POST /v1/billing/topups { "amount_cents": 10000 } ``` `amount_cents` must be between 500 ($5) and 5,000,000 ($50,000). Response: ```json { "url": "https://checkout.stripe.com/c/pay/…" } ``` ### 10.3 Top-up status ``` GET /v1/billing/topups/top_ ``` Reports `"succeeded"` once the payment landed; a top-up that has not succeeded (still pending, failed, abandoned) is a `404`. ```json { "id": "top_", "status": "succeeded", "amount": { "amount_cents": 5000, "currency": "USD" }, "stripe_payment_intent_id": "pi_…", "created_at": "2026-04-23T14:05:00.412Z" } ``` Top-up ids round-trip in their `top_`-prefixed form: that is what the response carries, and the path accepts it (a bare UUID works too, the prefix is stripped). ### 10.4 GET /v1/billing/transactions Full ledger: ``` GET /v1/billing/transactions?kind=trace_debit&limit=100&offset=0 ``` `kind` takes ONE exact value from `stripe_topup`, `trace_debit`, `cli_credit`, `cli_set`, `cli_reset`, `support_credit`, `refund`, `adjustment`, or the literal `all` to skip filtering (the same as omitting it). Anything else is `400 invalid_argument`; there is no `in:` operator and no `created_at` filter here. Pagination is `limit` (max 500) / `offset`, with a `total_estimate` (§3.4). **That list is the filter allowlist, not the set of values a row can hold.** The ledger records around twenty kinds (`inference_usage`, `operator_usage`, `proxy_linger_usage`, `dev_env_starter`, `blog_narration` and others), so an unfiltered page will return kinds you cannot filter on. Switch on `kind` defensively and give unknown values a generic row rather than dropping them. Each row: ```json { "id": "txn_", "kind": "trace_debit", "delta": { "amount_cents": -412, "currency": "USD" }, "balance_after": { "amount_cents": 41743, "currency": "USD" }, "reference_id": "", "reference_type": "run", "note": null, "created_at": "2026-04-23T14:05:00Z" } ``` ### 10.5 PATCH /v1/billing/controls Change monthly cap, auto-refill, alert settings. Requires `billing:write`. Runs already mid-flight are unaffected until their next debit tick. ### 10.6 GET /v1/billing/usage-trend Six-month bucketed spend, same shape as today's summary: ```json { "current_month_cents": 754, "previous_month_cents": 1230, "months": [ { "label": "Nov", "month": "2025-11", "cents": 0 }, … { "label": "Apr", "month": "2026-04", "cents": 754, "current": true } ] } ``` `current_month_cents` and `month_spent` on §10.1 are the same ledger rows, bucketed two ways, so the current bar and the figure under your cap agree by construction. This one is exact; `month_spent` is rounded to whole cents. ### 10.7 GET /v1/billing/export CSV export with stable column order and RFC 4180 quoting. The `Content-Disposition` header carries the filename. `?type=transactions` (default) exports the ledger; `?type=runs` exports per-run rows. ### 10.8 GET /v1/billing/statement Print-friendly HTML monthly statement. Returns `text/html` with a canonical layout (header, period range, debits + credits table, running balance, footer). One query parameter: ``` GET /v1/billing/statement?month=2026-04 ``` Drop it straight into a browser tab or stream into `wkhtmltopdf` for a PDF copy. The shape never changes mid-month, so it's also suitable as a stable artifact to attach to procurement tickets. ### 10.9 POST /v1/billing/subscription Changes the account's membership tier. `{"tier": ""}` starts a Stripe Checkout flow for a paid tier (response carries the checkout `url`); `{"tier": "free"}` cancels any active subscription at period end (idempotent — already-free accounts get an ack). This endpoint returns legacy `{"error": "…"}` bodies on failure (§3.9). An optional `"interval"` picks the billing period: `year` / `yearly` / `annual` selects the tier's annual price, anything else (including omitting the field) bills monthly. ### 10.10 POST /v1/billing/verify-subscription Reconciles a completed tier-upgrade Checkout without waiting on the Stripe webhook. Requires `billing:write`. ```json POST /v1/billing/verify-subscription { "session_id": "cs_…" } ``` Post the `session_id` Checkout hands back on the success return and the tier is applied server-side, idempotently, so a webhook that arrives later is a harmless no-op. Useful anywhere the webhook may not reach you, and as a belt-and-braces step after `POST /v1/billing/subscription` so a paid upgrade never sits unapplied. A session that is not yet complete answers `{"ok": false, "reason": "pending"}`; one belonging to another user is `403`. The webhook remains the authority for ongoing lifecycle (renewals, cancellations, dunning). --- ## 11. Resource: Settings Per-user KV settings, carried forward from today's `user_settings` table. The allowed keys are constrained to prevent the write route from becoming a database write primitive. ``` GET /v1/settings PUT /v1/settings/{key} { "value": } DELETE /v1/settings/{key} GET /v1/settings:schema ``` There is no per-key GET — read everything with `GET /v1/settings` and pick the key client-side. Response: ```json { "data": { "litellm_proxy_ui_enabled": { "value": true, "updated_at": "…" }, "run_narration_enabled": { "value": false, "updated_at": "…" } }, "meta": { "payer": { /* whose settings these are, §2.5 */ } } } ``` A team member reads the **owner's** settings, which is what `meta.payer` names: run defaults follow the wallet the runs bill to. `/v1/settings:schema` returns the flat allowlist of key names. Read it rather than hardcoding the list, since keys are added additively: ```json { "keys": ["score_display_format", "litellm_proxy_ui_enabled", "litellm_full_logging_enabled", "run_narration_enabled", "goal_narration_enabled", "phase_models", "analyzer_model", "operator_harness", "operator_model", "default_model_under_test", "default_harness", "narrate_enabled", "narrate_voice"] } ``` PUT or DELETE against a key outside that list returns `400 invalid_argument`. `value` is any JSON element, not just a boolean or a string: `analyzer_model` takes `{"model": "", "provider": "openrouter|together"}`. Keys that feed a launch or a pod boot are validated on write, so a bad value fails here rather than on every future run. All of these are `400 invalid_argument`: | Key | Accepted | |---|---| | `score_display_format` | `fraction` \| `percent` | | `operator_harness` | `gemini-cli` \| `claude-code` \| `grok-build` | | `operator_model` | a plain model id (letters, digits, `.` `_` `/` `-`) | | `default_harness` | blank, or `inspect` \| `lmeval` \| `lighteval` \| `harbor` \| `benchwright` | | `default_model_under_test` | blank, or `@` (e.g. `qwen/qwen3-32b@openrouter`) | | `narrate_voice` | one of a fixed set of prebuilt voice names (`Kore`, `Puck`, …) | --- ## 12. Resource: Bug reports Authenticated users can file a bug against any run they can see. Reports go to the Benchwright team's triage queue. ``` POST /v1/bug-reports { "description": "Atomize wrote a scorer that always returns 1.0. Repro: …", "trace_id": "" // optional, omit for a generic report } ``` `description` is capped at 8000 characters. Response is `201`: ```json { "id": "", "status": "open" } ``` `status` is one of `open`, `triaged`, `in_progress`, `resolved`, `wontfix` and is updated by the Benchwright team; you see your own reports surface in the run detail view they were reported against. --- ## 12a. Resource: Operators Message your operators and read their replies over the API — the same conversations you see under `/manage/operators`, without opening a browser. Handy for scripts, CI steps, and other agents that need to assign work and poll for an answer. ``` POST /v1/operators # create an operator and boot it GET /v1/operators # list your operators POST /v1/operators/{id}/start # start a stopped one, or restore chat POST /v1/operators/{id}/shutdown # release its machine (the operator is kept) GET /v1/operators/{id}/messages # read conversation (+ queued messages) GET /v1/operators/{id}/events # the same conversation as SSE GET /v1/operators/{id}/files/{name} # fetch an image the operator shared GET /v1/operators/{id}/pending # list messages still outstanding DELETE /v1/operators/{id}/pending/{ts} # take one outstanding message back DELETE /v1/operators/{id}/pending # cancel all queued messages POST /v1/operators/{id}/message # deliver one chat turn POST /v1/operators/{id}/stop # interrupt the current turn (not teardown) ``` CLI: `benchwright operator ls|msg|log|create|start|shutdown` (see [cli.md](./cli.md#operator)). **`stop` and `shutdown` are different, and the names are kept apart on purpose.** `stop` interrupts whatever the operator is working on right now and leaves it running. `shutdown` releases the machine it runs on. There is no `DELETE /v1/operators/{id}`: that verb is reserved for deleting an operator record, which nothing does yet, and a DELETE that left the record in place would be lying about what it did. | Method | Path | Scope | Notes | |--------|------|-------|-------| | POST | `/v1/operators` | `operators:write` | Create and boot. Body `{"name":"…"}`, all fields optional; omit `name` and one is picked. **200** `{"ok":true,"id":"…","status":"booting"}`. Booting continues in the background, so poll `GET /v1/operators`. **402** on a plan that does not include operators, or at your operator limit, or with no balance. **409** `name_taken`. | | GET | `/v1/operators` | `operators:read` | Returns `{ "data": [ { id, name, status, placement, mode }, … ] }` — your operators only. | | POST | `/v1/operators/{id}/start` | `operators:write` | Bring an operator back. On a stopped one this boots a fresh machine under the same id and name: **200** `{"ok":true,"action":"boot","status":"booting"}`. On one that is running but whose chat has stopped answering, it restores chat in place and keeps the machine, its working directory and the open session: **200** `{"ok":true,"action":"revive","detail":"…","death_log":"…"}`, where `death_log` is what the previous chat process left behind. Safe to call on a healthy operator. **402** no balance. **404** unknown or not owned. | | POST | `/v1/operators/{id}/shutdown` | `operators:write` | Release the operator's machine. The operator record is kept and reads `stopped`, so `start` brings it back. The machine and anything in its working directory are discarded. **200** `{"ok":true}`. **404** unknown or not owned. Same action as the Stop button under `/manage/operators`. | | GET | `/v1/operators/{id}/messages` | `operators:read` | Conversation snapshot. Query: `since`, `limit` (default **50**, max **500**), `wait` (seconds, max **50** — hold open until new turns), `wait_for` (`turn` \| `idle`). **200** body below. **400** bad `wait`/`wait_for`. **403** missing scope. **404** unknown or not owned. **409** operator not `status=running`. | | GET | `/v1/operators/{id}/events` | `operators:read` | **Server-Sent Events** stream of the conversation — push instead of polling. Query: `since`, `limit`, `wait_for`. Honours `Last-Event-ID` on reconnect. **501** if the host can't stream. Otherwise same errors as `/messages`. Details below. | | GET | `/v1/operators/{id}/files/{name}` | `operators:read` | One image the operator put in `~/chat-files/` on its own machine — the same picture you see in its chat. **200** image bytes. **403** missing scope. **404** unknown/not owned, or no such file. **409** not `status=running`. | | GET | `/v1/operators/{id}/pending` | `operators:read` | Everything of yours the operator has not answered yet (see below); this is **not** just the undelivered ones. **200** `{ "pending": […], "count": N, "outstanding": N, "states": {…}, "state_meanings": {…} }`. **403** / **404** / **409** as above. | | DELETE | `/v1/operators/{id}/pending/{ts}` | `operators:write` | Take one outstanding message back, addressed by its `ts`. **200** `{"ok":true,"id":"…"}`. **409** with the item's `state` when it is past withdrawal. **404** `pending_not_found` if already gone or unknown (never a partial failure). **400** bad id. | | DELETE | `/v1/operators/{id}/pending` | `operators:write` | Drop **all** items still queued on disk. **200** `{"ok":true,"deleted":N,"ids":[…]}`. | | POST | `/v1/operators/{id}/message` | `operators:write` | Body `{"text":"…"}` (required, non-blank, ≤ 64 KiB). **202** `{"ok":true}` when accepted — queued on the operator box, **not** fire-and-forget. The reply carries **no handle**: to cancel, read `GET …/pending` and DELETE by the item's `ts`. **403** missing scope. **404** unknown or not owned. **409** not `status=running`. | | POST | `/v1/operators/{id}/stop` | `operators:write` | Interrupt the **current turn** (not pod teardown). **200** `{"ok":true,"cancelled":true,"forced_clear":bool}`. Works when the agent process is already dead — force-clears a stuck `working` transcript so the UI unsticks. **403** / **404** / **409** as above. | A write-only PAT can post/cancel but not list or read; a read-only PAT can list/read pending but not post or cancel. Mint both scopes when you need the full loop (`operators:read` + `operators:write`). #### Cancelling a queued message `POST …/message` writes a file into the operator's on-box inbox (`$ACP_DIR/inbox` on the operator machine, fronted by `operator-ws`). The message sits there until the bridge is ready and drains it into a turn, and it stays withdrawable for a while after that: ```bash # See what is still outstanding curl -sS "$BW_API_URL/v1/operators/$OP_ID/pending" \ -H "Authorization: Bearer $BW_PAT" # Take one back. The handle is the item's `ts` from the list above; # POST /message returns no id. curl -sS -X DELETE "$BW_API_URL/v1/operators/$OP_ID/pending/1700000123456" \ -H "Authorization: Bearer $BW_PAT" # Or clear everything still sitting on disk curl -sS -X DELETE "$BW_API_URL/v1/operators/$OP_ID/pending" \ -H "Authorization: Bearer $BW_PAT" ``` **`/pending` answers "what am I still waiting on?", not "what is undelivered?"** It reports every message of yours the operator has not answered, and tags each one with the stage it reached: | `state` | Meaning | `withdrawable` | |---|---|---| | `on_disk` | still a file, the bridge has not taken it | yes | | `handed_off_unread` | the bridge has it, the operator has not seen it | yes | | `read` | the operator has it, no reply yet | no | | `answered` | replied to | counted in `states`, not listed | `count` and `outstanding` are both the length of `pending`, `states` carries the per-state counts (including `answered`), and `state_meanings` spells the same thing out inline. Only a `withdrawable: true` item can be taken back: DELETE on one that has been read answers **409** naming the state it reached, rather than claiming a withdrawal that did not happen. If the message is gone entirely, DELETE returns **404** `pending_not_found` — safe to retry; it will not corrupt an in-flight turn. There is no browser-only escape hatch required: the same callers that can send can list and cancel. #### Images an operator shares When an operator shows you a picture, the image is served straight from that operator's own machine — nothing is uploaded or stored on your account. The markdown in its reply points at `/v1/operators/{id}/files/{name}` (the browser uses the `/api` twin), and `{name}` is a plain filename: `png`, `jpg`, `jpeg`, `webp` or `gif`, up to 8 MB. These images are **temporary**. They live with the operator, so once it stops, the URL returns **409** and the picture is gone. Save anything you want to keep: ```bash curl -sS "$BW_API_URL/v1/operators/$OP_ID/files/chart.png" \ -H "Authorization: Bearer $BW_PAT" -o chart.png ``` #### Reading messages ```json { "state": "idle", "model": "…", "turns": [ { "role": "you|operator|note", "text": "…", "t0": 1700000000.0, "t1": null, "working": false, "index": 12 } ], "pending": [ { "ts": 1700000123456, "chars": 42, "preview": "first ~80 chars", "state": "on_disk", "withdrawable": true } ], "total_turns": 40, "limit": 50, "since": null, "since_mode": "tail", "cursor": { "next_index": 40, "next_since_t0": 1700000000.0, "has_more": false, "resume_index": "i:40", "resume_t0": 1700000000.0 }, "since_semantics": "see cursor fields; page with resume_index or resume_t0" } ``` | Field | Meaning | |-------|---------| | `state` | Session activity (e.g. `idle`, `working`, `booting`). | | `turns` | Conversation slice for this response. Roles: `you` (your messages), `operator` (replies), `note` (system notes). | | `pending` | Messages accepted but not yet shown as turns (still queued on the operator box under `$ACP_DIR/inbox`). Shape `{ts, chars, preview, state, withdrawable}` — oldest first, with **no `id`**. `ts` is the handle for `DELETE …/pending/{ts}`. Drop off once a real `you` turn appears; `GET …/pending` keeps tracking them past that point. | | `cursor` | Resume handles for the next poll/page. Prefer `resume_index` / `resume_t0`. | | `total_turns` | Full transcript length (not just this page). | **`since` / `limit` semantics** - **No `since`**: last `limit` turns (`since_mode=tail`) — good default for “what just happened?” - **`since` as turn index** — bare int < 1e9, or `i:N` / `index:N`: turns with absolute index ≥ N, then the first `limit` of those. Page with `cursor.resume_index` (e.g. `?since=i:40`). - **`since` as epoch seconds** — float, or int ≥ 1e9: turns whose `t0` is **strictly greater** than `since`, then first `limit`. Poll with `cursor.resume_t0` / `next_since_t0`. - Responses include enough `cursor` fields to resume without re-fetching the whole transcript. `has_more` is true when another page of index mode remains. - Very long turn texts may set `"truncated": true` and clip the body. Examples: ```bash # List operators curl -sS "$BW_API_URL/v1/operators" \ -H "Authorization: Bearer $BW_PAT" # POST a turn curl -sS -X POST "$BW_API_URL/v1/operators/$OP_ID/message" \ -H "Authorization: Bearer $BW_PAT" \ -H "Content-Type: application/json" \ -d '{"text":"ping from the front door"}' # Read recent turns (needs operators:read — write-only PATs get 403) curl -sS "$BW_API_URL/v1/operators/$OP_ID/messages?limit=5" \ -H "Authorization: Bearer $BW_PAT" # Read only what arrived after the last cursor curl -sS "$BW_API_URL/v1/operators/$OP_ID/messages?since=i:40&limit=50" \ -H "Authorization: Bearer $BW_PAT" ``` ### Waiting instead of polling Add **`wait=`** and the request is held open until the conversation moves past your cursor, so following a chat costs one request per reply rather than one per tick. If nothing arrives before the wait elapses you get a normal **200** with an empty `turns` — reissue immediately, no sleep and no backoff: ```bash CURSOR="" while :; do page=$(curl -sS -G "$BW_API_URL/v1/operators/$OP_ID/messages" \ --data-urlencode "wait=45" ${CURSOR:+--data-urlencode "since=$CURSOR"} \ -H "Authorization: Bearer $BW_PAT") echo "$page" | jq -e '.turns[]' >/dev/null 2>&1 && echo "$page" | jq '.turns' CURSOR=$(echo "$page" | jq -r '.cursor.resume_index // empty') done ``` | Param | Meaning | |---|---| | `wait` | Seconds to hold the request. Capped at **50**; `0` or omitted = immediate read (unchanged behaviour). | | `wait_for` | `turn` (default) returns as soon as **any** new turn lands, including one still being written. `idle` returns only once the reply is **complete** — use it for send-then-await. | `wait` is a **hard upper bound on the whole request**, enforced server-side around every upstream read — not just between them. Size your client timeout against it with a little slack and it will not fire. The **50s cap is set by the edge, not by this server**: requests through `*.benchwright.ai` are cut at 60s by the proxy in front of the API, which then *retries* the GET rather than failing it — so a longer wait costs you a 504 after ~180s and three duplicate reads on the origin. Asking for more than the cap is not an error; it is clamped. Poll in a loop rather than asking for one long wait. If the operator pod never answered inside the window you still get a **200** shaped like an empty page, with `"wait_timed_out": true` and the wait actually served as `"wait_effective_sec"`, so a wedged pod stays distinguishable from a quiet conversation. **Both fields appear only on that path.** An ordinary wait that simply elapsed with nothing to say carries neither, so treat their absence as "the conversation was quiet" rather than reading `wait_effective_sec` as a routine echo of your request. Your own `since` is echoed back as `cursor.resume_index` — never a fabricated cursor — so resuming from that response cannot skip turns. Omit `since` and the wait anchors to the newest turn, i.e. "block until the **next** thing is said". Pass `cursor.resume_index` from the previous response to resume exactly where you left off. Send a turn and block for the finished reply in one step: ```bash curl -sS -X POST "$BW_API_URL/v1/operators/$OP_ID/message" \ -H "Authorization: Bearer $BW_PAT" -H "Content-Type: application/json" \ -d '{"text":"what is the pass rate on the last run?"}' curl -sS -G "$BW_API_URL/v1/operators/$OP_ID/messages" \ --data-urlencode "since=$CURSOR" --data-urlencode "wait=45" \ --data-urlencode "wait_for=idle" -H "Authorization: Bearer $BW_PAT" ``` Read the cursor **before** sending, so a fast reply can't land in the gap. Keep each request under any proxy idle timeout on your side (45s is a safe default). From the CLI this is already wired up: ```bash benchwright operator msg yara "ship it" --wait 5m # blocks, prints the reply benchwright operator log yara --follow # server-side wait, no ticker ``` ### Streaming the conversation (`/events`) — prefer this over `wait` `GET /v1/operators/{id}/events` pushes turns as they happen over **Server-Sent Events**. One connection, no polling, no 50s ceiling: ```bash curl -sN "$BW_API_URL/v1/operators/$OP_ID/events" -H "Authorization: Bearer $BW_PAT" ``` ``` id: i:14 event: ready data: {"cursor":{"resume_index":"i:14"},"wait_for":"turn","heartbeat_sec":15} : ping id: i:16 event: turns data: {"state":"idle","turns":[…],"cursor":{"resume_index":"i:16"}} ``` | Event | Meaning | |---|---| | `ready` | Stream open. Carries the cursor it anchored to. | | `turns` | New turns. Body is the same shape as `GET /messages`. | | `error` | Upstream failed mid-stream; the stream then closes. | | `bye` | Server asked you to reconnect (`reason` says why). Reconnect from your last `id`. | | `: ping` | Keep-alive comment every 15s. Ignore it. | **Cursors.** No `since` anchors to the newest turn — you get what's said *next*, not a replay of history. `since=i:N` replays from N. Every event carries an `id:`, which a browser `EventSource` echoes back as `Last-Event-ID` automatically, so reconnects resume exactly where you left off without tracking anything yourself. `wait_for=idle` streams only completed replies. **Why this exists.** `wait` is capped at 50s because our edge cuts any request that stays silent for 60s — and then *retries* it, so a longer wait costs you a 504 after ~180s and duplicate work upstream. That limit is on **silence**, not duration: a stream that keeps emitting is never cut. The 15s heartbeat is what makes `/events` immune, so don't strip it if you proxy this. Note a browser `EventSource` can't set an `Authorization` header, so from a browser you need a same-origin session rather than a PAT. --- ## 13. Rate limits and quotas Read + write quotas are per **account** (not per token — multiple PATs share the same bucket), and scale with your membership tier. Team members count against the team owner's bucket. | Tier | Reads / min | Writes / min | |--------|-------------|--------------| | free | 60 | 10 | | entry | 300 | 60 | | growth | 600 | 120 | | scale | 1 200 | 240 | Tunable in `public.membership_tiers.{read_rpm,write_rpm}` — operators can revise the bands without a code change. Run-shape quotas (separate from API rpm): - Concurrent running runs / account: 1 (free), 3 (entry), 15 (growth), 50 (scale) — see `membership_tiers.max_run_concurrency`. Headers returned on every **authenticated** response (an anonymous read of a public run or of the catalog carries none, because there is no account to count against): ``` X-RateLimit-Limit: 300 X-RateLimit-Remaining: 247 X-RateLimit-Reset: 1713888360 ``` 429s include `Retry-After` in seconds and a problem+json body. Callers can read their own live counters: ``` GET /v1/users/me/rate-limits ``` ```json { "user_id": "", "read": { "limit": 300, "count": 53, "remaining": 247, "reset_seconds": 22, "window_seconds": 60 }, "write": { "limit": 60, "count": 2, "remaining": 58, "reset_seconds": 22, "window_seconds": 60 } } ``` For team members the counters are the team's shared bucket, not a private shadow. Admin can raise per-account concurrency and read ceilings; rate limits apply at the account level regardless of how many PATs the account holds. --- ## 14. Error reference Stable `code` values clients should handle: | HTTP | code | Meaning | |------|-----------------------------------|------------------------------------------------------| | 400 | `invalid_argument` | Malformed request body or parameters | | 400 | `unknown_provider` | `model.provider` not a pinnable provider | | 400 | `unknown_harness` | `harness` not in the enum | | 400 | `bridge_renamed_to_harness` | Legacy `bridge` field sent; use `harness` | | 400 | `unknown_aggregator` | `trials.aggregator` name not recognized | | 400 | `autonomous_model_required` | `metadata.autonomous` without an explicit model pin | | 400 | `parent_still_running` | Replay/fork of a run that has not finished | | 400 | `compute_not_configured` | Requested compute backend has no configuration | | 400 | `unknown_filter_field` | Unknown `?expand=` key on `GET /v1/billing/balance` | | 401 | `auth_required` | Missing or invalid Authorization header | | 403 | `scope_insufficient` | Token scope does not cover this operation | | 403 | `admin_required` | A pinned `driver` or `phase_models` spec outside the accepted set (a priced Gemini model, or `openrouter://…`) | | 403 | `forbidden` | Operation on a resource the caller does not own | | 403 | `forbidden_team_member` | Team member changing billing controls (owner only) | | 404 | `not_found` | Resource does not exist, or visibility hides it | | 409 | `idempotency_conflict` | Same key, different body | | 409 | `visibility_constraint_violated` | Publish on a run that is not `ok`, or has no benchmark | | 409 | `invalid_state` | Pause on a run that is not `running` | | 409 | `already_halted` | Pause on a run that is already halted | | 409 | `not_paused` | Resume on a run not in `operator_pause` | | 409 | `subset_frozen` | Append to a frozen (non-rolling) subset | | 409 | `name_taken` | Registry name already in use | | 409 | `name_conflict` | Secret with that name already exists | | 409 | `cloud_managed` | Secret is platform-managed under Cloud inference | | 409 | `already_on_team` | Create a team while already on one | | 412 | `precondition_failed` | `If-Match` ETag mismatch | | 412 | `egress_endpoint_unavailable` | Pinned local-egress machine is not serving (§9a.2) | | 400 | `no_provider_secret` | No provider API key in the vault at all | | 412 | `no_provider_secret` | Pinned provider's specific key missing from the vault | | 402 | `insufficient_balance` | Balance < $1 launch floor | | 429 | `rate_limited` | See `Retry-After` | | 429 | `tier_concurrency_exceeded` | Account at its tier's concurrent-run cap | | 501 | `resume_not_supported` | Pause cleared but the Driver cannot relaunch here | | 501 | `e2b_not_configured` | No compute backend configured on this deploy | | 501 | `not_configured` | The feature isn't wired on this deploy | | 501 | `not_supported` | The host cannot serve this operation (e.g. SSE) | | 502 | `upstream_unavailable` | Postgres, E2B, or Stripe not reachable | | 502 | `upstream_invalid` | Upstream answered, but not with something parseable | | 502 | `mint_failed` | Could not mint an upstream credential | | 502 | `subset_insert_failed` / `impl_insert_failed` | Registry write reached the database and failed | | 503 | `narration_unavailable` | LLM narration transiently offline | Also emitted, and worth handling where you touch the surface that raises them: `403 team_readonly` (the general team-write guard, §2.5), `409 fork_unsupported` (§4.3), `409 replay_unsupported` (§4.2.3), `409 delete_blocked` (§4.7), and `409 superseded` (§8.9). **Not every failure is problem+json.** An unhandled server error is served by the framework's default handler, so a `500` carries no `code` and no `request_id`. Key off the status alone there, and never assume a JSON body on a 5xx. There is also no `422` on this API: validate typed fields client-side, since bad values come back as `400 invalid_argument`. **501 vs 503.** `501` means *this deploy doesn't have that feature wired* — a permanent answer for the deploy you're talking to, so retrying won't help. `503` is reserved for a backend that IS configured but is currently down or degraded; that one is worth retrying. --- ## 15. Event type reference The set below mirrors `ai.benchwright.port.pipeline.events.PipelineEvent` and is what shows up inside the SSE stream and the `/events` JSON array. | Type | Produced by | Notable fields | |---------------------------|-----------------------|----------------------------------------------------| | `RunLaunched` | RunLauncher seq=0 | sandbox_id, template, agent, request | | `PipelineStarted` | AgentPipeline | model, request | | `PipelineFinished` | AgentPipeline | benchmark_name, duration_ms, error? | | `PhaseStarted` | AgentPipeline | phase, invocation | | `PhaseCompleted` | AgentPipeline | phase, duration_ms | | `PhaseOutput` | phase impl | phase, text (short line) | | `PhaseFailed` | phase impl | phase, error | | `PhaseRestartRequested` | phase impl | phase, reason | | `PhasePaused` | AgentPipeline | phase, reason, detail | | `PhaseResumed` | AgentPipeline | phase, leg | | `LlmCallEvent` | LLM wrapper | agent, model, input_tokens, output_tokens, cost_usd | | `NarrationLlmCall` | narration (server) | source, slice, model, input/output_tokens, cost_usd | | `ToolCallEvent` | agent | tool, args (summarized) | | `ToolResultEvent` | agent | tool, result (summarized) | | `AgentTextEvent` | agent | phase, text | | `AgentNudged` | AgentPipeline | phase, attempt, max_attempts, detail | | `SandboxCreated` | provider | sandbox_id, role, metadata | | `SandboxKilled` | provider | sandbox_id, cost_usd, duration_ms | | `SandboxHealthWarning` | provider | sandbox_id, reason | | `SandboxAutoPaused` | provider | phase, sandbox_ids, correlation_id | | `SandboxAutoResumed` | provider | phase, sandbox_ids, paused_duration_ms, canary_ok | | `CommandStarted` | sandbox | command (truncated) | | `CommandCompleted` | sandbox | exit_code, duration_ms | | `FileWritten` | sandbox | path, size_bytes | | `SnapshotCreated` | provider | snapshot_id | | `FetchArtifact` | FetchPhase | url, size_bytes | | `AnalyzeArtifact` | AnalyzePhase | dataset shape | | `DatasetTasksArtifact` | AnalyzePhase | subset_fingerprint, datasets, total_rows | | `SetupArtifact` | SetupPhase | entrypoints | | `SyntheticSubsetCreated` | AtomizePhase | subset_id, recipe | | `ModelProxyArtifact` | AttachPhase | model, config_yaml, proxy_url | | `ModelProxyCredentials` | AttachPhase | ui_url, ui_username | | `ProxySandboxLinger` | AgentPipeline | sandbox_id, requested_seconds, lingered, detail, granted_expires_at_ms | | `VaultTunnelWired` | AgentPipeline / Materialize | sandbox_id, role, catch_all, catch_all_installed, host_count, hosts | | `LocalEgressAttached` | AgentPipeline | egress_ip, attach_timeout_sec | | `PendingAtomizeCode` | AtomizePhase | content, invocation | | `AtomizeArtifact` | AtomizePhase | name, content, mime, invocation | | `MaterializeTaskResult` | MaterializePhase | task_id, index, score, duration_ms, output_preview | | `MaterializeTaskAggregated` | MaterializePhase | task_id, index, score, aggregator, trial_scores | | `MaterializeStoppedEarly` | MaterializePhase | reason, invocation, stopped_at | | `ExecutionProgress` | any phase | message, percent? | | `ReportFile` | ReportPhase | name, content, mime | | `HumanReviewRequested` | any phase | phase, reason, question | | `HumanReviewResolution` | resumed run | phase, instructions | | `LimitExceeded` | AgentPipeline | kind, limit, observed | | `ErrorEvent` | anywhere | phase?, error | > **`ModelProxyCredentials` no longer carries the proxy UI password.** It used > to, in plain text, which meant publishing a run published a live credential — > any `runs:read` token, and once the run was `public` or `unlisted`, anyone at > all. The password is now held server-side and never appears on an event; the > event keeps only the non-secret `ui_url` + `ui_username`, and the events API > drops secret-named fields from any older event that still has one. Reaching > the proxy dashboard goes through `/litellm/`, which authorizes the > caller. `ProxySandboxLinger` extends the window that proxy stays reachable. --- ## 16. Worked examples Two end-to-end workflows that exercise the API. ### 16.1 Launch a run, stream to terminal, gate CI on score ```bash # 1. Prerequisites (one-time): add a provider secret + rule. curl -sS {{site}}/v1/secrets \ -H "Authorization: Bearer $BW_PAT" \ -H "Idempotency-Key: setup-openrouter-v1" \ -d '{ "name": "OPENROUTER_API_KEY", "value": "sk-or-…", "rules": [{ "host_pattern": "openrouter.ai", "inject_kind": "bearer", "inject_target": "Authorization" }] }' # 2. Top up if needed. curl -sS {{site}}/v1/billing/topups \ -H "Authorization: Bearer $BW_PAT" \ -d '{"amount_cents": 5000}' # open the returned `url` in a browser, pay # 3. Launch a run. Record $RUN in the CI job to correlate later — # launch metadata is not echoed back (§4.2.1 natural-language notes). RUN=$(curl -sS {{site}}/v1/runs \ -H "Authorization: Bearer $BW_PAT" \ -H "Idempotency-Key: ci-build-8812" \ -d '{ "request": "Run MMLU Abstract Algebra on openai/gpt-4o-mini via OpenRouter", "model": { "model": "openai/gpt-4o-mini", "provider": "openrouter" }, "constraints": { "max_cost_cents": 500 }, "metadata": { "autonomous": true, "limit": 100 } }' | jq -r .id) # 4. Stream events (CLI sugar; under the hood this is SSE). curl -N {{site}}/v1/runs/$RUN/events:stream \ -H "Authorization: Bearer $BW_PAT" # 5. When the stream closes, pull the summary. curl -sS {{site}}/v1/runs/$RUN/tasks | jq .summary # { "total": 100, "passed": 82, "pass_rate": 0.82, … } # 6. Exit non-zero if below gate. SCORE=$(curl -sS {{site}}/v1/runs/$RUN/tasks | jq -r .summary.pass_rate) awk -v s="$SCORE" 'BEGIN { exit (s < 0.80) }' ``` ### 16.2 Fork to iterate Your run came back with a scoring bug. The fix is in a description, not a code patch: ```bash PARENT=$RUN FORK=$(curl -sS {{site}}/v1/runs/$PARENT/fork \ -H "Authorization: Bearer $BW_PAT" \ -d '{ "description": "Change score() to normalize whitespace before comparing", "from_phase": "atomize" }' | jq -r .id) # Same streaming and summary endpoints work on the fork. ``` --- ## 17. Deprecation and stability **`/v1` is not frozen yet.** Benchwright is pre-GA, and until it ships generally available the API can change shape without a version bump and without a deprecation window. If you are building against it now, pin nothing you cannot re-pin, and read this document rather than assuming a field you saw last month is still there. What already holds: - Additive changes (new fields, new endpoints, new event types) ship at any time. Clients must tolerate unknown fields. - Ids, error `code` values and event `type` names are the parts we are most careful with, because they end up in your code rather than on a screen. What starts at GA: - `/v1` freezes against breaking changes. - Deprecations are announced on the response via: ``` Sunset: Tue, 01 Jan 2027 00:00:00 GMT Deprecation: true Link: ; rel="deprecation" ``` at least 6 months before removal. (A machine-readable `/v1/status` deprecation feed is planned but not yet available.) - New resources land under `/v1/preview/...` first, with no SLA, for at least one month before promotion. - `/v2` will exist only if there is a shape we cannot evolve backwards-compatibly. When it ships, `/v1` receives security fixes and continues to serve for 18 months. --- ## Appendix B. OpenAPI An OpenAPI 3.1 spec lives at: ``` GET {{site}}/v1/openapi.json ``` The spec is hand-maintained (`api/src/main/resources/openapi.json`), and it currently lags the route table. The whole `/v1/operators` surface, `/v1/users/me/rate-limits`, `/v1/billing/subscription`, `/v1/billing/verify-subscription`, teams, local egress and egress endpoints are all absent from it. Use it as a reference for the paths it does describe; **this document is the authoritative list of what `/v1` serves.** The SDKs are hand-written rather than generated from it, and CI checks only that the spec parses and still describes `/v1/runs`, so a route missing from the spec is drift, not a signal that the route is unsupported.