Benchwright API

Last modified September 10, 2026

Base URL: https://benchwright.ai/v1/.

Every request and response is JSON, UTF-8, Content-Type: application/json (or application/problem+json for errors, or text/event-stream for SSE streams).

1. Versioning and headers

Version lives in the path: /v1/.... New fields may be added to responses without a version bump. Removing or renaming fields, changing semantics, or tightening validation requires /v2.

Clients should always send:

User-Agent: <product>/<version> (<runtime>)
Accept: application/json
Accept-Encoding: gzip, br

Responses include:

X-Benchwright-Request-Id: req_01HZ…
X-Benchwright-Api-Version: 1

Treat API responses as uncacheable; the server mostly does not say so. No Cache-Control header is emitted on ordinary /v1 responses at all, so an intermediary applying heuristic freshness could serve you a stale run. The two places it is stated explicitly are registry reads (Cache-Control: no-store) and SSE streams (Cache-Control: no-cache). Use the ETags in §3.8 for revalidation rather than relying on cache headers.

2. Authentication and scopes

2.0 Two prefixes: /v1 and /api

There are two HTTP surfaces and they authenticate differently. Knowing which one you are on explains most unexpected 401s.

/api is not a public API. It backs the pages you see when signed in, so it expects a session and will reject a token. Calling /api/... with a PAT returns 401, and that is the endpoint working as intended rather than a fault. Every /api route that is meant for programs has a /v1 equivalent, and those are the ones to use. For example, read a run's routing at GET /v1/runs/{id}/egress-routes rather than /api/runs/{id}/egress-routes.

The reverse also holds. A few operations require a browser session and refuse PATs on purpose, token creation among them, for the reason in §2.4.

2.1 Personal access tokens (PATs)

A user provisions tokens under /manage (API Keys) or POST /v1/tokens (API; browser session required, see §2.4). Tokens look like:

bw_live_pat_8f7a9…e3

The token is shown once at creation. Subsequent reads surface only the prefix + last 4, a label, scopes, and last-used timestamp.

Auth header:

Authorization: Bearer bw_live_pat_8f7a9…e3

Storage: the server keeps only SHA-256(token) — a DB dump cannot be replayed. last_used_at is bumped on every authenticated request so the web UI can surface dormant tokens for cleanup.

Revoked tokens are soft-deleted (revoked_at stamped) so audit queries retain "this token was live between X and Y". Rotate produces a fresh value under the same id; the previous hash is overwritten atomically.

For CI, provision a PAT under a shared service user account.

2.2 Scopes

Tokens declare the minimum privilege they need. benchwright token create defaults --scope to runs:read,runs:write,artifacts:read,secrets:read,billing:read,registry:read — registry:read is included so the default token can browse the catalog (bench ls and friends) under the §8 access rules. (benchwright login mints nothing; it stores a PAT you paste in, with whatever scopes that token already has.) POST /v1/tokens validates scopes against the known list and rejects wildcard grants (e.g. runs:*) with 400 invalid_argument; a token that already carries a wildcard is still honored at enforcement time, but wildcards cannot be minted through the API.

Registry reads (§8.1–§8.3) work unauthenticated — the public catalog is open to anyone, and a browser (JWT) session is full-access as usual. registry:read only gates PATs: a token that authenticates without it is rejected (403 scope_insufficient) even for public rows, so a CI token scoped to runs can't quietly enumerate the catalog. Anonymous access also extends to the per-run read surface of public and unlisted runs: run detail, events (poll and SSE), tasks, task payloads, artifacts and their content, fingerprint, narrate, and report all serve without credentials when the run's visibility allows it.

A 403 scope_insufficient response carries a structured required_scope field naming the exact scope the call needs, so a client can render an actionable "mint a token with this scope" hint without parsing detail.

2.3 Provider credentials

Keys needed by the pipeline itself (OpenRouter, OpenAI, Anthropic, Google, or arbitrary HTTP services the Driver reaches) live in the Secrets API (§9), not in the PAT. Rotating your OpenRouter key does not invalidate your PAT and vice versa.

2.4 Token lifecycle

GET    /v1/tokens/me               # what am I? identity + the scopes I hold
GET    /v1/tokens                  # list the caller's PATs
POST   /v1/tokens                  # create (returns `token` once)
PATCH  /v1/tokens/{id}             # set or clear a per-token spend cap
GET    /v1/tokens/{id}/spending    # per-token spend rollup
POST   /v1/tokens/{id}:rotate      # mint a new value, keep id+label+scopes
DELETE /v1/tokens/{id}             # revoke (soft-delete, audit-preserving)

GET /v1/tokens/me answers "who is this credential, and what may it do": auth type, user id, email, role, whether the caller is full-access, and the scope list. It is the only way a PAT can enumerate its own scopes (the list endpoint shows tokens, and a token cannot see itself listed with certainty), so it is the right preflight for a client that wants to fail with "this token is missing runs:write" instead of a 403 halfway through a workflow.

Token mutation (create, spend-cap patch, rotate, delete) is browser-session-only: a PAT calling any of these gets 403 scope_insufficient, with the explanation in title and the bare operation name in detail. GET /v1/tokens, GET /v1/tokens/me and GET /v1/tokens/{id}/spending work with a PAT.

This limit is deliberate, and it should stay. A token that could mint tokens would let anyone who obtained one quietly issue themselves a second, longer-lived, wider-scoped credential, and revoking the original would not take that away. Signing in is the one step that proves a person is present, so it is the step that gates minting.

The reason worth writing down is what makes the limit affordable. Recovering from trouble must never require minting a token, because the moment it does, this boundary is the thing standing between you and your own account. Every recovery path is therefore reachable with an ordinary PAT: operators:write creates, starts, and shuts down operators (§12a), runs:write halts runs. If you ever find yourself needing a new token purely to get out of a hole, that is a missing endpoint rather than a reason to loosen this.

GET /v1/tokens returns metadata only — the secret is never echoed on list responses. The full secret is returned exactly once, in the token field of the create response and the rotate response.

GET /v1/tokens
{
  "data": [
    {
      "id": "<token_uuid>",
      "label": "ci",
      "scopes": ["runs:read", "runs:write", "artifacts:read"],
      "prefix": "bw_live_pat_8f7a",
      "last4": "3c91",
      "created_at": "2026-02-14T…Z",
      "last_used_at": "2026-04-23T14:00:04Z",
      "spend_cap_cents": 5000,
      "cap_period": "month",
      "spent_cents_period": 312,
      "spend_period": "month",
      "expires_at": "2026-12-31T00:00:00Z",
      "expired": false
    }
  ],
  "meta": { "payer": { /* whose wallet this caller bills to, §2.5 */ } }
}

Token ids are bare UUIDs. Revoked tokens are filtered out of the list entirely (there is no revoked_at field on responses).

Spend caps. POST /v1/tokens and PATCH /v1/tokens/{id} both take {"spend_cap_cents": 5000, "cap_period": "month"}, which caps what runs launched with that token can spend. Set it at creation when you can: a cap added afterwards leaves a window the token spends without one. cap_period is one of day, month, lifetime (default month); on PATCH, {"spend_cap_cents": null} clears both the cap and its period. GET /v1/tokens/{id}/spending?period=month returns {token_id, period, spent_cents, spend_cap_cents, cap_period} — the same number the list response carries inline as spent_cents_period.

Expiry. PATCH /v1/tokens/{id} with {"expires_at": "2026-12-31"} gives a token a deadline, and POST /v1/tokens accepts the same field at creation. A bare date means the END of that day UTC, so a token you set to expire today still works for the rest of today; a full ISO-8601 timestamp is taken literally. {"expires_at": null} clears the deadline. The value must be in the future: to retire a token now, revoke it.

Expiry is checked when the token authenticates, not by a background sweep, so it takes effect on the very next request. An expired token keeps its row, so it stays visible in GET /v1/tokens with expired: true rather than vanishing the way a revoked one does. That distinction is the point: "this lapsed on Tuesday" and "somebody revoked this" are different incidents.

A request carrying an expired token gets its own error rather than the generic one, because the fix is different (the value is fine, the date is not) and auth_required would send you to re-check your copy-paste:

401 {
  "code": "token_expired",
  "title": "API token expired",
  "detail": "This token expired on 2026-08-11. It has not been revoked, so …",
  "expired_at": "2026-08-11T00:00:00+00:00"
}

This short-circuits: presenting an expired token fails even if the request also carries a session cookie that would have worked. You named a credential, so we answer about that credential rather than quietly serving you under another one. A string that was never a token of ours is not a credential and still falls through to whatever else you have.

Rotation is for compromise recovery: it overwrites the stored hash in place, so the previous value stops authenticating immediately. It replaces the secret, not the grant — scopes, spend cap and expiry all ride through untouched, so rotating an expired token returns a value that still does not authenticate (the response says so with expired: true). Revoke (DELETE) soft-deletes the row for audit; the next authenticated request with that token gets 401 auth_required.

2.5 Teams

A team is a shared workspace: members see each other's runs in GET /v1/runs (team-scope widening), and usage bills to the team owner's wallet. Billing-, token-, and secret-shaped responses carry a meta.payer object naming whose wallet the call is scoped to.

A member who may read the team's resources but not change them gets 403 team_readonly on a write. That is the general guard, and the code to handle. (403 forbidden_team_member exists at exactly one endpoint, the billing-controls PATCH, where only the owner may set the cap and refill.)

409 already_on_team comes from POST /v1/teams: you cannot create a team while you already belong to one. An invite you are not allowed to send or accept is 403 forbidden.

GET    /v1/teams/me                   # my team: team, members, invites, balance, is_owner
POST   /v1/teams                      # create a team (caller becomes owner)
POST   /v1/teams/me/invites           # invite by email to my team
POST   /v1/teams/{id}/invites         # invite by email to a specific team
POST   /v1/team-invites/{id}:accept   # accept a pending invite
POST   /v1/team-invites/{id}:revoke   # revoke a pending invite
POST   /v1/teams/me:leave             # member leaves (owners cannot leave)
POST   /v1/teams/me:delete            # owner dissolves the team

GET /v1/teams/me returns {team, members, invites, balance, is_owner}; team is null when the caller has no team. Owners cannot :leave (they are the payer); me:delete dissolves the workspace: all members are removed, pending invites are cancelled, and shared runs revert to being private to whoever launched them.

3. Conventions

3.1 Identifiers

Every resource id is a bare string. The route segment names the resource kind, so the id itself doesn't have to.

Subset and impl ids are content-addressed, and the trailing 12 hex characters are the first 12 of the content hash itself: of the task set for a subset, of the code for an impl. The fingerprint field on an impl repeats those same 12 characters, and the full hash is carried separately as impl_fingerprint. So two ids that share a prefix and differ in the hash are two versions of the same thing under one benchmark, which is what makes them comparable. Task / event / artifact ids encode the natural key as base64url so they round-trip through URLs and the server can decode them back into (run, seq) or (run, invocation, index) without a separate lookup table. UUID-shaped ids are validated for shape; bad shapes 400.

3.2 Timestamps

RFC 3339 strings in UTC:

"started_at": "2026-04-23T14:02:11.371738+00:00"

Parse them as RFC 3339, don't string-match on Z. The UTC offset is normally written +00:00 rather than Z, and the fractional part is typically microseconds (6 digits) rather than milliseconds. Both spellings are valid RFC 3339 and both may appear, so anything that slices the string or compares suffixes will break on one of them.

Duration fields are _ms integers (duration_ms). Never floats.

3.3 Money

Cents as integers, currency as ISO-4217, and (on run costs) an exact amount_usd beside them:

"cost": { "amount_cents": 4137, "currency": "USD", "amount_usd": 41.3719 }

amount_cents is a whole-cent integer and never mixes units. It is also lossy in one direction on purpose: any non-zero cost rounds up to at least 1¢, so real spend never reports as free. That makes it the wrong field to reconcile with, because a sub-cent figure is indistinguishable from a nearly-1¢ one once it is rounded.

amount_usd is the exact figure. It is a float carrying the underlying fractional-cent value, and it is what to reconcile a run against its billing_history rows. Every cost entry on a run (§4.1) carries it. Money objects on the billing endpoints (§10) are whole cents only, rounded half-up.

3.4 Pagination

Run, event, task, and artifact lists use cursor pagination:

GET /v1/runs?limit=50&page_token=<opaque>

Response:

{
  "data": [ /* up to 50 items */ ],
  "pagination": {
    "has_more": true,
    "next_page_token": "<opaque>"
  }
}

limit defaults to 50 and caps at 100 on these lists (the event stream is the exception, §5.1: 500 and 1000). A larger value is silently clamped, not rejected, so a client asking for 500 gets 100 and a next_page_token. Follow the cursor rather than trusting that you asked for everything. Registry lists (§8) clamp at 200.

next_page_token is opaque. Treat it as a blob to echo back, not as something to construct or parse: what it encodes (a sequence position, or a timestamp plus an id, depending on the list's ordering) is an implementation detail that can change without a version bump.

Cursor-paginated lists never report a total. Offset-paginated lists — GET /v1/billing/transactions — take limit / offset instead and include a best-effort total_estimate (PostgREST count=estimated); clients must not rely on it for correctness. Registry lists (§8) also use limit/offset where they paginate at all.

3.5 Filtering and sorting

Operators on endpoints that declare them:

?status=running             # exact match
?status=in:running,ok       # set membership
?started_at=gte:2026-04-01  # comparison
?sort=-started_at,cost_billed  # comma list, `-` reverses

The comparison prefixes are gte:, lte:, gt: and lt:; like: and ilike: do pattern matching (case-sensitive and case-insensitive). A value with no recognized prefix is an exact match.

The set of filterable and sortable fields is documented per endpoint. Unknown filter keys are silently ignored, not rejected. Text search exists only where an endpoint declares it — ?q= on the registry catalog (§8.1); there is no general search operator.

3.6 Partial responses and expansion

?fields= (partial responses) is not yet available; responses always return the full shape.

?expand= exists on exactly two endpoints:

GET /v1/runs/<uuid>?expand=phase_breakdown,child_forks
GET /v1/billing/balance?expand=topups,usage_trend,counters,tiers,subscription

An unknown ?expand= key on GET /v1/billing/balance is rejected with 400 unknown_filter_field. Expansion never recurses more than one level.

3.7 Idempotency

Any POST, PATCH, PUT, or DELETE that creates or mutates a resource accepts:

Idempotency-Key: 4f1c7a1a-3b2c-4f6b-9b8a-123456789abc

The server caches the response status + body for 24 hours, keyed by (token_id, key). Retrying with the same key returns the original result with an Idempotent-Replayed: true header on the reply. Conflicting retries (same key, different request body or path) get 409 idempotency_conflict. A retry that arrives while the first request is still in flight also 409s rather than risk running the handler twice.

Launch caveat (POST /v1/runs): a successful acceptance (202) is cached even when the run later dies at boot (status=error). A bare retry with the same key therefore returns that failed run's id with Idempotent-Replayed: true and does not launch a new run — which is dangerous right after an infra fix. The benchwright CLI heals this: on replay it GETs the run and, if the run is missing or terminal-failed (error / failed / cancelled), relaunches once under a derived key and prints a stderr note. Raw HTTP clients should do the same, or pass a fresh Idempotency-Key / omit the header after a failed attempt. Still- running or ok replays stay as intentional dedupe.

Keys longer than 256 chars are rejected with 400 invalid_argument. JWT (browser) callers share the cache keyed by user_id instead of token_id — rotating a PAT invalidates any outstanding keys tied to it.

3.8 Concurrency tokens

GET on mutable resources returns a weak ETag:

ETag: W/"<uuid>:1761234800123"

The digits after the colon are the row's updated_at in epoch milliseconds. ETags are weak (W/ prefix) because response JSON isn't byte-stable — two GETs match semantically as long as the row hasn't moved.

PATCH accepts If-Match to gate the write on the row still being at the expected revision:

If-Match: W/"<uuid>:1761234800123"

On mismatch the server responds 412 precondition_failed and sets the response ETag to the current value so the client can re-fetch and retry. If-Match is optional — omit it to opt out of the concurrency check. If-Match: * also bypasses the check and matches any current row.

Enforced today on /v1/runs/{id} (PATCH), /v1/benchmarks/{id} (PATCH), and both build-subset PATCHes (/v1/build/subsets/{id}/recipe and /tasks, §8.9), which also serve the matching ETag on their GET. Other mutable resources (tokens, secrets, host policies, catalog subsets, impls) accept If-Match permissively: the header is read and ignored, so a stale write there succeeds rather than answering 412. They will begin enforcing as their handlers migrate; clients that always send the ETag they last saw will not regress.

3.9 Errors

All 4xx/5xx responses follow RFC 9457 (problem+json):

{
  "type": "https://docs.benchwright.ai/errors/insufficient_balance",
  "title": "Balance below the $1.00 launch floor",
  "status": 402,
  "code": "insufficient_balance",
  "detail": "Top up to at least $1.00 before launching. The pipeline halts a run the moment a live debit would drop balance under $1.",
  "request_id": "req_01HZ…",
  "retryable": false
}

Clients should key off code, not detail. Two error codes add structured fields beside the standard ones: 403 scope_insufficient carries required_scope, and 409 delete_blocked carries id, deleted, halt_reason and sandboxes_killed (§4.7).

Exceptions, where the body is a plain {"error": "…"} rather than problem+json: the billing endpoints that delegate to the legacy billing handlers (top-ups, subscription, Stripe webhook), and the whole /v1/operators/* surface, which bridges to the same handlers the web app uses. Read error as the message on those, and see §14 for what a bare 500 looks like.

3.10 Null vs. missing

null means "server knows this field, value is absent". Missing keys mean "not applicable in this response shape". Clients that deserialize into typed structs treat both as None.

4. Resource: Runs

A run is the top-level unit of work: one pipeline execution from an initial request to a final report. Runs are the most important resource in the API; everything else is either a child of a run or a setting that configures future runs.

4.1 The Run object

{
  "id": "608721fe-54f1-43e3-ab9b-7e0269783f70",
  "object": "run",
  "user_id": "<user_uuid>",
  "status": "running",
  "display_status": "running",
  "current_phase": "materialize",
  "halted": false,
  "halt_reason": null,

  "request": "Run MMLU Abstract Algebra on qwen3-32b via OpenRouter",
  "driver": "openrouter://deepseek/deepseek-v4-flash",

  "benchmark": null,
  "benchmark_name": "MMLU",
  "benchmark_id": "<benchmark_id>",
  "subset": null,
  "subset_id": "<subset_id>",
  "impl": null,
  "impl_id": "<impl_id>",
  "model": { "identifier": "qwen/qwen3-32b", "source": "openrouter", "provenance": "specified" },
  "sut_config": { "temperature": 0.0, "max_tokens": 2048 },
  "backend": "sandbox0",
  "harness": "benchwright",
  "harness_config": null,
  "fingerprint_hash": "c8a1b2d3e4f5",

  "score": null,
  "task_count": 100,
  "cache_hits": { "count": 2, "phases": { "fetch": "<uuid>", "analyze": "<uuid>" } },
  "materialize_indices": "0-99",
  "canonical_indices": null,

  "cost": {
    "incurred": { "amount_cents": 412, "currency": "USD", "amount_usd": 4.1174 },
    "driver":   { "amount_cents": 291, "currency": "USD", "amount_usd": 2.9083 },
    "model":    { "amount_cents": 38,  "currency": "USD", "amount_usd": 0.3791 },
    "sandbox":  { "amount_cents": 121, "currency": "USD", "amount_usd": 1.2062 },
    "billed":   { "amount_cents": 495, "currency": "USD", "amount_usd": 4.9412 },
    "proxy_linger": { "amount_cents": 1, "currency": "USD", "amount_usd": 0.0027 },
    "narration": { "amount_cents": 3, "currency": "USD", "amount_usd": 0.0284, "calls": 7, "input_tokens": 42000, "output_tokens": 180 }
  },
  "tokens": {
    "input": 38211, "output": 12402, "total": 50613,
    "driver": { "input": 4712000, "output": 12000, "total": 4724000 }
  },
  "duration_ms": 421003,

  "visibility": "private",
  "forked_from": null,
  "forked_from_deleted": null,
  "fork_description": null,
  "relaunch_count": 0,
  "relaunch_budget_sec": null,
  "port_version": "0.5.15",
  "storage": {
    "event_count": 1204, "event_bytes": 882133,
    "payload_count": 100, "payload_bytes": 4118220,
    "total_bytes": 5000353
  },

  "created_at":  "2026-04-23T14:00:02.003Z",
  "started_at":  "2026-04-23T14:00:04.118Z",
  "finished_at": null,
  "updated_at":  "2026-04-23T14:07:05.889Z",

  "error": null,
  "error_type": null,
  "review_resolution": null,

  "inference": {
    "calls": 100, "ok": 98,
    "completion_tokens": 50000, "duration_ms": 120000,
    "by_model": [
      { "model": "qwen/qwen3-32b", "calls": 100, "ok": 98, "completion_tokens": 50000,
        "duration_ms": 120000, "status_codes": { "200": 98, "429": 2 } }
    ]
  },

  "links": {
    "self":       "/v1/runs/<uuid>",
    "events":     "/v1/runs/<uuid>/events",
    "artifacts":  "/v1/runs/<uuid>/artifacts",
    "tasks":      "/v1/runs/<uuid>/tasks",
    "report":     "/v1/runs/<uuid>/report",
    "dashboard":  "https://benchwright.ai/run/<uuid>"
  }
}

Every run has the same shape whether it's running, ok, or error. Costs are live: polling a running row shows the debit counter climbing in cents without any JSON reshaping. cost.model is the SUT's own inference spend, distinct from cost.driver, the Driver LLM's spend (§8 of concepts).

driver and cost.driver are the only names for the Driver's model id and the Driver's spend. Both values previously shipped under operator / cost.operator, and before that under agent / cost.agent. None of the older keys are emitted any more; a client reading them gets an absent field, not a zero. See §17 for the removal itself, which was a breaking change made deliberately.

The operator spelling predated the split that gave Operator its current meaning (the goal-level agent that launches runs, §12a). Once that split landed, the key named a different actor than the one whose identity and spend it carried, which made every reading of a cost breakdown a small guess about which agent was meant. It was removed rather than kept, because a wrong name that still works is read as correct.

Every cost entry carries an exact amount_usd beside the whole-cent amount_cents. Reconcile against amount_usd. amount_cents rounds any non-zero cost up to at least 1¢, so a run that spent $0.0007 reports 1, and no consumer reading cents can tell that apart from $0.0099 (§3.3).

cost.billed is what you are charged. cost.incurred is not. incurred is everything the run consumed including cost.model, and the SUT's spend is deliberately never charged here: on a cloud-inference run it is metered on the minted OpenRouter key's own stream, and on a BYOK run the user's provider already charged it. So the two legitimately differ, often by an order of magnitude, and the gap between them is cost.model by design rather than a billing discrepancy.

This field was called total until it had misled readers in both directions — quoted as the price of a run, and read as a platform subtotal that cost.model then got added back onto, double-counting the SUT. Neither reader was careless; total is simply the key you reach for when you want to know what something cost. incurred is named to make you ask.

cost.proxy_linger is not part of cost.incurred. It meters the LiteLLM proxy window kept alive after the run's own ledger closed (metadata.proxy_linger_seconds), so it is accrued by a different writer and is always reported separately. A client that sums the entries under cost to get an all-in figure must add it explicitly; one that reads cost.incurred alone will under-report a run that lingered.

cache_hits is an object, not a count: {"count": N, "phases": {…}}, where phases maps each reused phase to the id of the run its output came from. Use cache_hits.count where you want the number.

storage is always present, with every counter defaulting to 0 rather than the block being null. relaunch_budget_sec is null when unset, never 0, because 0 would mean "no time allowed", which is a different claim.

tokens mirrors that same split, and the two halves are separate streams — don't read one as the run total. tokens.input / tokens.output / tokens.total count the model under test: its own inference, metered off the run's proxy. tokens.driver counts the Driver (the LLM that executes the run). A driver-heavy run can bill orders of magnitude more driver tokens than model tokens, so a client showing a single "tokens" figure should say which one it means, or add them explicitly.

model is the run's frozen model pin. model.identifier is the model string (e.g. qwen/qwen3-32b); model.source is the provider id (e.g. openrouter, together); model.provenance records how the pin was chosen — "specified" when the launch body carried an explicit model, other values when the pipeline resolved it from the request text or account defaults.

sut_config is the sampling configuration the harness actually applied to the model under test (temperature, seed, max tokens, …), captured once per run. {} means provider defaults with no overrides; null means it was not captured.

benchmark is null on list rows; the flat benchmark_name is always present, so key off benchmark_name / benchmark_id. The nested subset and impl objects are always null today — use subset_id / impl_id.

benchmark_id, subset_id and impl_id are absent, not null, until the run finalizes. They come from the catalog projection, which is written when the run reaches a terminal status, so an in-flight run simply has no such keys. Per §3.10 a typed client sees None either way, but code that checks for key presence must not read the absence as "this run has no benchmark".

forked_from_deleted names the run that was removed when this one was adopted out of a deleted parent (§4.7); it is null on runs that never lost a parent.

Launch metadata is not part of the Run object: it is behavioral launch configuration only (§4.2.1), and with the single exception of publish_on_complete it is neither persisted nor echoed back.

backend records which compute backend ran the sandboxes for this run: "e2b" (E2B managed microVMs) or "sandbox0" (Kubernetes pods, either ours or your own). Set by the launcher from the compute request field (or the account default, which is sandbox0 unless you configured your own provider) and is immutable after launch.

4.1.1 status, display_status, and halt_reason

status is the pipeline's own state (running, ok, error). display_status is what a UI should show: it folds halted and halt_reason back into the status so that "stopped on purpose" does not render as a failure.

A finished run that was killed reads killed rather than error, so gate CI on status and label the UI from display_status.

halt_reason says why. These are prefixed strings, not bare enum values, and several carry a payload after the prefix, so match them with a prefix test and never with equality:

inference is LiteLLM proxy telemetry, present on the detail response only (not on list rows): totals for calls, ok, completion_tokens, and duration_ms, plus a by_model array with the same counters per model id. Used to compute tok/s, mean latency, and success rate. null on older runs.

4.2 POST /v1/runs: launch a run

Runs have three launch modes, distinguished by which top-level field the body carries. Exactly one must be present.

All modes share model, constraints, and metadata. Idempotency comes from the Idempotency-Key header (§3.7); there is no idempotency_key body field.

Pinned mode takes a catalog benchmark id or exact name alone — { "benchmark": "<id-or-name>", "model": {…} }. Do not also send request (that is a second launch mode and returns 400). Optional subset / impl further pin a split or scorer when you have them; they are not required. The launcher resolves the catalog row, writes a synthetic request ("Run <display name>"), and — for benchwright-built (product) benchmarks — seeds metadata.product_subset_id so ProductBridge runs the stored recipe. Unknown id → 404 benchmark_not_found. Wire prefix bm_ is accepted and stripped.

4.2.1 Natural-language launch

The pipeline decides everything from a description string.

POST /v1/runs
Authorization: Bearer bw_live_pat_…
Idempotency-Key: 4f1c7a1a-3b2c-4f6b-9b8a-123456789abc
Content-Type: application/json

{
  "request": "Run MMLU Abstract Algebra on openai/gpt-4o-mini via OpenRouter",
  "driver": "openrouter://deepseek/deepseek-v4-flash",

  "model":   { "model": "openai/gpt-4o-mini", "provider": "openrouter" },
  "compute": "e2b",

  "constraints": {
    "max_cost_cents": 2000,
    "max_duration_ms": 900000,
    "max_tasks": 100
  },

  "trials": { "plan": { "k": 3 }, "aggregator": { "name": "mean" } },

  "metadata": {
    "autonomous": true,
    "limit": 25
  }
}

• model replaces the natural-language model parsing the Find phase does.

• driver pins the Driver, the LLM that drives the pipeline (Find, Analyze, Atomize and friends), as distinct from model, which is the model being evaluated. driver is the only key read for this. operator and agent were the previous names and are no longer accepted: a body sending either pins no Driver at all and the run uses the default below, so a script still on an old key launches on a model it did not choose. operator was the more misleading of the two, since it set the Driver while naming the Operator, which is a different actor (§12a). Omit driver entirely and the run uses your configured Operator's own model, falling back to the platform default. An explicit pin must name a priced Gemini model or an openrouter:// spec; anything else is refused with 403 admin_required. A driver arrived at through your own operator settings is always allowed, since that one runs on your own credit.

• phase_models (object) overrides the driver for individual phases, {"analyze": "openrouter://…"}. Each spec must satisfy the same rule as driver, with the same 403 admin_required when it does not.

• compute overrides which backend runs this run's sandboxes — "e2b" or "sandbox0" — without changing your saved default (Settings → Compute). Omit it (or send "default" / "auto") to use your account default. An explicit "sandbox0" uses your configured sandbox0 endpoint even if it's currently disabled as your default, so a one-off run can target it. Unknown value → 400 invalid_argument. The chosen backend is stamped on the run as backend.

• model.provider pins the inference provider. Only "openrouter" and "together" are currently validated as pinnable providers; any other value returns 400 unknown_provider before any sandbox spawns.

• hints.benchmark is read only in combination with metadata.canonical_indices (below), to resolve canonical task identity against that benchmark; it does not skip the catalog search on its own. hints.subset is ignored.

• trials runs each task multiple times: plan.k (1–20) is the trial count and aggregator names how per-trial scores fold into the row score — any_pass (default), all_pass, mean, or the parameterized majority_on / weighted_mean / min_above. An unrecognized aggregator name fails 400 unknown_aggregator.

• limits declares every ceiling, spelled <level>.<quantity>:

  { "limits": { "run": { "cost": 5.0, "duration": 3600, "tokens": 2000000 },
                "task": { "attempts": 1 },
                "attempt": { "duration": 600 } } }
  

  Money is in whole dollars and time is in seconds. The unit belongs to the quantity and never to the name, so there is no _cents or _ms suffix to get backwards.

  Level	Quantity	Bounds
  run	cost	total spend: sandbox + driver + model
  run	model_cost	SUT spend only — your own provider's bill
  run	duration	wall clock for the whole run
  run	tokens	prompt + completion tokens
  run	tasks	how many dataset rows run at all
  task	cost	model spend on one task, across all its attempts
  task	tokens	tokens on one task, across all its attempts
  task	duration	wall clock for one task
  task	attempts	tries before a task is failed
  attempt	duration	wall clock for ONE try of one task
  phase	calls	LLM calls one phase may spend
  phase	duration	wall clock for one phase

  run and phase ceilings stop the run. task ceilings stop only that task — it is recorded as failed and the remaining tasks carry on, because a task over its own budget is a failed task, not a failed run. A task stopped this way is never retried: retrying would spend the ceiling again, up to task.attempts times.

  Task ceilings bound the whole retry chain, which is what makes them the useful shape — a per-attempt money cap still allows N attempts x the number you wrote down.

  phase is the only level that may name its target: "phase:atomize": {"calls": 250} overrides "phase": {"calls": 100} for that phase alone. Use it to give a build-heavy phase more room, not just less.

  Reaching a run-level ceiling halts the run with halt_reason="limit_exceeded:<level>.<quantity>". run.cost compares cost.incurred (cost_total), which includes model spend, so it fires on a BYOK run where cost_billed would barely move.

  A limit the platform does not enforce is refused with 400, never stored and ignored: a ceiling that silently does nothing is worse than no ceiling. attempt.cost is the current example — the proxy tags spend per task, so retries share one tally and an attempt has no cost of its own to measure.

  The pre-limits spellings still work and mean the same thing: constraints.max_cost_cents (cents), constraints.max_duration_ms (ms), constraints.max_tasks, metadata.limit, metadata.task_attempts, and metadata.task_timeout_sec — which maps to attempt.duration, because that is the level it always bounded.

  Halts are acted on between phases, so a single long-running phase — notably a harness bridge run (inspect/lmeval/lighteval/ harbor), where the whole eval is one phase — can overrun a ceiling until the phase ends. For a hard pre-launch bound on harness runs, pair a ceiling with max_tasks (sample count).

• metadata.autonomous: true runs the pipeline without the requestHumanReview tool registered on any phase, so the run can never halt at awaiting_review — it succeeds or fails on its own (requestPhaseRestart and normal error paths still apply). For unattended runs — CI, catalog/smoke sweeps — where no operator is watching to resolve a review. Requires an explicit model pin (400 autonomous_model_required otherwise).

• metadata.limit: <N> caps Materialize to the first N samples for any run — ad-hoc (Atomize→Materialize) or inspect-harness. The Driver is told the dataset total is N (so it submits indices 0..N-1) and the materializeStart tool hard-clamps any submitted index >= N, so a "run everything" request can't exceed it. Used for cheap smoke runs and bounded ad-hoc runs. (The legacy metadata.inspect_harness.limit is still honored as a fallback for harness runs.)

• metadata.indices: "<spec>" restricts Materialize to specific dataset indices, as an index-spec string ("0-99", "0,2,4").

• metadata.canonical_indices: "<spec>" pairs with hints.benchmark: the given canonical benchmark positions are resolved to task identities so the run scores against the catalog's canonical tasks.

• metadata.proxy_linger_seconds: <N> keeps the run's LiteLLM proxy sandbox alive for N seconds after the run finishes, instead of tearing it down at completion. Useful for inspecting the proxy post-run. Clamped to [0, 86400] (24h); 0 (default) tears it down immediately. CLI: --proxy-linger-seconds N.

• metadata.local_egress: true routes the run's outbound traffic through a machine you control, and metadata.egress_endpoint names a specific one. See §9a.

• metadata.harbor_task: "<name>" pins a single Harbor task by name on a harbor-harness run.

• materialize.orchestrator.storage: "8Gi" (a top-level field, not under metadata) asks for a work sandbox with more ephemeral disk for the orchestrator, the sandbox that downloads the dataset and drives the harness. Reach for it when a run dies on disk pressure with a large dataset.

• metadata is otherwise behavioral launch configuration only — limit, autonomous, indices, canonical_indices, and the per-harness config objects. It is not persisted, not returned on the run resource or its events, and not filterable. To correlate runs to CI jobs, record the returned run id on the CI side.

• The one exception is metadata.publish_on_complete: true, which is persisted on the run. It is an owner opt-in: set it at launch and the run flips to visibility=public the moment it finalizes status=ok, saving a publish call (§8.4) on a run you already intend to share. Only a successful run is flipped: one that errors or is halted stays private, so this cannot publish a failure you would not have published by hand.

Choosing the eval harness

harness selects how the benchmark is evaluated. It's an enum — an unknown value is rejected with 400 unknown_harness before any sandbox spawns.

scratch, native, autonomous, adhoc, agent, none are accepted as synonyms for benchwright; lm-eval / lm_eval for lmeval.

A third-party harness (inspect / lmeval / lighteval / harbor) takes its per-run config — the task id and its args — under metadata.<harness>_harness:

{
  "request": "Run MMLU abstract_algebra on qwen3-32b via Inspect",
  "harness": "inspect",
  "model": { "model": "qwen/qwen3-32b", "provider": "openrouter" },
  "metadata": {
    "inspect_harness": {
      "task": "inspect_evals/mmlu_0_shot",
      "args": { "subjects": ["abstract_algebra"] }
    }
  }
}

An explicit third-party harness with no task is valid: the Find phase resolves the task id from the request text. harness itself is optional too — omit it and the harness is inferred from which metadata.<h>_harness object is present, defaulting to benchwright.

Renamed from bridge. This field and its metadata.<h>_harness config keys were previously bridge / metadata.<h>_bridge. The old names are rejected with 400 bridge_renamed_to_harness (rather than silently ignored) so a stale recipe fails loudly instead of running on the wrong harness.

Reasoning models + harness exact-match scorers. A harness task that caps generation for a non-CoT multiple-choice scorer (e.g. inspect_evals/mmlu_0_shot, which applies max_non_cot_tokens) truncates a reasoning SUT (qwen3, etc.) inside its thinking block, so the final answer never reaches the scorer and the run reads near the random floor. For reasoning SUTs prefer a CoT variant (cot=True), a larger token budget, or the benchwright harness (which captures the model's answer content directly).

4.2.2 Pinned benchmark launch

Rerun a catalog entry without inventing a natural-language description. benchmark is the mode discriminator; request must be omitted.

POST /v1/runs
{
  "benchmark": "example-landing-screenshot",
  "model": { "model": "qwen/qwen3-32b", "provider": "openrouter" },
  "metadata": { "autonomous": true, "limit": 1 }
}

4.2.3 Replay launch

Re-materialize a prior run's subset + impl against a different model (or the same one, as a noise check). The pipeline rewinds to Materialize and inherits Find / Fetch / Analyze / Setup / Atomize from the parent.

Attach is deliberately not inherited when Materialize needs the model proxy: a replay exists to point the same eval at a different model, so re-running Attach is what makes the new pin take effect instead of silently reusing the parent's proxy config.

POST /v1/runs
{
  "replay": { "from_run": "<uuid>" },
  "model": { "model": "qwen/qwen3-235b-a22b-thinking", "provider": "openrouter" }
}

Accepted keys under replay:

The replay does not necessarily run from the id you passed. If that run is itself a fork, the server walks forked_from up to the first ancestor that is not one (up to 16 levels) and replays from that root, since the root is where the inherited phase output actually lives. So the new run's forked_from may name a different run than the one you sent. Read it back rather than assuming.

Omitting model reuses the parent's model (noise check). Providing a new model is the cross-model A/B. The resulting run shares the parent's impl_id, so the registry can render a same-impl comparison without needing any extra correlation.

Replay over a rolling subset pins to the parent's sample_count snapshot, not the subset's current size, so repeated replays produce stable comparisons even as the subset accretes new tasks.

4.2.4 Response

Every launch mode returns the same 202 Accepted shape:

{
  "id": "<uuid>",
  "status": "launching",
  "mode": "replay",
  "sandbox_id": "<driver_sandbox_id>",
  "links": {
    "self":      "/v1/runs/<uuid>",
    "events":    "/v1/runs/<uuid>/events",
    "dashboard": "https://benchwright.ai/run/<uuid>"
  }
}

sandbox_id and links.dashboard are present when known at launch time. There is no cost estimate on the launch response.

4.2.5 Preflight errors

Returned synchronously, no run created:

• 400 invalid_argument: multiple launch modes present, or none. Combining request with benchmark is rejected — pinned mode is benchmark alone. An empty description after mode checks names the three accepted shapes in the detail.
• 404 benchmark_not_found: pinned benchmark id/name is not in the catalog.
• 404 subset_not_found: pinned subset does not exist at all.
• 400 invalid_argument: pinned subset exists but belongs to a different benchmark than the one you pinned. The two cases are kept apart because they mean different things: a typo versus a mismatched pair.
• 403 admin_required: a non-admin caller explicitly pinned a non-Gemini driver, or sent a phase_models override naming one.
• 409 replay_unsupported: replay.from_run resolves to a parent that ran the benchwright harness. That harness authors its impl per run, so there is nothing fixed to re-materialize against.
• 402 insufficient_balance: balance < $1 launch floor.
• 404 parent_not_found: replay.from_run does not exist or is not accessible to the caller.
• 400 parent_still_running: replay/fork of a run that has not finished.
• 403 forbidden: fork of a run the caller does not own.
• 409 idempotency_conflict.
• 400 no_provider_secret: no provider API key exists in the user's vault at all.
• 412 no_provider_secret: a specific pinned provider's key is missing (the user has vault secrets, but none match the pinned model's required provider). Both variants carry code + detail only — add the key under /manage/vault.
• 400 autonomous_model_required: metadata.autonomous without an explicit model pin.
• 400 unknown_aggregator: unrecognized trials.aggregator name.
• 429 tier_concurrency_exceeded: the account is already at its tier's concurrent-run cap.
• 501 e2b_not_configured / 400 compute_not_configured: the requested (or default) compute backend has no credentials configured on this deploy.

4.3 POST /v1/runs/{id}/fork

Same shape as launch, but seeds from the parent run:

POST /v1/runs/<uuid>/fork
Idempotency-Key: …

{
  "description": "Retry Atomize with a stricter scoring function",
  "from_phase": "atomize",
  "model": { "model": "qwen/qwen3-32b", "provider": "openrouter" },
  "indices": "0-49"
}

description is required (400 without it), and it is not just a label: with no from_phase, the server's ForkPhase reads it to choose the rewind point. Say what you want changed, not what the fork is called.

from_phase is optional. model (object) overrides the parent's model pin; indices (index-spec string) restricts which dataset indices the fork materializes.

Only your own runs can be forked (admins excepted); forking someone else's run fails 403 forbidden. Forking a run of the benchwright harness is refused 409 fork_unsupported. The parent preflight is inherited from launch: 404 parent_not_found for an unknown or inaccessible parent, and 400 parent_still_running for one that has not finished.

4.4 GET /v1/runs/{id}

Returns the Run object. ?expand=phase_breakdown adds:

{
  "phase_breakdown": [
    {
      "phase": "find",
      "invocation": 1,
      "duration_ms": 8200,
      "tokens": { "input": 1200, "output": 340 },
      "cost": { "llm_cents": 3 },
      "llm_calls": 1,
      "model": "gemini-3-flash-preview"
    },
    …
  ]
}

?expand=child_forks adds the list of traces forked from this one (not recursive).

4.5 GET /v1/runs: list

GET /v1/runs
  ?status=in:running,error
  &benchmark=MMLU
  &visibility=private
  &started_at=gte:2026-04-01T00:00:00Z
  &sort=-started_at
  &limit=50

Returns { data: [Run...], pagination }. The filterable fields are exactly status, benchmark (matches benchmark_name), visibility, started_at, and finished_at; unknown filter keys are silently ignored (§3.5). Sortable keys: started_at, finished_at, created_at, cost_total, cost_billed. There are no model.* or metadata.* filters.

Team members see teammates' runs in this list alongside their own (§2.5). Admins can pass ?scope=all to list every user's runs.

4.6 POST /v1/runs/{id}/halt

Stops the run. Halt is an immediate synchronous teardown (the same kill core as DELETE, §4.7), not a deferred graceful stop.

POST /v1/runs/<uuid>/halt
{ "reason": "superseded by build 8813" }

The body is optional, but a reason is recorded, not ignored: the run's halt_reason becomes operator_kill by <user id>: <reason>, with whitespace collapsed and the text truncated to 200 characters. Send one: six months later it is the only thing that distinguishes a deliberate stop from a failure.

Returns the updated Run object with halted=true. For a non-destructive stop that preserves the work sandbox and can be resumed, use POST /v1/runs/{id}/pause (§4.11) instead.

4.7 DELETE /v1/runs/{id}: stop it, then destroy it

DELETE means delete. It first issues the eager teardown path — every live sandbox tagged with this run is killed, then status=error + halt_reason=operator_kill by <user id> — and then removes the run record itself. The event stream, the per-task payloads and the registry summary go with it. This cannot be undone. To stop a run and keep it readable, use POST /v1/runs/{id}/halt (§4.6). To keep a run but hide it, PATCH visibility to private (§4.8).

Teardown finds those sandboxes by asking the compute backend which live sandboxes carry this run's id, rather than by replaying the event stream for SandboxCreated without a matching SandboxKilled. The event walk misses sandboxes that came from a snapshot-and-restore cycle, which do not always emit a kill for the handle they replaced, so it would leave real machines running while reporting a clean teardown.

Returns 200 OK:

{
  "id": "<uuid>",
  "deleted": true,
  "halt_reason": "operator_kill by <user_uuid>",
  "sandboxes_killed": 2,
  "snapshots_freed": 1,
  "forks_adopted": ["<uuid>"],
  "events_adopted": 6,
  "reparented_to": "<uuid>"
}

It deletes one run, never a chain. A fork does not copy its parent's work, it inherits it by reference, so removing a run that others forked from would otherwise strand them. Instead the fork children are adopted: everything they were inheriting across the removed link — the latest phase output per phase, plus the fetch, atomize and model artifacts — is copied onto each child before the parent goes, and each child is re-pointed at the deleted run's own parent so the rest of its ancestry still resolves. A child's own event always wins over an inherited one, exactly as it did before the delete, so nothing a fork produced itself is overwritten.

forks_adopted lists the runs kept this way (empty when there were none), events_adopted counts the events copied down across all of them, and reparented_to is the run they now point at (null if the deleted run was itself a root). On each adopted child, forked_from_deleted records the run that was removed, and every copied event carries data._adopted_from naming its origin — the provenance moves, it is not erased.

deleted: false never accompanies a 2xx. If the record cannot be removed the response is 409 with code: "delete_blocked" and retryable: false; the run is halted but still present, and repeating the request will not change that. Treat any non-2xx as "the run is still there".

That 409 carries structured fields beside the standard problem+json ones (id, deleted, halt_reason and sandboxes_killed), so a client can report what the attempt did accomplish (the run was stopped, its machines were released) without a second request.

4.8 PATCH /v1/runs/{id}

Exactly two fields are patchable:

{
  "visibility": "public",
  "review_resolution": "Scoring was too strict, reran with expected=normalize(str)."
}

metadata is not patchable. visibility takes private, unlisted or public. unlisted is a real state and not a synonym for either neighbour: an unlisted run is readable by anyone who has its id, including anonymously, but it is not projected into the public catalog. visibility=public requires only status == "ok"; otherwise the server returns 409 visibility_constraint_violated.

Making a run non-private also recomputes and restamps its fingerprint_hash from the run's stored artifact events, so a published run always carries a hash a reader can re-derive for themselves from GET /v1/runs/{id}/fingerprint.

This route honours If-Match (§3.8) against the run's updated_at, which is worth sending here: publishing is the one run write where two clients racing can disagree about what state you meant to leave it in.

Setting review_resolution has a side effect: it clears halted/halt_reason and wakes the driver sandbox, so a run parked at awaiting_review resumes with the resolution text as its instructions.

4.9 GET /v1/runs/{id}/narrate

Live one-sentence narration of the run. Only generated while a dashboard has this endpoint open — if nobody's polling, no narration work runs and no narration cost accrues. Responses are cached per (run, last_seq) so concurrent viewers share one generation.

GET /v1/runs/<uuid>/narrate?slice=recent

• slice=recent: present tense, what the Driver is doing right now.
• slice=full: past tense, what the run did end to end.

Response:

{
  "sentence": "Rerunning Atomize after the score function threw on task 12.",
  "generated_at": "2026-04-23T14:07:05.889Z",
  "slice": "recent",
  "from_cache": false,
  "first_seq": 102,
  "last_seq": 158
}

first_seq / last_seq bound the event window the sentence was generated from.

• 400 invalid_argument for a slice that is neither recent nor full.
• 204 No Content when the caller has not enabled run_narration_enabled. Narration is opt-in, so an account that never touched the setting gets 204 exactly like one that turned it off. Clients should stop polling — no work was done, nothing will change until the flag is set. (An anonymous request against a public run has no settings to consult and skips the check entirely.)
• 503 narration_unavailable when the narration model is offline; callers fall back to phase-derived labels.

4.10 Aggregate stats

One non-admin read endpoint summarizes the caller's runs without paginating through them:

GET /v1/runs:phase-stats    # per-phase duration + attempt aggregates

The response is keyed by phase:

{ "find": { "avg_ms": 8200, "attempts": 41, "success": 40, "failed": 1 }, … }

It pairs PhaseStarted with PhaseCompleted server-side so a client can render "where does my time go" charts without pulling event rows. There are no cost or token aggregates on this endpoint.

The scope is narrower than "your runs", and the counters are worth reading literally:

• Only your most recent 100 finalized runs (status of ok or error) are considered. This is a recent-history sample, not a lifetime total, and it does not move while runs are in flight.
• attempts counts phases that started; success counts those that completed.
• failed counts a phase that started and never completed on a run that errored. The same shape on a run that finished ok counts toward neither success nor failed, so success + failed can be less than attempts.
• With no finalized runs yet the response is a bare {} rather than a zero-filled skeleton. Render "no data" from an empty object.

4.11 POST /v1/runs/{id}/pause and /resume

Operator pause: a non-destructive stop that preserves the work sandbox. Requires runs:write and ownership (admins can pause any run).

POST /v1/runs/<uuid>/pause     # halted=true, halt_reason=operator_pause: <user>
POST /v1/runs/<uuid>/resume    # clears the pause and relaunches the Driver

Pause sets halted=true with an operator_pause reason; the Driver suspends at the next phase boundary, keeping the work sandbox intact. Returns the updated Run object. Errors: 409 invalid_state when the run is not running; 409 already_halted when it is already halted (including awaiting_review — do not clobber a Driver question with a pause).

Resume only lifts an operator_pause halt — a run waiting at awaiting_review is unblocked via PATCH review_resolution (§4.8), not /resume; that case returns 409 not_paused. If the halt was cleared but the Driver could not be relaunched on this host, resume returns 501 resume_not_supported — fork the run to continue from that point.

5. Resource: Events and streaming

Pipeline events are the primary reason a run resource is event-sourced. Every phase start, every LLM call, every sandbox creation, every materialize-task result is a typed event. Two consumption modes: polling and SSE.

5.1 GET /v1/runs/{id}/events

Polling / backfill. Returns an array in seq order.

GET /v1/runs/<uuid>/events?after_seq=42&limit=500&types=MaterializeTaskResult

Additional query parameters: before_seq (upper bound, pairs with after_seq for a window), order=desc (newest first; default is seq-ascending), exclude_types (comma list, the complement of types), and limit (default 500, capped at 1000).

types and exclude_types do not compose. They write the same underlying filter, so sending both is not an intersection: exclude_types wins and types is discarded. Pick one per request.

Response:

{
  "data": [
    {
      "id": "<event_id>",
      "seq": 43,
      "type": "MaterializeTaskResult",
      "created_at": "2026-04-23T14:05:01.003Z",
      "data": {
        "task_id": "abstract_algebra_0",
        "index": 0,
        "score": 1.0,
        "duration_ms": 822,
        "output_preview": "The answer is (D). …"
      }
    }
  ],
  "pagination": { "has_more": true, "next_page_token": "<opaque>" }
}

after_seq is preferred over page_token for event streams because seq is monotonic and stable across reconnects.

has_more is not a liveness signal. It means only "this page came back full", so a running run that happens to have no further events answers has_more: false, and a finished run whose last page filled exactly answers has_more: true. Decide whether to keep polling from the run's status (§4.1) or from the SSE run.finished event, never from has_more.

5.2 GET /v1/runs/{id}/events:stream (SSE)

Server-sent events for live dashboards and CLIs.

GET /v1/runs/<uuid>/events:stream
Accept: text/event-stream
Last-Event-ID: 42

Each event is emitted as:

id: 43
event: MaterializeTaskResult
data: {"type":"MaterializeTaskResult","task_id":"abstract_algebra_0","index":0,"score":1.0,…}

The event type appears twice on purpose: in the SSE event: line and folded into the data object as "type", so a consumer that only ever parses data still knows what it is holding.

Keep-alives are event: ping every 20 seconds. The stream closes cleanly with a final event: run.finished when status becomes terminal. Opening a stream on a run that has already finished is well defined and cheap: you get the backfill, then run.finished immediately.

Browsers and SDKs use the native Last-Event-ID semantics for resume; clients that cannot set headers (e.g. a bare EventSource) can pass ?last_event_id=42 as a query parameter instead.

5.3 Event identity and ordering

• (run_id, seq) is globally unique and monotonic per run.
• Event IDs are globally unique but not monotonic across runs.
• Clients dedupe by event id; both modes can deliver the same event twice (SSE reconnect, poll overlap).

The two modes use different identifiers, and mixing them fails quietly. GET …/events gives each row an opaque id that encodes (run, seq). SSE emits the bare seq as its id:, and reads Last-Event-ID as a number. Feed a polling id into Last-Event-ID and it does not parse, so the stream resumes from the beginning and replays the whole run instead of erroring.

The rule that works for both: resume from seq, dedupe by id. Track the highest seq you have processed and hand that to Last-Event-ID or after_seq; use the id only for deduplication.

6. Resource: Artifacts

Everything the pipeline emits that is larger than a routine event becomes an artifact: the Driver's task_impl.py, each pending save during Atomize, report files, fingerprint breakdowns. One endpoint covers all of them, filtered by type.

6.1 GET /v1/runs/{id}/artifacts

GET /v1/runs/<uuid>/artifacts
  ?type=in:atomize,report,fetch
  &include_versions=true

Response:

{
  "data": [
    {
      "id": "<artifact_id>",
      "type": "atomize",
      "name": "task_impl.py",
      "mime": "text/x-python",
      "size_bytes": 2841,
      "version": 3,
      "latest_version": 3,
      "seq": 117,
      "invocation": 2,
      "created_at": "2026-04-23T14:03:42.000Z",
      "download_url": "/v1/runs/<uuid>/artifacts/<artifact_id>/content"
    },
    {
      "id": "<artifact_id>",
      "type": "report",
      "name": "report.html",
      "mime": "text/html",
      "size_bytes": 48129,
      "seq": 402,
      "invocation": 1,
      "download_url": "/v1/runs/<uuid>/artifacts/<artifact_id>/content",
      "created_at": "2026-04-23T14:09:14.000Z"
    }
  ],
  "pagination": { "has_more": false }
}

Every row carries seq and invocation. size_bytes is approximate: it counts characters in the stored JSON string rather than bytes on the wire, so anything non-ASCII reads low. Size a progress bar with it, not a buffer.

type values:

type=atomize* is accepted as a wildcard covering atomize + atomize_draft. There is no materialize artifact type — per-task results are served by /tasks/{id}/payload (§7.2).

6.2 GET /v1/runs/{id}/artifacts/{art_id}

Artifact metadata, as one row of the §6.1 list, with the exception that version and latest_version are not included here even for a versioned type. If you need the version of a specific artifact, read it from the list response rather than from this one.

6.3 GET /v1/runs/{id}/artifacts/{art_id}/content

Raw bytes with the artifact's mime as Content-Type. Range headers are not supported — content is always served whole. Active-content mime types (text/html, SVG, XML, JavaScript) are downgraded to text/plain with X-Content-Type-Options: nosniff so an artifact can never script against the API origin.

6.4 GET /v1/runs/{id}/fingerprint

A run's identity hash and the step-by-step derivation:

{
  "hash": "c8a1b2d3e4f5",
  "components": [
    { "type": "ModelProxyArtifact", "seq": 42, "data": { /* … */ },
      "canonical": "{\"model\":\"…\"}", "cumulative_hash": "a1…" },
    { "type": "AtomizeArtifact",  "seq": 117, "data": { /* … */ },
      "canonical": "{\"content\":\"…\"}", "cumulative_hash": "c8…" }
  ]
}

Each component carries data (the filtered event payload the canonical string was derived from) alongside the hash chain, so an audit can see what went into the digest and not only that it changed.

A run that emitted no artifact events has no fingerprint to break down and answers 404 not_found.

Useful for reproducibility audits ("did two runs use the same code and model config?") without downloading the artifact bodies.

6.5 GET /v1/runs/{id}/report

Responds 302 with a redirect to the content URL of the run's first ReportFile (lowest seq), so clients don't have to list artifacts first. A run that wrote several report files is not choosing a canonical one here; list ?type=report (§6.1) if you want the others. The content serves under the §6.3 rules (HTML downgrades to text/plain). The route is gated by run visibility only — no artifact scope is required.

7. Resource: Tasks and results

A task is one row of the benchmark dataset plus the per-run result for that row.

7.1 GET /v1/runs/{id}/tasks

GET /v1/runs/<uuid>/tasks
  ?version=2
  &passed=false
  &error_class=timeout
  &include_retries=true
  &slim=1

Query parameters: passed (true/false), version (one materialize invocation; default latest, and an out-of-range number is clamped to the latest rather than rejected), error_class (exact match on the derived bucket below), include_retries=true (keep superseded attempts instead of only the final one per index), slim=1 (a lighter row, see below). sort and limit are not implemented — the full task set is always returned in one response and pagination.has_more is always false.

include_retries is matched against the literal string true; include_retries=1 is not an error and not an enablement, it simply does nothing.

slim=1 drops six fields, not two: id, run_id, output_preview, passed, meta and links. Losing id and links.payload means you cannot follow a slim row to its payload (§7.2), so use it for aggregate views and not as the first half of a drill-down.

Response:

{
  "data": [
    {
      "id": "<task_id>",
      "run_id": "<uuid>",
      "invocation": 2,
      "index": 0,
      "trial": 0,
      "task_id": "abstract_algebra_0",
      "output_preview": "The answer is (D). …",
      "score": 1.0,
      "passed": true,
      "duration_ms": 822,
      "error": null,
      "error_class": "ok",
      "timestamp": 1777039501003,
      "meta": { },
      "links": {
        "payload": "/v1/runs/<uuid>/tasks/<task_id>/payload"
      }
    }
  ],
  "pagination": { "has_more": false },
  "summary": {
    "total": 100,
    "passed": 82,
    "failed": 18,
    "pass_rate": 0.82,
    "invocation": 2,
    "invocation_count": 2,
    "error_class_counts": { "timeout": 3 },
    "duration_ms_p50": 744,
    "duration_ms_p95": 1801
  }
}

Rows do not carry input / expected / full output — those live behind links.payload (§7.2). timestamp is epoch milliseconds, not an ISO string, and defaults to 0 when the row carries none. passed is score >= 1.0.

error_class is derived, never null, and always one of a fixed set of lowercase buckets: a row with no error reads "ok", so filter on "ok" rather than on a null. The buckets:

ok · truncation · disk_full · rate_limited · timeout · code_error
sandbox_died · missing_result_file · task_no_result · spawn_failed
cancelled · cancelled_pre_launch · inference_rejected · other

summary is computed from a different row set than data, so summary.total normally does not equal data.length. summary reports the run's result: it prefers the aggregated per-task rows when the run produced them (trials folded into one score per task), and otherwise uses the per-trial rows with errored attempts filtered out. data reports the rows themselves, under whatever filters you passed. Read summary for "how did this run do", and do not recompute it from data expecting the same number.

duration_ms_p50 / p95 are omitted when no rows carry durations, and runs launched with trials add trials, attempts_per_task, and aggregator.

7.2 GET /v1/runs/{id}/tasks/{tsk_id}/payload

The full record for one task: input, expected, output, plus raw_output, atif, meta, score, passed, duration_ms, error, error_class, size_bytes, and created_at. It also carries the addressing fields id, run_id, invocation, index, trial and task_id.

Several of those are computed at read time rather than stored: passed, error_class (the §7.1 bucket) and the atif timing breakdown are all derived per request, so do not expect this response to match a raw row of the underlying table field for field.

Two query parameters worth knowing:

• ?trial=N pins one trial of a multi-trial task. Omit it and you get the best-scoring trial, not the first or the last. Convenient for "show me what this task can do", misleading if you assumed it was trial 0.
• ?raw=0 drops raw_output. That field holds the model's untrimmed output and dominates the response on agentic harnesses (a single harbor task measured 5.3 MB with it and 67 kB without), so pass it whenever you only need the scored fields.

8. Resource: Benchmarks, subsets, impls

The public registry of benchmarks, their subsets (splits), their impls (distinct task_impl.py fingerprints), and their aggregated leaderboard rows.

Access. Registry reads are part of the public catalog: an unauthenticated request returns every benchmark/subset/impl that has at least one public run. A browser (JWT) session additionally folds in the caller's own private runs, each row carrying a visibility field. A PAT must declare registry:read — a token that authenticates without it gets 403 scope_insufficient. The authoring endpoints (§8.4–§8.8) require registry:publish.

Two id spaces coexist. Projected catalog rows (runs the pipeline finalized) use composite ids — a benchmark slug (MMLU) and a <benchmark>:<hash> subset id (MMLU:7281bbc31736). Rows created through the authoring endpoints carry the bm_/sub_/imp_ prefixes. Both shapes are accepted on the read routes.

Only the benchmark id is opaque there (bm_<uuid>). Subsets and impls are content-addressed: sub_<benchmark id>:<12 hex> fingerprints the task set, imp_<benchmark id>:<12 hex> fingerprints the code. So the same tasks or the same scorer under the same benchmark always resolve to the same id, and reading an id tells you which benchmark it belongs to without a lookup.

8.1 GET /v1/registry/benchmarks

The catalog list. By default it returns the full nested tree (benchmarks → subsets → impls/runs), wrapped in a data envelope:

{ "data": [ /* benchmark tree elements */ ] }

For catalog rendering pass ?summary=1 to get the lightweight shape — per-benchmark counts plus per-subset top-score, no nested run arrays — which scales in benchmark count rather than run count. The summary mode returns its own envelope:

{ "benchmarks": [ … ], "total": 42, "offset": 0, "limit": 50,
  "facets": { … }, "agg": { … } }

facets and agg are present only when you pass ?counts=1; without it the keys are absent, so ask for them rather than expecting them. Summary mode also takes list/facet parameters — limit / offset paging (clamped to 200 here), ?q= text search, and category/facet filters — so the catalog page can search and paginate server-side.

GET /v1/registry/benchmarks?summary=1
  &hide_private=1     # drop the caller's own private rows
  &hide_public=1      # drop public rows (show only my private)

Each element of benchmarks (abbreviated; the shape carries more counters than are shown):

{
  "id": "MMLU",
  "name": "MMLU",
  "category": "knowledge",
  "visibility": "public",
  "tags": [],
  "highlights": ["Tests multi-subject knowledge…"],
  "subsetCount": 5, "implCount": 5, "runCount": 13, "modelCount": 1,
  "hasOfficialImpl": false,
  "lastRunAt": "2026-06-17T18:07:43.819Z",
  "subsets": [
    {
      "id": "MMLU:7281bbc31736",
      "slug": "subset-7281bbc31736",
      "name": "mmlu_first_1500",
      "sampleCount": 1500, "runCount": 1, "implCount": 1,
      "topScore": 0.859, "topModel": "qwen/qwen3-32b"
    }
  ]
}

8.2 GET /v1/registry/benchmarks/{id}

A single benchmark with its subsets, impls, native + derived leaderboards, covering-run counts, per-subset index spans, and any per-user range names. This is the page-load payload for /registry/{id}; per-task content and the difficulty distribution are fetched separately (§8.3) so the overview stays fast.

8.3 First-class read endpoints

Stable URLs for the pieces a dashboard links to directly, so it doesn't have to fetch the whole benchmark tree. All inherit the §8 access rules (anon-readable; registry:read for PATs).

GET /v1/registry/stats
      ?hide_private=1 &hide_public=1
        → catalog counters: { benchmarks, subsets, impls, runs, models, tasks }

GET /v1/registry/benchmarks/{id}/coverage
        → per-subset { covering_runs, derived_runs } for the family

GET /v1/registry/benchmarks/{id}/range?indices=0-5
        → ad-hoc virtual range: every run that ran the tasks at those
          canonical indices (0-5, 0,2,4 — ≤2000 positions), scored on
          just them, + a best-per-model leaderboard. No stored subset
          needed; coverage is by task content, so only runs that
          actually ran those tasks appear.

GET /v1/registry/benchmarks/{id}/range/dataset?indices=0-5
        → the canonical task tuples for any set of canonical positions
          (full benchmark, a slice, or a named range — ≤50000
          positions) as JSONL, served as an attachment

GET /v1/registry/benchmarks/{id}/range/dataset/info?indices=0-5
        → { rows, bytes } for the JSONL above (cheap badge metadata)

GET /v1/registry/subsets/{id}
        → one subset: its impls, runs, leaderboard

GET /v1/registry/subsets/{id}/leaderboard
        → best score per model across impls (public runs only)

GET /v1/registry/subsets/{id}/derived
        → models that ran a SUPERSET of this subset, re-scored on this
          subset's tasks (no re-run); excludes models with a native run

GET /v1/registry/subsets/{id}/covering-runs
        → every run on a superset of this subset, one row per run,
          scored on this subset's tasks

GET /v1/registry/subsets/{id}/distribution
        → difficulty histogram + score/run summary + slim per-task rows
          (index · difficulty · pass-rate, NO content)

GET /v1/registry/subsets/{id}/tasks/{index}
        → one dataset index's full content + per-model results

GET /v1/registry/subsets/{id}/dataset
        → the canonical task tuples as JSONL (one {index, input,
          expected} per line) — feed straight into an eval harness

GET /v1/registry/subsets/{id}/dataset/info
        → { rows, bytes } for the JSONL above (cheap badge metadata)

GET /v1/registry/impls/{id}
        → one impl: label, fingerprint, official flag, its runs

distribution → tasks/{index} is a three-tier load: the overview (§8.2) ships precomputed aggregates, distribution adds per-task difficulty rows with no content, and tasks/{index} resolves a single task's full input/expected/output on drill-in — so opening a 1500-task subset never serializes 1500 prompts up front.

Per-user range naming (a viewer labels an index range as a named subset) is a registry:publish write:

POST   /v1/registry/benchmarks/{id}/names   { subset_id, name, visibility? }
DELETE /v1/registry/benchmarks/{id}/names/{slug}

8.4 POST /v1/runs/{id}/publish

Shortcut for the "publish this run" flow. The body carries one field:

POST /v1/runs/<uuid>/publish
{ "visibility": "public" }

The run's fingerprint is recomputed if needed, run visibility flips, and the registry projection updates. Benchmark visibility is derived from run visibility — a benchmark is public when it has at least one public run, so there is no separate "publish the benchmark" flag.

visibility is private, unlisted or public, and defaults to public when you omit it: this endpoint exists to publish, so the empty body does the obvious thing. unlisted makes the run readable by anyone holding its URL without listing it in the catalog.

Response:

{ "id": "<uuid>", "visibility": "public", "fingerprint_hash": "c8a1b2d3e4f5" }

fingerprint_hash is present when the run has one.

Publishing is refused 409 visibility_constraint_violated when the run did not finish ok, or when it has no benchmark to publish under. Both are about the registry rather than about permission: a leaderboard row needs a score and something to hang it on.

8.5 POST /v1/benchmarks: create a benchmark

The catalog is a first-class resource, not just a side effect of running things. Creating a benchmark requires registry:publish on PATs; it starts private and only your runs reference it until you flip visibility.

POST /v1/benchmarks
{
  "name": "BFCL v4 (my mirror)",
  "description": "Mirror of Berkeley Function Calling Leaderboard v4.",
  "category": "tool-use",
  "tags": ["function-calling", "parallel"]
}

Response: the Benchmark object with id: "bm_<uuid>".

Two things about that response are worth knowing before you build on it. The "visibility": "private" it reports is a fixed literal, not a stored field: benchmarks have no visibility column, PATCH ignores visibility if you send it, and a benchmark's real visibility is derived from its runs (§8.4). And while source, sources, paper and dataset do exist as provenance fields on a benchmark, they are not settable through this endpoint.

8.6 POST /v1/benchmarks/{bm_id}/subsets: create a subset

Subsets hold the actual task rows. Non-rolling subsets freeze after creation; rolling subsets accept appends indefinitely.

Pass a grader to make the subset runnable: the recipe (setup/boot/run/capture, judge settings) the pipeline executes your samples with. Without one this is a catalog-only publish, which is the right shape for a harness-backed benchmark that brings its own scorer. Your samples are stored either way; the response reports which you got in runnable, and a run pinned to a subset that has none fails immediately with product_benchmark_no_recipe rather than part-way through setup.

POST /v1/benchmarks/bm_01J…/subsets
{
  "slug": "parallel-function-v4",
  "name": "Parallel Function v4",
  "description": "200 parallel function calling questions from BFCL v4.",
  "rolling": false,

  "tasks": [
    {
      "task_id": "parallel_function_001",
      "input":   { "question": "…", "functions": [ /* OpenAI tool shape */ ] },
      "expected": [ { "name": "search_flights", "args": { /* … */ } } ]
    }
  ]
}

slug and name are both required. The response is a short acknowledgement, not the full Subset object:

{ "id": "sub_bm_01J…:7281bbc31736", "slug": "parallel-function-v4",
  "name": "Parallel Function v4", "created": true, "rolling": false,
  "sample_count": 200, "benchmark_id": "bm_01J…", "runnable": true }

Like an impl id, a subset id is content-addressed: sub_<benchmark id>:<first 12 hex of a fingerprint over the task set>.

The fingerprint covers the TASKS, not the slug, so posting the same task set under a different slug is an update of the existing subset, not a second one: same id, and your slug / name / recipe replace the ones already on that row. That is what makes re-authoring idempotent, but it means the slug cannot be used to keep two variants of one task set apart. The response says which happened: 201 + "created": true for a new row, 200 + "created": false for an update, and renamed_from_slug appears only when your slug displaced a different one. Watch for that field: the row you renamed may already be referenced by finished runs.

Shape constraints:

• input + expected must be JSON-serializable.
• 1000 tasks per request (the only enforced size cap). Chunk larger datasets and use Idempotency-Key per chunk to make resumable uploads safe.

Keep task_id unique within the subset. Nothing rejects a duplicate, but task_id is how §8.9's upsert / delete address a sample, so a repeated one makes those edits ambiguous.

Rolling subsets accept appends:

POST /v1/subsets/<subset_id>/tasks:append
Idempotency-Key: bugs-append-2026-04-23

{ "tasks": [ /* same shape */ ] }

The server appends the rows and bumps sample_count. Appends are not versioned: there is no snapshot marker and no way to address the subset as it stood at an earlier size, so a run resolves the task set as it stands at the moment it runs. If you need a set that cannot move under a comparison, publish it as a non-rolling subset. Appending to a frozen (non-rolling) subset fails 409 subset_frozen.

8.7 POST /v1/impls: register a scorer

An impl is a task_impl.py (or future equivalents) bound to a subset. Impls are immutable: the code defines the fingerprint, and changing the code means registering a new impl.

Required fields are subset, label, and code; benchmark and language are ignored if sent (the subset already implies the benchmark, and Python is the only impl language).

POST /v1/impls
{
  "subset": "<subset_id>",
  "label":  "strict-set scorer v1",
  "code":   "import json\nfrom bench_task import BenchTask\n\nclass Task(BenchTask): …"
}

Returns the Impl object:

{
  "id": "imp_bm_01J…:c8a1b2d3e4f5",
  "subset": { "id": "sub_bm_01J…:7281bbc31736" },
  "label": "strict-set scorer v1",
  "fingerprint": "c8a1b2d3e4f5",
  "created_at": "2026-04-23T14:00:00.412+00:00"
}

The id is not opaque: it is imp_<benchmark id>:<first 12 hex of the code's content hash>, and fingerprint repeats those same 12 characters. Register the same code twice and you get the same id back, which is what "impls are immutable" means in practice.

GET /v1/impls/{id}/code (fetching the stored code back) is not yet available.

8.8 Updating and deleting

PATCH /v1/benchmarks/{id}
PATCH /v1/subsets/{id}
PATCH /v1/impls/{id}

DELETE routes for benchmarks, subsets, and impls are not yet available.

A PATCH does not respond with the updated object. Benchmark and subset answer {"id": "…", "ok": true}; impl answers the same plus an echo of whichever of label / description / official it changed. Re-read the resource if you need its full state back.

If-Match (§3.8) is enforced on PATCH /v1/benchmarks/{id} only. The subset and impl handlers accept the header and ignore it, so a stale write there succeeds silently rather than answering 412. Send it anyway, since it costs nothing and starts working as those handlers migrate, but do not rely on it for read-modify-write on subsets or impls yet.

official=true on an impl is a privileged write. A non-admin PATCH that touches official fails 403 admin_required before the rest of the patch is considered, so a mixed patch changes nothing. A PATCH naming no recognized field at all is 400 invalid_argument, and the detail lists what is patchable.

The rolling→false transition is one-way: once you freeze a subset, you can't reopen it. Create a new subset if you need to keep accreting.

Task rows are never patched in place. On a rolling subset you can append; on a frozen subset you can't. If a task is wrong, register a new subset with the correction, and let the registry show both so consumers can see which runs used which.

8.9 Editing a BUILT benchmark (recipe + samples)

The routes above edit catalog prose. A benchwright-BUILT product benchmark also has a recipe (the sandbox script set that boots the product, drives it, captures its output and judges the result) and its samples. Those live in the authored definition, and the owner edits them here:

GET   /v1/build/subsets/{id}
PATCH /v1/build/subsets/{id}/recipe
PATCH /v1/build/subsets/{id}/tasks

{id} is a subset id, a benchmark id, or a benchmark name — the last two resolve to that benchmark's live subset, so the handle the registry printed is enough and you do not have to look an id up first. A subset id from /v1/registry/benchmarks also works: catalog ids and authored ids come from different fingerprints, and a catalog one is mapped to its benchmark's authored subset for you. A name matching more than one benchmark answers 409 ambiguous listing the ids rather than picking one, since this route changes how a benchmark scores. A subset that reached the catalog through a run has no authored recipe at all, and says so (404 not_authored) rather than implying the id was wrong. All three are gated on the benchmark's owner (admins too); anyone else gets 404. The GET needs registry:read (diffing a recipe before proposing a fix is a read); both PATCHes need registry:publish.

# broaden a mint that was too narrow, raise a timeout
curl -X PATCH https://benchwright.ai/v1/build/subsets/sqlbot:9f2c1a7b0e44/recipe \
  -H "Authorization: Bearer $BW_PAT" -H 'Content-Type: application/json' \
  -d '{"config":{"boot":"mint --match \"^clip-[0-9]+$\"","run_timeout":900},
       "note":"mint missed multi-digit clips"}'

# retarget one sample
curl -X PATCH https://benchwright.ai/v1/build/subsets/sqlbot:9f2c1a7b0e44/tasks \
  -H "Authorization: Bearer $BW_PAT" -H 'Content-Type: application/json' \
  -d '{"upsert":[{"task_id":"q2","input":{"at":"00:04:40","clip":"c3"}}]}'

How each one versions. A recipe patch is a merge: keys you send replace theirs, an explicit null deletes one, the rest is untouched. The subset id does not move (it addresses the samples, which did not change) and the scorer version moves instead: the recipe is content-addressed into task_impl.py, so the next run mints a new impl id and every run stays stamped with the recipe that graded it. The response carries recipe_fingerprint and changed_keys; an edit that changes nothing answers changed: false.

A tasks patch (tasks to replace the set, or upsert/delete by task_id) forks: the edited sample set gets its own content-addressed subset carrying the recipe forward, the response names both ids, and the old subset is retired so runs stop resolving to it. Editing a retired subset answers 409 superseded naming its successor; editing samples back to a retired set revives that subset rather than minting a third.

The GET returns an ETag, and both PATCHes honor If-Match (§3.8) — worth using here, since editing a recipe is read-modify-write and a mismatch answers 412 precondition_failed instead of overwriting someone else's fix. Other failures use the standard codes: 404 not_found (unknown subset, or not yours), 403 scope_insufficient, 400 invalid_argument (an empty sample set, over 2000 tasks, or a recipe patch carrying neither config nor type; either one alone is fine, so {"type": "…"} with no config is a valid retype).

The same three, from the CLI:

benchwright bench recipe show my-bench
benchwright bench recipe set  my-bench --set run_timeout=900 --set trials=3 --note 'why'
benchwright bench samples set my-bench --upsert-file fix.jsonl --note 'why'

9. Resource: Secrets and host policy

Two related collections:

• Secrets: named blobs of credential value (an API key, an HTTP bearer token, a basic-auth pair).
• Rules: per-secret host-pattern entries that tell the server-side injection proxy how to attach the secret to outbound requests.
• Host policy: per-user allow/block list for outbound HTTP from the pipeline.

9.1 GET /v1/secrets

{
  "data": [
    {
      "id": "<secret_uuid>",
      "name": "OPENROUTER_API_KEY",
      "description": "personal workspace key, sandboxed org",
      "created_at": "2026-02-14T…Z",
      "updated_at": "2026-04-10T…Z",
      "rules": [
        {
          "id": "<rule_uuid>",
          "secret_id": "<secret_uuid>",
          "host_pattern": "openrouter.ai",
          "inject_kind": "bearer",
          "inject_target": "Authorization",
          "created_at": "2026-02-14T…Z"
        }
      ],
      "injections": ["product_sandbox"]
    }
  ],
  "meta": { "payer": { /* whose vault this call is scoped to, §2.5 */ } }
}

Secret values never surface — there is no masked preview and no last-used timestamp on responses. Rule ids are bare UUIDs.

rules and injections are two different mechanisms. A rule tells the injection proxy to attach the secret to outbound HTTP matching a host pattern. An injection exports the secret as an environment variable, under its own name, when a machine boots. injections is a flat array of surfaces, each one product_sandbox (the sandbox your benchmark runs in) or operator (your operator's machine); an empty array means the secret is only ever used through its rules.

9.2 POST /v1/secrets

{
  "name": "OPENROUTER_API_KEY",
  "description": "...",
  "value": "sk-or-...",
  "rules": [
    {
      "host_pattern": "openrouter.ai",
      "inject_kind": "bearer",
      "inject_target": "Authorization"
    }
  ],
  "injections": ["product_sandbox"]
}

inject_kind must be one of bearer, header, basic, query; anything else is 400 invalid_argument. inject_target is the header name or query parameter name. Creating a rule with host_pattern=foo.example.com also upserts user_host_policy(mode=allow) for that host, so the proxy doesn't block it.

A rule sent inline here is skipped silently when any of host_pattern, inject_kind or inject_target is blank: the secret is still created, minus that rule. Send complete rules, or add them afterwards through §9.5, which validates each one on its own.

injections (optional) is the boot-time env-var surface list from §9.1: each entry must be product_sandbox or operator. Because an injected secret becomes an environment variable under its own name, a non-empty injections also requires the secret name to be a valid identifier (^[A-Za-z_][A-Za-z0-9_]*$); otherwise the create fails 400 invalid_argument rather than quietly renaming your variable. Renaming a secret through PATCH is held to the same rule while it has injections.

POST /v1/secrets returns 201 with {"id": "<secret_uuid>"}. Two conflict cases: 409 name_conflict when a secret with that name already exists, and 409 cloud_managed when the name is OPENROUTER_API_KEY while the account is enrolled in Cloud inference (that key is platform-managed).

9.3 Secret rotation

PATCH /v1/secrets/<secret_uuid>
Idempotency-Key: …
{ "value": "sk-or-new-key-value" }

Returns 200 with {"id": "<secret_uuid>"}. A run already in flight keeps using the previous value (the proxy caches per-run); new runs get the new one.

One qualifier on that: a run resuming from a halt or a pause re-reads the vault as it comes back, so a run that was paused across a rotation resumes on the new value. Only a run that never stops keeps the old one for its whole life.

PATCH without a value field updates only the description / name — the stored secret is not touched. Use this to relabel without invalidating in-flight runs.

9.4 Deleting a secret

DELETE /v1/secrets/<secret_uuid>

Returns 204 and cascades into the secret's rule rows. Active runs are unaffected (the proxy already has its per-run copy); the next launch that needs a matching credential will fail 412 no_provider_secret until you create a replacement.

Deleting also withdraws the host-policy entries the secret's rules added for it (the ones §9.6 reports with source=auto_from_secret), so removing a credential does not leave its hosts allowed. An entry you added yourself (source=user) survives, since you asked for that one independently.

9.5 Managing rules independently

For day-to-day allowlist tuning the rule collection has its own URLs, so you don't need to PATCH the whole secret to add or drop one host pattern:

POST   /v1/secrets/<secret_uuid>/rules
       { "host_pattern": "openrouter.ai",
         "inject_kind":  "bearer",
         "inject_target": "Authorization" }

DELETE /v1/secrets/<secret_uuid>/rules/<rule_uuid>

POST returns 201 with the new SecretRule wrapped in a single-element array (the database's representation is passed through verbatim), so read [0] rather than the object directly. inject_kind must be one of bearer / header / basic / query (anything else is 400 invalid_argument); inject_target is the header name or query parameter. Creating a rule auto-allows the host pattern in user_host_policy (same behavior as inline rules under §9.2). DELETE is 204.

9.6 Host policy

GET    /v1/host-policy
POST   /v1/host-policy              { host_pattern, mode }
DELETE /v1/host-policy/{id}
POST   /v1/host-policy:starter-pack  # adds github, pypi, huggingface…
PATCH  /v1/host-policy/default        { mode: "allow" | "block" }

GET /v1/host-policy
{
  "data": [
    { "id": "<uuid>", "host_pattern": "openrouter.ai", "mode": "allow",
      "source": "auto_from_secret", "created_at": "2026-02-14T…Z" }
  ],
  "default_mode": "allow"
}

default_mode is allow. The list is not an allowlist that everything else falls outside of: by default a run may reach any host, and the entries refine that. Flip the default with PATCH /v1/host-policy/default {"mode": "block"} when you want the opposite posture, and read default_mode before you interpret the list, because the same rows mean different things under each.

source records who added an entry: user (you did), auto_from_secret (a secret rule added it, §9.2) or starter_pack. It matters on delete (§9.4).

mode on POST is allow or block, and the reply is 201 {"ok": true}. POST /v1/host-policy:starter-pack adds the five hosts almost every run wants (github.com, raw.githubusercontent.com, pypi.org, files.pythonhosted.org, huggingface.co) and answers {"ok": true, "added": 5}.

GET /v1/host-policy requires the secrets:read scope on PATs; writes require host_policy:write.

9.7 Blocked-host log

GET    /v1/runs/{id}/blocked-hosts
GET    /v1/blocked-hosts
DELETE /v1/blocked-hosts?host=<host>

Returns hosts that the injection proxy refused during the run, with seen_count, first_seen, last_seen. Makes "trust on observation" a first-class workflow: run once, see what got blocked, decide which to allow, rerun.

The two reads take different scopes, because they answer different questions. GET /v1/runs/{id}/blocked-hosts is part of the run, so it needs runs:read (same as /v1/runs/{id}/egress-routes, §9a.4). The account-wide GET /v1/blocked-hosts is a vault-shaped read and needs secrets:read.

DELETE /v1/blocked-hosts?host=<host> clears the log rows for one host (requires host_policy:write; returns 204) — used after allowing a host so it stops appearing in the recently-blocked list.

9a. Resource: Local egress

Runs reach the internet from the datacenter by default. Local egress lets a run exit from a machine you control instead, which is what you want when a site will not serve a datacenter address or a page only renders correctly from a real home connection.

You register the machine once, then name it on a run. CLI: benchwright egress up|ls|rm and benchwright run --egress (see cli.md).

9a.1 Registering a machine

POST   /v1/egress/endpoints        # register, and heartbeat while serving
GET    /v1/egress/endpoints        # your registered machines
DELETE /v1/egress/endpoints/{id}   # forget one

These sit on runs:* rather than a scope of their own. A machine only ever exists to serve your own runs, so a token that can launch runs can offer a machine to run them from, and no existing token needs re-minting.

You choose the fingerprint, the server does not mint it. Send it as id in the shape eg_ plus 10 characters from a-z2-7 (^eg_[a-z2-7]{10}$); anything else is a 400, and the response echoes the id you sent. It is public by design (it travels in run commands and shared recipes), which is exactly why the registration is signed: the request must also carry public_key, a nonce, and a signature over them, and a fingerprint already held by a different key is refused 409 rather than re-pointed. benchwright egress up generates and stores the keypair for you.

The declared policy is mode plus allow_hosts, and the two are read together. allow_hosts carries the same list in every mode; mode decides what the list means, so neither field is meaningful alone:

A host that is not routed through your machine still reaches the internet by the normal path. This is a routing choice, not an access rule. To restrict what a run may reach at all, use host policy (§9).

Registering is also the heartbeat, and a heartbeat cannot change policy. The agent repeats this call while it serves. A request that omits mode keeps whatever is already stored rather than falling back to a default, so an older client that does not send the field cannot quietly rewrite a policy simply by staying alive. Only a request that states mode changes it, and the response echoes the policy now in force so a caller can see what it is actually serving instead of assuming its own defaults applied.

If a different agent process claims a fingerprint whose current holder is still live, the claim is allowed, since the usual cause is a restart, and it is recorded. GET /v1/egress/endpoints then returns contended_at and contended_detail on that endpoint. This matters because two agents on one fingerprint each declare their own policy, and a run freezes whichever was in force at launch.

9a.2 Asking for it on a run

Launch with metadata.local_egress: true to use whichever of your machines is serving, or name one with metadata.egress_endpoint: "eg_…".

metadata.wait_for_egress applies to the pinned form only. The launch-time liveness check runs when you named a specific machine: without wait_for_egress the launch is refused 412 egress_endpoint_unavailable if that machine is not serving right now, and with it the run is held until the machine comes online. A bare local_egress: true names nothing to check, so it gets no launch-time check at all: the run starts either way and picks up whichever machine is serving when it needs one. Pin the endpoint when you want the launch to fail fast on a machine that is down.

9a.3 Serving work

GET  /v1/local-egress/pending        # the next run waiting for this machine
GET  /v1/local-egress/{id}           # liveness while serving
POST /v1/local-egress/{id}/attach    # connected, the run may proceed
POST /v1/local-egress/{id}/fail      # could not connect, fail the run
POST /v1/local-egress/{id}/denials   # hosts your own allowlist refused

Your machine is behind your router, so nothing can call it. It asks for work instead: poll pending (runs:read), connect, then attach (runs:write) to release the run. GET /v1/local-egress/{id} is status only and returns no connection details, so a long-running session can check liveness without re-fetching credentials. benchwright egress up does all of this for you.

POST …/denials (runs:write, body {"denials": [{"host": "…", "count": N}]}) is how your machine reports what it refused. Those hosts land on the run's blocked-hosts card (§9.7) tagged source=home, alongside the ones the platform proxy blocked. Report them: a run that died because your own allowlist said no otherwise shows an empty card, and the reason it stopped is invisible from our side. The session id scopes the report to that run, so a stray call cannot land on someone else's.

9a.4 Where the traffic actually went

GET /v1/runs/{id}/egress-routes

Requires runs:read. One row per host the run reached:

{
  "data": [
    { "host": "example.com",   "route": "tunnel", "seen_count": 34,
      "first_seen": "2026-08-08T01:12:03Z", "last_seen": "2026-08-08T01:14:57Z" },
    { "host": "image.tmdb.org", "route": "direct", "seen_count": 8,
      "first_seen": "2026-08-08T01:12:11Z", "last_seen": "2026-08-08T01:13:40Z" }
  ]
}

route is tunnel when the host left through your machine and direct when it took the normal datacenter path. This is the run's own record of each decision as it made it, so you can confirm that only what you intended crossed your connection. The same list is on the run page. If you were also watching from your own machine, the two should agree, and a disagreement is worth a look.

10. Resource: Balance and top-ups

10.1 GET /v1/billing/balance

{
  "object": "balance",
  "balance":            { "amount_cents": 42155, "currency": "USD" },
  "month_spent":        { "amount_cents": 1230,  "currency": "USD" },
  "monthly_cap":        { "amount_cents": 10000, "currency": "USD" },
  "auto_refill": {
    "enabled": true,
    "amount":    { "amount_cents": 2500, "currency": "USD" },
    "threshold": { "amount_cents": 500,  "currency": "USD" }
  },
  "alert":        { "enabled": true, "at_cap_pct": 80 },
  "updated_at":   "2026-04-23T14:07:05.889Z"
}

?expand=topups,usage_trend,counters,tiers,subscription folds those blocks into the same response (§3.6); an unknown expand key fails 400 unknown_filter_field.

The payer block (§2.5) is top-level payer here, not meta.payer. This one endpoint differs from the token, secret, and settings responses, which nest it under meta. Read both spellings if your client is generic over the payer.

Every money object on this endpoint is whole cents rounded HALF_UP, with no amount_usd beside it; that exact field exists on run costs (§3.3, §4.1), where sub-cent figures are routine.

10.2 POST /v1/billing/topups

Browser-facing hosted Stripe Checkout (requires an interactive user; PATs need billing:write):

POST /v1/billing/topups
{ "amount_cents": 10000 }

amount_cents must be between 500 ($5) and 5,000,000 ($50,000).

Response:

{ "url": "https://checkout.stripe.com/c/pay/…" }

10.3 Top-up status

GET /v1/billing/topups/top_<uuid>

Reports "succeeded" once the payment landed; a top-up that has not succeeded (still pending, failed, abandoned) is a 404.

{
  "id": "top_<uuid>",
  "status": "succeeded",
  "amount": { "amount_cents": 5000, "currency": "USD" },
  "stripe_payment_intent_id": "pi_…",
  "created_at": "2026-04-23T14:05:00.412Z"
}

Top-up ids round-trip in their top_-prefixed form: that is what the response carries, and the path accepts it (a bare UUID works too, the prefix is stripped).

10.4 GET /v1/billing/transactions

Full ledger:

GET /v1/billing/transactions?kind=trace_debit&limit=100&offset=0

kind takes ONE exact value from stripe_topup, trace_debit, cli_credit, cli_set, cli_reset, support_credit, refund, adjustment, or the literal all to skip filtering (the same as omitting it). Anything else is 400 invalid_argument; there is no in: operator and no created_at filter here. Pagination is limit (max 500) / offset, with a total_estimate (§3.4).

That list is the filter allowlist, not the set of values a row can hold. The ledger records around twenty kinds (inference_usage, operator_usage, proxy_linger_usage, dev_env_starter, blog_narration and others), so an unfiltered page will return kinds you cannot filter on. Switch on kind defensively and give unknown values a generic row rather than dropping them.

Each row:

{
  "id": "txn_<uuid>",
  "kind": "trace_debit",
  "delta":         { "amount_cents": -412, "currency": "USD" },
  "balance_after": { "amount_cents": 41743, "currency": "USD" },
  "reference_id": "<uuid>",
  "reference_type": "run",
  "note": null,
  "created_at": "2026-04-23T14:05:00Z"
}

10.5 PATCH /v1/billing/controls

Change monthly cap, auto-refill, alert settings. Requires billing:write. Runs already mid-flight are unaffected until their next debit tick.

10.6 GET /v1/billing/usage-trend

Six-month bucketed spend, same shape as today's summary:

{
  "current_month_cents": 754,
  "previous_month_cents": 1230,
  "months": [
    { "label": "Nov", "month": "2025-11", "cents": 0 },
    …
    { "label": "Apr", "month": "2026-04", "cents": 754, "current": true }
  ]
}

current_month_cents and month_spent on §10.1 are the same ledger rows, bucketed two ways, so the current bar and the figure under your cap agree by construction. This one is exact; month_spent is rounded to whole cents.

10.7 GET /v1/billing/export

CSV export with stable column order and RFC 4180 quoting. The Content-Disposition header carries the filename. ?type=transactions (default) exports the ledger; ?type=runs exports per-run rows.

10.8 GET /v1/billing/statement

Print-friendly HTML monthly statement. Returns text/html with a canonical layout (header, period range, debits + credits table, running balance, footer). One query parameter:

GET /v1/billing/statement?month=2026-04

Drop it straight into a browser tab or stream into wkhtmltopdf for a PDF copy. The shape never changes mid-month, so it's also suitable as a stable artifact to attach to procurement tickets.

10.9 POST /v1/billing/subscription

Changes the account's membership tier. {"tier": "<name>"} starts a Stripe Checkout flow for a paid tier (response carries the checkout url); {"tier": "free"} cancels any active subscription at period end (idempotent — already-free accounts get an ack). This endpoint returns legacy {"error": "…"} bodies on failure (§3.9).

An optional "interval" picks the billing period: year / yearly / annual selects the tier's annual price, anything else (including omitting the field) bills monthly.

10.10 POST /v1/billing/verify-subscription

Reconciles a completed tier-upgrade Checkout without waiting on the Stripe webhook. Requires billing:write.

POST /v1/billing/verify-subscription
{ "session_id": "cs_…" }

Post the session_id Checkout hands back on the success return and the tier is applied server-side, idempotently, so a webhook that arrives later is a harmless no-op. Useful anywhere the webhook may not reach you, and as a belt-and-braces step after POST /v1/billing/subscription so a paid upgrade never sits unapplied. A session that is not yet complete answers {"ok": false, "reason": "pending"}; one belonging to another user is 403. The webhook remains the authority for ongoing lifecycle (renewals, cancellations, dunning).

11. Resource: Settings

Per-user KV settings, carried forward from today's user_settings table. The allowed keys are constrained to prevent the write route from becoming a database write primitive.

GET    /v1/settings
PUT    /v1/settings/{key}      { "value": <any JSON> }
DELETE /v1/settings/{key}
GET    /v1/settings:schema

There is no per-key GET — read everything with GET /v1/settings and pick the key client-side.

Response:

{
  "data": {
    "litellm_proxy_ui_enabled": { "value": true,  "updated_at": "…" },
    "run_narration_enabled":    { "value": false, "updated_at": "…" }
  },
  "meta": { "payer": { /* whose settings these are, §2.5 */ } }
}

A team member reads the owner's settings, which is what meta.payer names: run defaults follow the wallet the runs bill to.

/v1/settings:schema returns the flat allowlist of key names. Read it rather than hardcoding the list, since keys are added additively:

{ "keys": ["score_display_format", "litellm_proxy_ui_enabled", "litellm_full_logging_enabled",
           "run_narration_enabled", "goal_narration_enabled", "phase_models", "analyzer_model",
           "operator_harness", "operator_model", "default_model_under_test", "default_harness",
           "narrate_enabled", "narrate_voice"] }

PUT or DELETE against a key outside that list returns 400 invalid_argument.

value is any JSON element, not just a boolean or a string: analyzer_model takes {"model": "<id>", "provider": "openrouter|together"}.

Keys that feed a launch or a pod boot are validated on write, so a bad value fails here rather than on every future run. All of these are 400 invalid_argument:

12. Resource: Bug reports

Authenticated users can file a bug against any run they can see. Reports go to the Benchwright team's triage queue.

POST /v1/bug-reports
{
  "description": "Atomize wrote a scorer that always returns 1.0. Repro: …",
  "trace_id": "<uuid>"        // optional, omit for a generic report
}

description is capped at 8000 characters. Response is 201:

{ "id": "<uuid>", "status": "open" }

status is one of open, triaged, in_progress, resolved, wontfix and is updated by the Benchwright team; you see your own reports surface in the run detail view they were reported against.

12a. Resource: Operators

Message your operators and read their replies over the API — the same conversations you see under /manage/operators, without opening a browser. Handy for scripts, CI steps, and other agents that need to assign work and poll for an answer.

POST   /v1/operators                         # create an operator and boot it
GET    /v1/operators                         # list your operators
POST   /v1/operators/{id}/start              # start a stopped one, or restore chat
POST   /v1/operators/{id}/shutdown           # release its machine (the operator is kept)
GET    /v1/operators/{id}/messages           # read conversation (+ queued messages)
GET    /v1/operators/{id}/events             # the same conversation as SSE
GET    /v1/operators/{id}/files/{name}       # fetch an image the operator shared
GET    /v1/operators/{id}/pending            # list messages still outstanding
DELETE /v1/operators/{id}/pending/{ts}       # take one outstanding message back
DELETE /v1/operators/{id}/pending            # cancel all queued messages
POST   /v1/operators/{id}/message            # deliver one chat turn
POST   /v1/operators/{id}/stop               # interrupt the current turn (not teardown)

CLI: benchwright operator ls|msg|log|create|start|shutdown (see cli.md).

stop and shutdown are different, and the names are kept apart on purpose. stop interrupts whatever the operator is working on right now and leaves it running. shutdown releases the machine it runs on. There is no DELETE /v1/operators/{id}: that verb is reserved for deleting an operator record, which nothing does yet, and a DELETE that left the record in place would be lying about what it did.

A write-only PAT can post/cancel but not list or read; a read-only PAT can list/read pending but not post or cancel. Mint both scopes when you need the full loop (operators:read + operators:write).

Cancelling a queued message

POST …/message writes a file into the operator's on-box inbox ($ACP_DIR/inbox on the operator machine, fronted by operator-ws). The message sits there until the bridge is ready and drains it into a turn, and it stays withdrawable for a while after that:

# See what is still outstanding
curl -sS "$BW_API_URL/v1/operators/$OP_ID/pending" \
  -H "Authorization: Bearer $BW_PAT"

# Take one back. The handle is the item's `ts` from the list above;
# POST /message returns no id.
curl -sS -X DELETE "$BW_API_URL/v1/operators/$OP_ID/pending/1700000123456" \
  -H "Authorization: Bearer $BW_PAT"

# Or clear everything still sitting on disk
curl -sS -X DELETE "$BW_API_URL/v1/operators/$OP_ID/pending" \
  -H "Authorization: Bearer $BW_PAT"

/pending answers "what am I still waiting on?", not "what is undelivered?" It reports every message of yours the operator has not answered, and tags each one with the stage it reached:

count and outstanding are both the length of pending, states carries the per-state counts (including answered), and state_meanings spells the same thing out inline. Only a withdrawable: true item can be taken back: DELETE on one that has been read answers 409 naming the state it reached, rather than claiming a withdrawal that did not happen. If the message is gone entirely, DELETE returns 404 pending_not_found — safe to retry; it will not corrupt an in-flight turn. There is no browser-only escape hatch required: the same callers that can send can list and cancel.

Images an operator shares

When an operator shows you a picture, the image is served straight from that operator's own machine — nothing is uploaded or stored on your account. The markdown in its reply points at /v1/operators/{id}/files/{name} (the browser uses the /api twin), and {name} is a plain filename: png, jpg, jpeg, webp or gif, up to 8 MB.

These images are temporary. They live with the operator, so once it stops, the URL returns 409 and the picture is gone. Save anything you want to keep:

curl -sS "$BW_API_URL/v1/operators/$OP_ID/files/chart.png" \
  -H "Authorization: Bearer $BW_PAT" -o chart.png

Reading messages

{
  "state": "idle",
  "model": "…",
  "turns": [
    { "role": "you|operator|note", "text": "…", "t0": 1700000000.0,
      "t1": null, "working": false, "index": 12 }
  ],
  "pending": [ { "ts": 1700000123456, "chars": 42, "preview": "first ~80 chars",
                 "state": "on_disk", "withdrawable": true } ],
  "total_turns": 40,
  "limit": 50,
  "since": null,
  "since_mode": "tail",
  "cursor": {
    "next_index": 40,
    "next_since_t0": 1700000000.0,
    "has_more": false,
    "resume_index": "i:40",
    "resume_t0": 1700000000.0
  },
  "since_semantics": "see cursor fields; page with resume_index or resume_t0"
}

since / limit semantics

• No since: last limit turns (since_mode=tail) — good default for “what just happened?”
• since as turn index — bare int < 1e9, or i:N / index:N: turns with absolute index ≥ N, then the first limit of those. Page with cursor.resume_index (e.g. ?since=i:40).
• since as epoch seconds — float, or int ≥ 1e9: turns whose t0 is strictly greater than since, then first limit. Poll with cursor.resume_t0 / next_since_t0.
• Responses include enough cursor fields to resume without re-fetching the whole transcript. has_more is true when another page of index mode remains.
• Very long turn texts may set "truncated": true and clip the body.

Examples:

# List operators
curl -sS "$BW_API_URL/v1/operators" \
  -H "Authorization: Bearer $BW_PAT"

# POST a turn
curl -sS -X POST "$BW_API_URL/v1/operators/$OP_ID/message" \
  -H "Authorization: Bearer $BW_PAT" \
  -H "Content-Type: application/json" \
  -d '{"text":"ping from the front door"}'

# Read recent turns (needs operators:read — write-only PATs get 403)
curl -sS "$BW_API_URL/v1/operators/$OP_ID/messages?limit=5" \
  -H "Authorization: Bearer $BW_PAT"

# Read only what arrived after the last cursor
curl -sS "$BW_API_URL/v1/operators/$OP_ID/messages?since=i:40&limit=50" \
  -H "Authorization: Bearer $BW_PAT"

Waiting instead of polling

Add wait=<seconds> and the request is held open until the conversation moves past your cursor, so following a chat costs one request per reply rather than one per tick. If nothing arrives before the wait elapses you get a normal 200 with an empty turns — reissue immediately, no sleep and no backoff:

CURSOR=""
while :; do
  page=$(curl -sS -G "$BW_API_URL/v1/operators/$OP_ID/messages" \
    --data-urlencode "wait=45" ${CURSOR:+--data-urlencode "since=$CURSOR"} \
    -H "Authorization: Bearer $BW_PAT")
  echo "$page" | jq -e '.turns[]' >/dev/null 2>&1 && echo "$page" | jq '.turns'
  CURSOR=$(echo "$page" | jq -r '.cursor.resume_index // empty')
done

wait is a hard upper bound on the whole request, enforced server-side around every upstream read — not just between them. Size your client timeout against it with a little slack and it will not fire.

The 50s cap is set by the edge, not by this server: requests through *.benchwright.ai are cut at 60s by the proxy in front of the API, which then retries the GET rather than failing it — so a longer wait costs you a 504 after ~180s and three duplicate reads on the origin. Asking for more than the cap is not an error; it is clamped. Poll in a loop rather than asking for one long wait.

If the operator pod never answered inside the window you still get a 200 shaped like an empty page, with "wait_timed_out": true and the wait actually served as "wait_effective_sec", so a wedged pod stays distinguishable from a quiet conversation. Both fields appear only on that path. An ordinary wait that simply elapsed with nothing to say carries neither, so treat their absence as "the conversation was quiet" rather than reading wait_effective_sec as a routine echo of your request. Your own since is echoed back as cursor.resume_index — never a fabricated cursor — so resuming from that response cannot skip turns.

Omit since and the wait anchors to the newest turn, i.e. "block until the next thing is said". Pass cursor.resume_index from the previous response to resume exactly where you left off.

Send a turn and block for the finished reply in one step:

curl -sS -X POST "$BW_API_URL/v1/operators/$OP_ID/message" \
  -H "Authorization: Bearer $BW_PAT" -H "Content-Type: application/json" \
  -d '{"text":"what is the pass rate on the last run?"}'

curl -sS -G "$BW_API_URL/v1/operators/$OP_ID/messages" \
  --data-urlencode "since=$CURSOR" --data-urlencode "wait=45" \
  --data-urlencode "wait_for=idle" -H "Authorization: Bearer $BW_PAT"

Read the cursor before sending, so a fast reply can't land in the gap.

Keep each request under any proxy idle timeout on your side (45s is a safe default). From the CLI this is already wired up:

benchwright operator msg yara "ship it" --wait 5m   # blocks, prints the reply
benchwright operator log yara --follow              # server-side wait, no ticker

Streaming the conversation (/events) — prefer this over wait

GET /v1/operators/{id}/events pushes turns as they happen over Server-Sent Events. One connection, no polling, no 50s ceiling:

curl -sN "$BW_API_URL/v1/operators/$OP_ID/events" -H "Authorization: Bearer $BW_PAT"

id: i:14
event: ready
data: {"cursor":{"resume_index":"i:14"},"wait_for":"turn","heartbeat_sec":15}

: ping

id: i:16
event: turns
data: {"state":"idle","turns":[…],"cursor":{"resume_index":"i:16"}}

Cursors. No since anchors to the newest turn — you get what's said next, not a replay of history. since=i:N replays from N. Every event carries an id:, which a browser EventSource echoes back as Last-Event-ID automatically, so reconnects resume exactly where you left off without tracking anything yourself. wait_for=idle streams only completed replies.

Why this exists. wait is capped at 50s because our edge cuts any request that stays silent for 60s — and then retries it, so a longer wait costs you a 504 after ~180s and duplicate work upstream. That limit is on silence, not duration: a stream that keeps emitting is never cut. The 15s heartbeat is what makes /events immune, so don't strip it if you proxy this.

Note a browser EventSource can't set an Authorization header, so from a browser you need a same-origin session rather than a PAT.

13. Rate limits and quotas

Read + write quotas are per account (not per token — multiple PATs share the same bucket), and scale with your membership tier. Team members count against the team owner's bucket.

Tunable in public.membership_tiers.{read_rpm,write_rpm} — operators can revise the bands without a code change.

Run-shape quotas (separate from API rpm):

• Concurrent running runs / account: 1 (free), 3 (entry), 15 (growth), 50 (scale) — see membership_tiers.max_run_concurrency.

Headers returned on every authenticated response (an anonymous read of a public run or of the catalog carries none, because there is no account to count against):

X-RateLimit-Limit: 300
X-RateLimit-Remaining: 247
X-RateLimit-Reset: 1713888360

429s include Retry-After in seconds and a problem+json body.

Callers can read their own live counters:

GET /v1/users/me/rate-limits

{
  "user_id": "<uuid>",
  "read":  { "limit": 300, "count": 53, "remaining": 247, "reset_seconds": 22, "window_seconds": 60 },
  "write": { "limit": 60,   "count": 2,  "remaining": 58,  "reset_seconds": 22, "window_seconds": 60 }
}

For team members the counters are the team's shared bucket, not a private shadow.

Admin can raise per-account concurrency and read ceilings; rate limits apply at the account level regardless of how many PATs the account holds.

14. Error reference

Stable code values clients should handle:

Also emitted, and worth handling where you touch the surface that raises them: 403 team_readonly (the general team-write guard, §2.5), 409 fork_unsupported (§4.3), 409 replay_unsupported (§4.2.3), 409 delete_blocked (§4.7), and 409 superseded (§8.9).

Not every failure is problem+json. An unhandled server error is served by the framework's default handler, so a 500 carries no code and no request_id. Key off the status alone there, and never assume a JSON body on a 5xx. There is also no 422 on this API: validate typed fields client-side, since bad values come back as 400 invalid_argument.

501 vs 503. 501 means this deploy doesn't have that feature wired — a permanent answer for the deploy you're talking to, so retrying won't help. 503 is reserved for a backend that IS configured but is currently down or degraded; that one is worth retrying.

15. Event type reference

The set below mirrors ai.benchwright.port.pipeline.events.PipelineEvent and is what shows up inside the SSE stream and the /events JSON array.

ModelProxyCredentials no longer carries the proxy UI password. It used to, in plain text, which meant publishing a run published a live credential — any runs:read token, and once the run was public or unlisted, anyone at all. The password is now held server-side and never appears on an event; the event keeps only the non-secret ui_url + ui_username, and the events API drops secret-named fields from any older event that still has one. Reaching the proxy dashboard goes through /litellm/<run_id>, which authorizes the caller. ProxySandboxLinger extends the window that proxy stays reachable.

16. Worked examples

Two end-to-end workflows that exercise the API.

16.1 Launch a run, stream to terminal, gate CI on score

# 1. Prerequisites (one-time): add a provider secret + rule.
curl -sS https://benchwright.ai/v1/secrets \
  -H "Authorization: Bearer $BW_PAT" \
  -H "Idempotency-Key: setup-openrouter-v1" \
  -d '{
    "name": "OPENROUTER_API_KEY",
    "value": "sk-or-…",
    "rules": [{
      "host_pattern": "openrouter.ai",
      "inject_kind": "bearer",
      "inject_target": "Authorization"
    }]
  }'

# 2. Top up if needed.
curl -sS https://benchwright.ai/v1/billing/topups \
  -H "Authorization: Bearer $BW_PAT" \
  -d '{"amount_cents": 5000}'
# open the returned `url` in a browser, pay

# 3. Launch a run. Record $RUN in the CI job to correlate later —
#    launch metadata is not echoed back (§4.2.1 natural-language notes).
RUN=$(curl -sS https://benchwright.ai/v1/runs \
  -H "Authorization: Bearer $BW_PAT" \
  -H "Idempotency-Key: ci-build-8812" \
  -d '{
    "request": "Run MMLU Abstract Algebra on openai/gpt-4o-mini via OpenRouter",
    "model": { "model": "openai/gpt-4o-mini", "provider": "openrouter" },
    "constraints": { "max_cost_cents": 500 },
    "metadata": { "autonomous": true, "limit": 100 }
  }' | jq -r .id)

# 4. Stream events (CLI sugar; under the hood this is SSE).
curl -N https://benchwright.ai/v1/runs/$RUN/events:stream \
  -H "Authorization: Bearer $BW_PAT"

# 5. When the stream closes, pull the summary.
curl -sS https://benchwright.ai/v1/runs/$RUN/tasks | jq .summary
# { "total": 100, "passed": 82, "pass_rate": 0.82, … }

# 6. Exit non-zero if below gate.
SCORE=$(curl -sS https://benchwright.ai/v1/runs/$RUN/tasks | jq -r .summary.pass_rate)
awk -v s="$SCORE" 'BEGIN { exit (s < 0.80) }'

16.2 Fork to iterate

Your run came back with a scoring bug. The fix is in a description, not a code patch:

PARENT=$RUN
FORK=$(curl -sS https://benchwright.ai/v1/runs/$PARENT/fork \
  -H "Authorization: Bearer $BW_PAT" \
  -d '{
    "description": "Change score() to normalize whitespace before comparing",
    "from_phase": "atomize"
  }' | jq -r .id)

# Same streaming and summary endpoints work on the fork.

17. Deprecation and stability

/v1 is not frozen yet. Benchwright is pre-GA, and until it ships generally available the API can change shape without a version bump and without a deprecation window. If you are building against it now, pin nothing you cannot re-pin, and read this document rather than assuming a field you saw last month is still there.

What already holds:

• Additive changes (new fields, new endpoints, new event types) ship at any time. Clients must tolerate unknown fields.
• Ids, error code values and event type names are the parts we are most careful with, because they end up in your code rather than on a screen.

What starts at GA:

• /v1 freezes against breaking changes.

• Deprecations are announced on the response via:

  Sunset: Tue, 01 Jan 2027 00:00:00 GMT
  Deprecation: true
  Link: <https://docs.benchwright.ai/migrate-v2>; rel="deprecation"
  

  at least 6 months before removal. (A machine-readable /v1/status deprecation feed is planned but not yet available.)

• New resources land under /v1/preview/... first, with no SLA, for at least one month before promotion.

• /v2 will exist only if there is a shape we cannot evolve backwards-compatibly. When it ships, /v1 receives security fixes and continues to serve for 18 months.

Appendix B. OpenAPI

An OpenAPI 3.1 spec lives at:

GET https://benchwright.ai/v1/openapi.json

The spec is hand-maintained (api/src/main/resources/openapi.json), and it currently lags the route table. The whole /v1/operators surface, /v1/users/me/rate-limits, /v1/billing/subscription, /v1/billing/verify-subscription, teams, local egress and egress endpoints are all absent from it. Use it as a reference for the paths it does describe; this document is the authoritative list of what /v1 serves.

The SDKs are hand-written rather than generated from it, and CI checks only that the spec parses and still describes /v1/runs, so a route missing from the spec is drift, not a signal that the route is unsupported.