Benchwright CLI

Last modified September 10, 2026

benchwright is a single Go binary that wraps the public API. It covers most of what the dashboard does — launch and stream runs, author benchmarks, message operators and read their replies, manage provider secrets and host policy, top up your balance — plus a couple of shell-only conveniences (@last, --gate, idempotency key auto-derivation, table/ndjson output for piping). Tokens themselves are minted in the web UI (/manage → API Keys); webhook management ships once /v1/webhooks does.

For the underlying HTTP surface, see api.md. The CLI is just an ergonomic shell over the same endpoints, with the same auth (PAT) and the same error codes.

Install

# macOS / Linux — install script
curl -fsSL https://benchwright.ai/install.sh | sh

# Pin a specific release in CI:
curl -fsSL https://benchwright.ai/install.sh | BW_VERSION=v0.1.2 sh

The install script is at https://benchwright.ai/install.sh. It reads a handful of env vars:

Unless BW_NO_CONFIG=1, the script also seeds ~/.config/benchwright/config.toml (mode 0600) with default_profile and that profile's api_url, so login doesn't have to be told --api-url. It only writes the file when it is missing, and never overwrites a config you already have.

Or grab a binary directly:

• benchwright-linux-amd64
• benchwright-linux-arm64
• benchwright-darwin-amd64
• benchwright-darwin-arm64
• benchwright-windows-amd64.exe
• benchwright-windows-arm64.exe

Pin a release by inserting /vX.Y.Z/ before the filename — e.g. https://benchwright.ai/dl/v0.10.121/benchwright-linux-amd64.

Verify:

benchwright version

Once installed, upgrade in place with benchwright update. Pass --check to report what's available without touching anything, --version 0.4.1 to pin a release, or --force to reinstall the version you already have.

First run

benchwright login                                          # paste PAT (input hidden)
benchwright doctor                                         # config + auth + balance + secrets sanity
benchwright run "Run MMLU on openai/gpt-4o-mini via OpenRouter"

login writes a config file (TOML) under ~/.config/benchwright/config.toml (Linux/macOS) or %APPDATA%\benchwright\config.toml (Windows), respecting $XDG_CONFIG_HOME if set. The PAT is verified against the API before the file is written; bad tokens fail closed.

doctor is the safe pre-flight before automation: it confirms your balance is over the $1 launch floor, that you have at least one secret, and that at least one secret has a host injection rule. If any of those fail, doctor exits non-zero with the exact fix command to run.

Conventions

Shorthands

Aliases

bench = benchmark, billing ledger = billing transactions, secrets rm = secrets delete, token = tokens, token revoke = token rm, operator = operators = op, egress ls = egress list.

run --agent has been removed. It was the old spelling of run --driver and now exits with an unknown-flag error. "Agent" was ambiguous once Operator came to mean the goal-level agent that launches runs, so the per-run LLM is the Driver everywhere. The error is deliberate: a flag that still parsed but no longer pinned anything would launch the run on a different model and bill you for it.

Profiles

benchwright login --profile-name staging --api-url https://staging.benchwright.ai
benchwright --profile staging run ...

Resolution order at every command:

1. --profile <name> flag.
2. BW_PROFILE env var.
3. default_profile from the config file.
4. Hardcoded fallback default.

BW_PAT and BW_API_URL env vars override the resolved profile's fields — handy for CI where you don't want a config file on disk.

Output

Global --output chooses the format. Defaults are smart: table on a TTY, ndjson when piped or redirected. Override with one of:

• table — human-friendly aligned columns. Mostly used for ls-style commands.
• json — single pretty-printed JSON object.
• ndjson — one JSON object per line; ideal for jq -c pipelines.
• yaml — YAML, round-tripping the JSON field names.

Two caveats. run ls is the only command that implements all four formats. The other ls-style commands (secrets ls, operator ls, egress ls) branch on table and yaml only and fall through to a single pretty-printed JSON object, so --output ndjson and a bare pipe both give you one JSON object rather than a line per row.

And some commands don't take the format at all. run show, token ls and policy ls always emit JSON regardless of --output. run tasks prints a summary: line plus a task table unless you pass --full or --output json, and billing balance prints a human text block unless you pass --output json; both ignore yaml and ndjson, and neither changes shape when you pipe it.

Idempotency

run / run launch / run fork and billing topup derive an idempotency key from sha256(argv ‖ cwd ‖ $GIT_COMMIT) and send it as Idempotency-Key, so re-running the same command in the same directory at the same git SHA returns the same run/topup rather than creating a duplicate. Override with --idempotency-key=<value> (run/launch/fork), or disable with --no-idempotency (run/launch). Other mutating commands (secrets, bench, policy, token, settings) send no idempotency key.

In practice: re-running an identical benchwright run command re-attaches to the already-launched run (the CLI prints "identical command already launched — showing that run") and self-heals by launching fresh if that run has vanished.

Exit codes

The CLI maps HTTP status to a stable exit code so shell pipelines can distinguish failure modes:

Pass --debug to print the response body that produced the error.

Independently of HTTP status, a streaming run and watch exit non-zero when the run itself terminates with status=error, so CI can gate on the exit code without parsing output.

Commands

Running benchwright with no args, or benchwright <cmd> --help, prints the full Cobra-style help. The summary below is the commands grouped by topic.

login / logout

benchwright login   [--profile-name N] [--api-url URL] [--set-default]
benchwright logout  [--profile-name N]

Save or remove a PAT under a profile. PATs are validated against the API before the file is written. The PAT can also be piped via BW_PAT, but login is the friendly entry point.

doctor

benchwright doctor

Smoke-test the active profile: prints profile, api_url, balance, month, secrets count, rules count, and a scope coverage block naming which command groups the active token can actually run, based on its scopes. The groups it names are launch + read runs, download artifacts, manage secrets, manage host policy, top up balance, browse registry, author benchmarks, list/read operators and message operators. Exits non-zero with a fix command if balance is below $1, no secrets exist, or no secrets have host rules. Missing scopes are reported but don't fail the check (you may not need them); a MISS line points at the web UI (/manage → API Keys) — token minting is browser-session-only, so mint the wider token there and re-run login (see the token section).

run

The biggest command. Without subcommand, it launches and streams. With subcommands, it inspects, forks, halts, kills, publishes.

benchwright run [request]                                   # launch via natural language
benchwright run --benchmark bm_… --subset sub_… --impl latest:official --model m@p   # pinned
benchwright run --replay-from run_… --model m@p                                       # replay (full parent subset)

Pinned mode is a preview. The API accepts --benchmark / --subset / --impl, but the launcher does not resolve the pin yet. To anchor a run to a prior run's exact subset today, use --replay-from.

Replay subsetting (--only-failed / --indices / --task-id) is not honored yet. The API replays the full parent subset, so the CLI refuses these flags (rather than silently running — and billing — the whole subset). Replay without them re-runs everything; scoped replay lands when the launcher reads replay.{only_failed, indices,task_ids}.

Common flags (all modes):

• --model model@provider — the model.
• --driver <name> — the run's Driver, the LLM that executes it (default: server's choice). --agent was the old name for this flag and has been removed; it now errors. See Driver vs SUT vs Operator.
• --harness inspect|lmeval|lighteval|harbor|benchwright — eval harness (default benchwright; aliases like lm-eval, scratch, native are canonicalized).
• --harness-task <id> — harness task id, e.g. inspect_evals/mmlu_0_shot. When omitted, the server resolves the task from the request text.
• --limit-to <level>.<quantity>=<value> (repeatable) — every other ceiling, in one spelling. See Limits below.
• --metadata k=v (repeatable) — arbitrary tags.
• --public — auto-publish the run to public when it finishes successfully. Runs are private by default and nothing else flips that for you, so this is the flag that makes a score visible to everyone.
• --proxy-linger-seconds N — keep the LiteLLM proxy sandbox alive this many seconds after the run finishes; 0 tears it down immediately.
• --idempotency-key K / --no-idempotency — see above.
• --no-wait — print the run id and exit; don't stream.
• --verbose — stream every event type, not just phase + task results.
• --gate pass_rate=0.90 — exit non-zero if the gate fails. Currently pass_rate is the only key. The gate reads the finished run's tasks, so it can't be combined with --no-wait; the CLI rejects the pair up front rather than after you've paid for the run.

Limits

Every ceiling is spelled --limit-to <level>.<quantity>=<value>, repeatable.

benchwright run "..." --limit-to run.cost=5 \
                      --limit-to task.attempts=1 \
                      --limit-to attempt.duration=600

Money is in whole dollars and time is in seconds. The unit belongs to the quantity, never to the name — there is no _cents or _ms suffix to get backwards.

• run.cost vs run.model_cost — cost is everything (sandbox + driver + model); model_cost is only what your own provider key pays for. On a BYOK connection that is the one that matters, and it can also be declared on the connection itself so it applies to every run without a flag.
• run.tokens is the ceiling to reach for when a connection declares no rate: spend is then unknowable and a money ceiling can never fire, but tokens are still counted.
• Task ceilings fail that task and never retry it, and the run carries on. A task over its own budget is a failed task, not a failed run. They bound the whole retry chain, so task.cost=0.50 means fifty cents including retries — a per-attempt money cap would still allow task.attempts x that.
• attempt.duration multiplies. Worst case per task is attempt.duration x task.attempts, so 600 alone is 40 minutes on the default of 4, not 10. A hard 10-minute cap is --limit-to task.attempts=1 --limit-to attempt.duration=600.
• phase may name one phase: --limit-to phase:atomize.calls=250 overrides --limit-to phase.calls=100 for that phase alone. Useful for giving a build-heavy phase more room. Phases: find, fetch, analyze, setup, attach, atomize, materialize, report, fork.

A run stopped by a ceiling halts with halt_reason=limit_exceeded:<level>.<quantity>, so the reason says which one fired.

A limit the platform cannot enforce is refused, not accepted and ignored — attempt.cost is an error today, because spend is attributed per task and an attempt has no cost of its own. A ceiling that silently does nothing is worse than no ceiling.

A ceiling is where teardown starts, not a figure the run is guaranteed to stay under: the check runs every 2s and teardown keeps accruing after it fires.

The older flags — --max-cost, --max-duration-ms, --max-tasks, --task-attempts, --task-timeout-sec and --limit — have been removed. There is one spelling now. A script still passing them fails with an unknown-flag error rather than running uncapped, which is the failure you want: the units were in those names (--max-cost took cents), so a silent fallback would have been a 100x ceiling.

The raw API still accepts the old body keys (constraints.max_cost_cents, metadata.task_attempts, …) and converts them, so an un-upgraded client keeps its ceilings rather than losing them quietly.

Subcommands:

benchwright run launch  [request] …            # explicit form of bare `run`
benchwright run show    <run_id|@last>         # full run JSON
benchwright run ls      [--status … --benchmark … --visibility … --limit N]
benchwright run tasks   <run_id|@last> [--passed true|false --limit N --full]
benchwright run fork    <run_id|@last> -m "what changed" [--from-phase atomize]
benchwright run halt    <run_id|@last> [--reason …]    # stop it; the run stays readable
benchwright run kill    <run_id|@last> [-y] [--reason …]  # stop it now (eager teardown)
benchwright run delete  <run_id|@last> [-y]            # DESTROY it — irreversible
benchwright run publish <run_id|@last> [--public=false]

Publishing a run publishes its benchmark. Benchmark visibility is derived from having at least one public run rather than stored on the benchmark, so the two move together and there is no flag to separate them. If you want the benchmark to stay private, keep the run private.

The list subcommand is ls, and only ls. run itself takes an optional request argument, so benchwright run list is not an error and not a listing: it launches a run whose request is the word "list", and bills you for it. Same for any other word that isn't one of the subcommands above.

Stopping a run never destroys it. halt and kill both tear the sandboxes down and finalize the run with a kill reason — run show and run tasks keep working afterwards, so a run you stopped is still a run you can investigate. --reason "…" is recorded on the run and is what run show and watch report on its halted: line afterwards; without one, the record says only who stopped it. Only run delete removes anything, and it removes everything: the run record, its event stream, its per-task results and its registry summary. That cannot be undone, so its prompt asks you to type the run id rather than y.

delete removes one run, not a fork chain. Runs forked from the one you delete are kept. Whatever they were inheriting from it — the phase outputs and artifacts a fork reads from its ancestors instead of re-running — is copied onto them first, and they re-point at the deleted run's own parent, so a fork keeps working and keeps showing the same results. The prompt tells you how many will be kept before you confirm, and the response lists them under forks_adopted. Each one then reports forked_from_deleted in run show, because a fork whose parent is gone should say so rather than quietly present an unbroken lineage.

publish flips the run to visibility=public by default (--public defaults to true); pass --public=false to record the publish without the visibility flip.

Note the run object returned by show / ls carries the resolved model pin as {identifier, source, provenance} (e.g. identifier qwen/qwen3-32b); the {model, provider} shape is launch-body only.

halt flips a database flag the running pipeline polls every ~2s. kill is the immediate one — it tears down the agent JVM and any spawned sandboxes synchronously, asks for confirmation on a TTY, and prints the teardown summary. Use halt first.

run (and run launch) accept --harness … (default benchwright) and --harness-task to pin the harness task id; compute backend selection is account-level, not a CLI flag.

watch

benchwright watch [run_id|@last] [--compact] [--timeout 20m] [--verbose]

Follow a run until it finishes, then print its outcome: final status, the halt reason if it was halted, the score, the cost and the dashboard link. Use it after launching with --no-wait, or to re-attach from another shell. The run id is optional: with no argument watch follows @last, which is the run you just launched.

This is the supported way to monitor a run — you shouldn't need to write a polling loop. watch blocks until the run genuinely reaches a terminal state, reconnecting by itself (resuming from the last event it saw) if the connection drops, so a proxy timeout or a brief network blip doesn't end the wait early.

By default it renders a live event feed as the run progresses — phase boundaries, task scores, errors; --verbose shows every event type.

--compact drops the feed and prints only the final outcome block. That's the form to use in scripts and automation: one command, one answer, nothing to parse mid-stream.

--timeout (-t) bounds the wait, e.g. -t 20m. If the run is still going when the clock runs out, watch reports where it got to and exits non-zero — the wait didn't finish, so a watch && … chain must not proceed. Watch it again to keep waiting. Without --timeout it waits indefinitely.

Exit status: 0 when the run finishes successfully, non-zero when it ends in status=error, when it was halted (the halt reason is in the error message), or when --timeout expires with the run still in flight.

open

benchwright open <run_id|@last> [--print]

Open the run's dashboard URL in your default browser. --print writes the URL to stdout instead.

bench

benchwright bench ls
benchwright bench range <bm_id> <indices>                   # show existing runs that covered an ad-hoc task range, e.g. 0-5 or 0,2,4 (no stored subset needed)
benchwright bench create <name> [--description … --category … --tag t]

benchwright bench subset create <bm_id> --slug … --file tasks.jsonl [--grader-file grader.json --render-kind product_sandbox --name … --description … --rolling]
benchwright bench subset append <subset_id> --file new_tasks.jsonl
benchwright bench subset show   <subset_id>                 # the subset, grader recipe included

benchwright bench recipe show <subset_id|bm_id|bm_name>
benchwright bench recipe set  <subset_id|bm_id|bm_name> [--set k=v --json-set k=<json> --unset k --file recipe.json --note …]
benchwright bench samples set <subset_id|bm_id|bm_name> [--upsert-file fix.jsonl --delete task_id --file tasks.jsonl --note …]

benchwright bench impl ls <subset_id>
benchwright bench impl create <subset_id> [--file task_impl.py --language python --label … --benchmark … --description …]
benchwright bench impl from-run --from-run <run_id|@last> --benchmark … --subset … --label …   # not yet available

subset create / append read JSONL — one task object per line with fields {task_id, input, expected, …}. --grader-file takes a JSON file holding the subset's grader recipe; without it the subset is catalog-only and cannot run as a product benchmark, so pass it unless you only want the tasks catalogued. subset show reads a subset back with its grader recipe, which is how you tell which of the two you ended up with. On impl create, --file defaults to task_impl.py and --language defaults to python, so both are optional.

The recipe file may be written either flat (the runtime keys — base_image, boot, run, setup, capture, display, judge_model, … at the top level, the way recipe show prints them) or already wrapped as {"type": …, "config": {…}}. Both work: the flat form is wrapped for you, and render_kind is inferred as product_sandbox when the recipe carries sandbox keys, so --render-kind is only needed to override that inference. grader_type is accepted as a spelling of type, because that is the name recipe show prints. A file with no recipe keys at all is rejected outright, rather than published as a subset that reports runnable: true and then fails partway through a paid run.

There is no bench publish. A benchmark has no visibility of its own: it is public exactly when one of its runs is public. To put a benchmark on a public leaderboard, publish the run you want listed with benchwright run publish <run_id>; the registry derives the benchmark's visibility from its runs.

Fixing a built benchmark: recipe and samples

A benchwright-BUILT (product) benchmark carries a recipe — the sandbox scripts that boot the product, drive it, capture its output and judge the result, plus trials and the timeouts. recipe and samples edit that authored definition. Both take a subset id, a benchmark id, or a benchmark name (the last two resolve to that benchmark's live subset), and both require you to own the benchmark. A subset id copied from the registry works too: catalog ids and authored ids are minted by different fingerprints, and the catalog one is mapped across for you. A name that matches two benchmarks is refused with both ids rather than guessed at.

benchwright bench recipe show my-bench                       # read it back first
benchwright bench recipe set my-bench --set run_timeout=900 --set trials=3
benchwright bench recipe set my-bench --set 'boot=mint --match ^clip-[0-9]+$' \
  --note 'mint missed multi-digit clips'
benchwright bench samples set my-bench --upsert-file fix.jsonl \
  --note 'q2 pointed at the wrong frame'

The recipe IS the implementation. A product run records bench_task_product.py plus its recipe header as that run's task_impl.py, so impl_id is a hash of the two — edit the recipe and the next run registers a new impl, with the previous ones kept as history. That is why a subset carries one live recipe rather than a menu of them: to compare two ways of running the same tasks, publish them as separate benchmarks (or separate subsets with their own task sets), not as two recipes on one subset.

recipe set is a merge: keys you pass replace theirs, --unset k deletes one, and anything you don't mention is left alone. Values type themselves the way their JSON counterpart reads (900 is a number, true a bool, everything else a string); --json-set k=<json> is the escape hatch for objects, arrays, or a string that looks like a number.

samples set takes --upsert-file (edit or add, matched on task_id) with --delete (remove by task_id), or --file to replace the whole set. --file is a whole-set replacement, so it can't be combined with the surgical flags.

Both edits are versioned, not silent. A recipe change is content-addressed into task_impl.py, so the next run records a new impl id and runs from before the edit keep pointing at the scorer that actually graded them. A sample change forks a new content-addressed subset (the response names both ids) and retires the old one, because a subset id is a hash of its tasks. An edit that changes nothing answers changed: false rather than forking.

impl from-run is not yet available. The API only accepts a raw-code impl, so from-run exits with a "not available yet" message. To register the scorer the pipeline atomized for a run, grab its task_impl.py from the run's dashboard (Atomize artifact) and submit it as raw code with benchwright bench impl create <subset_id> --file task_impl.py.

secrets

benchwright secrets ls
benchwright secrets add    <name> [value]  --rule host:kind[:target] (repeatable) [--description …]
benchwright secrets rotate <name|sec_id>  [value]
benchwright secrets rm     <name|sec_id>  [-y]

The plaintext secret can be passed as the second positional, via --value, or piped on stdin. After creation values are never returned. rotate only changes the value — id, name, scopes, host rules carry over. In-flight runs keep using the old value (rotation is non-disruptive). add's --rule flag is repeatable; the host pattern supports *.foo.com suffix wildcards.

policy

benchwright policy ls
benchwright policy allow   <host>                          # e.g. *.openrouter.ai
benchwright policy block   <host>
benchwright policy default <allow|block>                   # mode for unmatched hosts
benchwright policy starter-pack                            # github / pypi / huggingface / …
benchwright policy blocked [run_id]                        # hosts the proxy denied

The proxy enforces this list at the mitmproxy layer in the proxy sandbox. default block + policy allow github.com pypi.org … gives you a tight outbound allowlist; default allow + a small blocklist is the looser default.

egress {#egress}

Runs normally reach the internet from the datacenter. Some sites will not serve a datacenter address, and some pages only look right from a real home connection. egress lets a run exit the network from your own machine instead.

benchwright egress up   [--mode hosts|all|except] [--allow HOST]… [--except HOST]…
                        [--label NAME] [--max-bytes 500MB] [--max-duration 30m]
                        [--max-runs N] [--run RUN_ID] [--save]
benchwright egress ls
benchwright egress rm <fingerprint>

up serves runs from this machine until you press Ctrl-C. It prints a fingerprint like eg_7k2m9x4p3q, which is how a run asks for this machine by name. The fingerprint is safe to paste into a run command or share in a recipe.

Three modes decide what actually crosses your connection:

hosts routes nothing until you say so, on purpose:

benchwright egress up --allow example.com --allow '*.example.com'

*.example.com covers the subdomains and the apex. Passing --except on its own selects except mode, so the common case is one flag rather than two.

What the host list means. It chooses which hosts leave through your machine. It does not restrict what the run may reach: a host you did not name still works, it just takes the normal path. To limit what a run can reach at all, use policy. That is a separate control and it applies whether or not you are serving egress.

The same host list means opposite things in different modes. This is worth seeing rather than reading. Four runs of one benchmark, same model, same machine, changing only the declared policy:

Runs A and B are the pair to look at. image.tmdb.org is named in B and not in A, and it is the one host that does not cross your connection in B. The seven hosts that never reached your machine in A all reached it in B, without either list changing meaning by accident: hosts says carry these, except says carry everything but these.

The control run is why the rest means anything. Without local egress the site served a block page and the run scored 0.0, so a passing run is evidence the tunnel did something rather than evidence it merely existed.

One thing you will notice comparing runs: the exit address can differ between sessions on the same machine. That is normal and not a fault. What a declared policy controls is which hosts cross your connection, not which address they emerge from.

Budgets end a session on your terms. All three are unlimited unless set:

benchwright egress up --allow example.com \
  --max-bytes 500MB --max-duration 30m --max-runs 3

--run <run_id> serves exactly one run and ignores every other. --save writes the current settings to ~/.config/benchwright/egress.toml so later sessions need no flags, and any flag you pass still wins over the file for that session.

ls lists the machines you have registered, and always shows the mode alongside the hosts, because a bare host list cannot tell you which of two opposite policies is in force. rm <fingerprint> forgets one.

   FINGERPRINT    LABEL        STATE    MODE    HOSTS
   eg_7k2m9x4p3q  laptop       online   hosts   only: example.com, *.example.com

Your declared policy is not changed by anything except you declaring it. up both registers the machine and keeps it alive, so it repeats while it serves. Those keep-alives only prove the machine is still there. They cannot alter the mode or the host list, including from an older build of the CLI that does not know about modes. Only an explicit --mode, --allow or --except changes what is in force.

If a second copy of the agent starts serving the same fingerprint, up and ls both say so. Both keep working, since the usual cause is a restart rather than a mistake, but you are told, because two agents on one fingerprint each declare their own policy and a run captures whichever was in force when it launched.

Using it in a run:

# exit from whichever of your machines is serving
benchwright run "screenshot the example.com landing page" --local-egress

# exit from one specific machine
benchwright run "…" --egress eg_7k2m9x4p3q

# queue the run until that machine comes online instead of failing now
benchwright run "…" --egress eg_7k2m9x4p3q --wait-for-egress

--egress implies --local-egress. Without --wait-for-egress a run whose machine is not serving fails straight away rather than hanging.

Checking where the traffic went. The run page has a Network routes card listing every host the run reached, one row each, with the exit it used: "your machine" or "datacenter", plus how many requests and when it was last seen. It opens with the summary you usually want, for example 3 via your machine, 7 direct, so the question "how much of this crossed my connection?" is answered without reading the table.

An empty card says no routing was recorded, rather than claiming everything went direct. Absence of a record is not a record of absence.

The card sits under an Egress card that names the policy the run actually used: the mode, the host list labelled by what it means in that mode, the exit address observed, and the machine's fingerprint. That is read from what the run captured when it launched, so it keeps telling the truth about that run even if you re-register the machine differently afterwards.

The reproduce block on the same page pins both halves, the machine and its policy, so pasting it reruns the run you are looking at rather than whatever your machine happens to be serving later:

# 1. on your machine, first:
benchwright egress up --allow 'example.com' --allow '*.example.com'

# 2. then:
benchwright run '…' --egress 'eg_7k2m9x4p3q' --limit-to run.tasks=1

The same record is available at GET /v1/runs/{id}/egress-routes (api.md).

billing

benchwright billing balance
benchwright billing topup  <usd>      [--flow hosted] [--open=false]   # min $5
benchwright billing ledger [--kind trace_debit|stripe_topup|… --limit N]   # alias: billing transactions
benchwright billing export [--type transactions|runs]      # streams CSV to stdout

topup 50 makes a Stripe Checkout session and opens it in your browser (--open defaults to true; pass --open=false to just print the URL). Top-ups must be between $5.00 and $50,000; the CLI checks the minimum client-side and the server enforces both bounds. export writes CSV directly to stdout for easy > file.csv redirection.

--flow payment_intent is not yet available. Only the hosted Checkout flow is wired up server-side; the CLI rejects any other --flow value. The client-secret flow (for embedding in your own UI) lands when the API exposes it.

hooks

Not yet available. The webhooks API isn't implemented server-side, so every hooks subcommand currently exits with "the webhooks API is not available yet". The commands are listed here for the shape they'll take once /v1/webhooks ships. Until then, poll run show/watch or gate on run's exit code.

benchwright hooks ls
benchwright hooks add    <url> [--events run.finished,run.failed --description …]
benchwright hooks test   <hook_id> [--event run.finished]
benchwright hooks rotate <hook_id>

Webhook deliveries are signed; rotate mints a new signing secret and returns it once. test fires a synthetic delivery so you can verify your endpoint's signature check before a real run.

operator {#operator}

Alias: operators, op. Message your operators and read their replies from the shell — same API surface as api.md §12a.

benchwright operator ls
benchwright operator msg <id|name> "text" [--wait 5m]
benchwright operator log <id|name> [--limit N] [--since i:N|epoch] [--follow]
benchwright operator create [name] [--wait 5m]
benchwright operator start <id|name>
benchwright operator shutdown <id|name> [-y]

• ls — your operators (id, name, status, placement, mode). Needs operators:read. Table on a TTY; YAML or JSON via --output.
• msg — queue one chat turn on a running operator. <id|name> resolves a unique case-insensitive name via ls (or pass the UUID). Needs operators:write. Exits non-zero on 409 when the operator is not running. --wait 5m blocks until the reply completes and then prints it, so a conversation is one command per turn instead of a log --follow loop. The wait is server-side, nothing is re-requested on a timer, and it needs operators:read on top of operators:write.
• log — recent turns + any still-queued messages. --limit (default 50) and --since match the API. --follow / -f polls with the index cursor until interrupted (Ctrl-C). Needs operators:read.
• create creates a new operator and boots it. The name is optional and one is picked for you if you leave it off. Names are unique per account. The operator starts unbriefed, so tell it what you want with msg. --wait 5m blocks until it is running instead of returning straight away. Needs operators:write. --help also lists --super, --harness and --placement; those configure super operators on non-production deploys and aren't part of the normal flow.
• start brings an operator back. Needs operators:write.
• shutdown releases the operator's machine. Needs operators:write.

When chat stops answering, use start. An operator can keep running while its chat stops responding: replies stop arriving and messages come back as errors, even though the operator itself is alive and its status still reads running. start repairs that in place, keeping the machine, its working directory and the open session:

benchwright operator start yara

It is safe to run against a healthy operator, and safe to run when you are not sure which case you are in. That is deliberate. From the outside a stopped operator and a mute one look the same, so one verb covers both: a stopped operator gets a fresh machine under the same id and name, and a mute one gets its chat restored without losing anything.

Reach for shutdown only when you actually want the machine gone. It keeps the operator (it stays in ls as stopped, and start brings it back with the same id), but the machine and anything left in its working directory are discarded. If chat has simply gone quiet, start is the one you want.

There is no rm. Nothing deletes an operator record today, and a command named rm that does not remove would be worse than not having one.

benchwright operator ls
benchwright operator msg yara "please tighten the landing hero copy"
benchwright operator msg yara "now make it shorter" --wait 5m   # send, then print the reply
benchwright operator log yara --limit 10
benchwright operator log yara --follow

Mint a PAT with both operators:read and operators:write for the full send-and-poll loop (token create --full includes them).

token

Alias: tokens. token revoke also answers to token rm.

benchwright token ls
benchwright token create --name <label> [--scope runs:read --scope runs:write …]
benchwright token create --name <label> --full          # grant the full non-admin scope set
benchwright token rotate <pat_id>
benchwright token revoke <pat_id>

Only token ls works over PAT auth, and that is deliberate. Creating, rotating and revoking tokens require a signed-in browser session at /manage (API Keys). Over a PAT they return 403 scope_insufficient ("requires a full-access browser session").

The reason is containment. If a token could mint tokens, then any leaked token could mint itself a wider one, and every narrow scope you set would be advisory rather than binding. Token management is the one operation that has to sit behind something a token cannot impersonate, so it stays with the session. This is not a gap waiting to be filled: expect it to keep working this way, and mint tokens in the web UI.

Default scopes when none are specified: runs:read runs:write artifacts:read secrets:read billing:read registry:read. --full grants every non-admin scope (adds secrets:write, host_policy:write, billing:write, registry:publish, operators:read, operators:write) — use it for an interactive workstation token that should drive every command; prefer a narrow --scope set for CI / shared tokens. Scopes are fixed at creation, so a 403 scope_insufficient (which names the exact required_scope) means minting a new token in the web UI and re-running login. registry:read is in the default set so the token can browse the catalog (bench ls); registry reads are gated on it for PATs (anon callers stay open). Catalog writes (bench create/subset/impl, and run publish) need registry:publish, which is not granted by default — add it when minting a token for authoring. Operator commands need operators:read / operators:write (not in the default set — add them or use --full). Token plaintext is shown exactly once at mint time; copy it immediately.

settings

benchwright settings ls
benchwright settings get <key>
benchwright settings set <key> <value>           # auto-typed: true/false/null/int/float/string
benchwright settings schema                      # the allowlist of valid keys

Per-user KV. set coerces obvious literals: true/false/null and integer/float strings come through with the right JSON type. Anything else is a string. Unknown keys are rejected — use settings schema to see what's allowed.

version

benchwright version

Prints benchwright <version> (<short-sha>). Both are baked in at build time via -ldflags.

update

benchwright update [--check] [--force] [--version 0.4.1]

Replace this binary with the latest published build. It downloads through the API host's /dl redirects, verifies the sha256 against the published checksums, and swaps the executable in place, keeping the old one alongside as <name>.old until the next update.

• --check reports what's available and changes nothing.
• --version 0.4.1 installs that exact release instead of the latest.
• --force reinstalls even when the versions already match.

completion

benchwright completion bash|zsh|fish|powershell

Print a shell autocompletion script to stdout; source it from your shell profile.

Common workflows

Smoke test a new model:

benchwright run "MMLU Abstract Algebra on $MODEL via OpenRouter" --limit-to run.cost=0.50
benchwright open @last

Replay a previous run against a new model (full parent subset):

benchwright run --replay-from <uuid> --model $MODEL@openrouter

(Replaying only the failed indices isn't available yet — see the run section.)

Gate a CI deploy on pass rate:

benchwright run --benchmark bm_… --subset sub_… --impl latest:official \
                --model $MODEL@openrouter --limit-to run.cost=2 --gate pass_rate=0.90
echo $?       # 0 if pass, 1 if gate failed, 3 if out of balance

(Pinned mode is a preview; the launcher doesn't resolve the pin yet. For a stable CI anchor today, use --replay-from <run_id> against a known-good parent run instead of the pin flags.)

Pipe a run list into jq:

benchwright run ls --status running --output ndjson | jq -r '.id'

Rotate a leaked PAT and update the local config:

Rotate the token in the web UI at /manage (API Keys); token rotate over PAT auth is blocked until session auth ships. Then:

benchwright login --set-default          # paste the new value

Tighten the egress allowlist before a sensitive run:

benchwright policy default block
benchwright policy allow github.com
benchwright policy allow pypi.org
benchwright policy allow huggingface.co
benchwright run "…" --metadata locked-egress=true
benchwright policy blocked @last         # see what got blocked

Configuration files

On Windows the equivalents are %APPDATA%\benchwright\… and %LOCALAPPDATA%\benchwright\…. Override with XDG_CONFIG_HOME / XDG_CACHE_HOME.

Environment variables