--- order: 4 subtitle: The benchwright binary. Launch runs, author benchmarks, manage secrets and billing from the shell. --- # Benchwright CLI `benchwright` is a single Go binary that wraps the public API. It covers most of what the dashboard does — launch and stream runs, author benchmarks, message operators and read their replies, manage provider secrets and host policy, top up your balance — plus a couple of shell-only conveniences (`@last`, `--gate`, idempotency key auto-derivation, table/ndjson output for piping). Tokens themselves are minted in the web UI (`/manage` → API Keys); webhook management ships once `/v1/webhooks` does. For the underlying HTTP surface, see [api.md](./api.md). The CLI is just an ergonomic shell over the same endpoints, with the same auth (PAT) and the same error codes. ## Install ```bash # macOS / Linux — install script curl -fsSL {{site}}/install.sh | sh # Pin a specific release in CI: curl -fsSL {{site}}/install.sh | BW_VERSION=v0.1.2 sh ``` The install script is at [`{{site}}/install.sh`]({{site}}/install.sh). It reads a handful of env vars: | Var | Effect | |---|---| | `BW_VERSION` | Pin a release, e.g. `v0.1.2`. Defaults to `latest`. | | `BW_INSTALL_DIR` | Where the binary lands. Default is the first writable of `$HOME/.local/bin`, `/usr/local/bin`. | | `BW_API_URL` | The API URL written into a freshly seeded config. Defaults to whichever instance served the script. | | `BW_NO_VERIFY` | Skip the sha256 check against the released `checksums.txt`. Not recommended. | | `BW_NO_CONFIG` | Skip config seeding entirely. | Unless `BW_NO_CONFIG=1`, the script also seeds `~/.config/benchwright/config.toml` (mode 0600) with `default_profile` and that profile's `api_url`, so `login` doesn't have to be told `--api-url`. It only writes the file when it is missing, and never overwrites a config you already have. Or grab a binary directly: - [`benchwright-linux-amd64`]({{site}}/dl/benchwright-linux-amd64) - [`benchwright-linux-arm64`]({{site}}/dl/benchwright-linux-arm64) - [`benchwright-darwin-amd64`]({{site}}/dl/benchwright-darwin-amd64) - [`benchwright-darwin-arm64`]({{site}}/dl/benchwright-darwin-arm64) - [`benchwright-windows-amd64.exe`]({{site}}/dl/benchwright-windows-amd64.exe) - [`benchwright-windows-arm64.exe`]({{site}}/dl/benchwright-windows-arm64.exe) Pin a release by inserting `/vX.Y.Z/` before the filename — e.g. [`{{site}}/dl/v0.10.121/benchwright-linux-amd64`]({{site}}/dl/v0.10.121/benchwright-linux-amd64). Verify: ```bash benchwright version ``` Once installed, upgrade in place with `benchwright update`. Pass `--check` to report what's available without touching anything, `--version 0.4.1` to pin a release, or `--force` to reinstall the version you already have. ## First run ```bash benchwright login # paste PAT (input hidden) benchwright doctor # config + auth + balance + secrets sanity benchwright run "Run MMLU on openai/gpt-4o-mini via OpenRouter" ``` `login` writes a config file (TOML) under `~/.config/benchwright/config.toml` (Linux/macOS) or `%APPDATA%\benchwright\config.toml` (Windows), respecting `$XDG_CONFIG_HOME` if set. The PAT is verified against the API before the file is written; bad tokens fail closed. `doctor` is the safe pre-flight before automation: it confirms your balance is over the $1 launch floor, that you have at least one secret, and that at least one secret has a host injection rule. If any of those fail, doctor exits non-zero with the exact fix command to run. ## Conventions ### Shorthands | Where | Shorthand | Expands to | |---|---|---| | Any `` arg | `@last` | The id of the most recent run launched or forked by this CLI (per-user, shared across shells). Written by `run` / `run launch` / `run fork`, cached at `~/.cache/benchwright/last_run`. | | `--model` | `model@provider` | `{"model":"…","provider":"…"}`. JSON also accepted: `--model '{"model":"…","extras":{"…"}}'`. | | `--impl` | `latest:official` / `latest:mine` | Server-side resolves to the latest official / your-latest impl on the chosen subset. | | `--rule` (secrets) | `host:kind[:target]` | E.g. `openrouter.ai:bearer:Authorization`. Default target for `bearer` is `Authorization`. | | `--indices` (replay) | `0,3,5-9` | Mix of singletons and inclusive ranges. | | `--metadata k=v` | repeatable | Pairs flatten into a single `{}` object. | ### Aliases `bench` = `benchmark`, `billing ledger` = `billing transactions`, `secrets rm` = `secrets delete`, `token` = `tokens`, `token revoke` = `token rm`, `operator` = `operators` = `op`, `egress ls` = `egress list`. **`run --agent` has been removed.** It was the old spelling of `run --driver` and now exits with an unknown-flag error. "Agent" was ambiguous once Operator came to mean the goal-level agent that launches runs, so the per-run LLM is the Driver everywhere. The error is deliberate: a flag that still parsed but no longer pinned anything would launch the run on a different model and bill you for it. ### Profiles ```bash benchwright login --profile-name staging --api-url https://staging.benchwright.ai benchwright --profile staging run ... ``` Resolution order at every command: 1. `--profile ` flag. 2. `BW_PROFILE` env var. 3. `default_profile` from the config file. 4. Hardcoded fallback `default`. `BW_PAT` and `BW_API_URL` env vars override the resolved profile's fields — handy for CI where you don't want a config file on disk. ### Output Global `--output` chooses the format. Defaults are smart: `table` on a TTY, `ndjson` when piped or redirected. Override with one of: - `table` — human-friendly aligned columns. Mostly used for `ls`-style commands. - `json` — single pretty-printed JSON object. - `ndjson` — one JSON object per line; ideal for `jq -c` pipelines. - `yaml` — YAML, round-tripping the JSON field names. Two caveats. `run ls` is the only command that implements all four formats. The other `ls`-style commands (`secrets ls`, `operator ls`, `egress ls`) branch on `table` and `yaml` only and fall through to a single pretty-printed JSON object, so `--output ndjson` and a bare pipe both give you one JSON object rather than a line per row. And some commands don't take the format at all. `run show`, `token ls` and `policy ls` always emit JSON regardless of `--output`. `run tasks` prints a `summary:` line plus a task table unless you pass `--full` or `--output json`, and `billing balance` prints a human text block unless you pass `--output json`; both ignore `yaml` and `ndjson`, and neither changes shape when you pipe it. ### Idempotency `run` / `run launch` / `run fork` and `billing topup` derive an idempotency key from `sha256(argv ‖ cwd ‖ $GIT_COMMIT)` and send it as `Idempotency-Key`, so re-running the same command in the same directory at the same git SHA returns the same run/topup rather than creating a duplicate. Override with `--idempotency-key=` (run/launch/fork), or disable with `--no-idempotency` (run/launch). Other mutating commands (secrets, bench, policy, token, settings) send no idempotency key. In practice: re-running an identical `benchwright run` command re-attaches to the already-launched run (the CLI prints "identical command already launched — showing that run") and self-heals by launching fresh if that run has vanished. ### Exit codes The CLI maps HTTP status to a stable exit code so shell pipelines can distinguish failure modes: | Exit | Meaning | |---|---| | 0 | Success. | | 1 | Generic error (bad flags, parse failure, etc). | | 2 | 401 / 403 — auth. | | 3 | 402 — out of balance. Topup and retry. | | 4 | 404. | | 5 | 429 — rate limited. Back off. | | 6 | 5xx — server error. | Pass `--debug` to print the response body that produced the error. Independently of HTTP status, a streaming `run` and `watch` exit non-zero when the run itself terminates with `status=error`, so CI can gate on the exit code without parsing output. ## Commands Running `benchwright` with no args, or `benchwright --help`, prints the full Cobra-style help. The summary below is the commands grouped by topic. ### `login` / `logout` ``` benchwright login [--profile-name N] [--api-url URL] [--set-default] benchwright logout [--profile-name N] ``` Save or remove a PAT under a profile. PATs are validated against the API before the file is written. The PAT can also be piped via `BW_PAT`, but `login` is the friendly entry point. ### `doctor` ``` benchwright doctor ``` Smoke-test the active profile: prints `profile`, `api_url`, `balance`, `month`, `secrets` count, `rules` count, and a **scope coverage** block naming which command groups the active token can actually run, based on its scopes. The groups it names are `launch + read runs`, `download artifacts`, `manage secrets`, `manage host policy`, `top up balance`, `browse registry`, `author benchmarks`, `list/read operators` and `message operators`. Exits non-zero with a fix command if balance is below $1, no secrets exist, or no secrets have host rules. Missing scopes are reported but don't fail the check (you may not need them); a `MISS` line points at the web UI (`/manage` → API Keys) — token minting is browser-session-only, so mint the wider token there and re-run `login` (see the `token` section). ### `run` The biggest command. Without subcommand, it launches and streams. With subcommands, it inspects, forks, halts, kills, publishes. ``` benchwright run [request] # launch via natural language benchwright run --benchmark bm_… --subset sub_… --impl latest:official --model m@p # pinned benchwright run --replay-from run_… --model m@p # replay (full parent subset) ``` > **Pinned mode is a preview.** The API accepts `--benchmark` / > `--subset` / `--impl`, but the launcher does not resolve the pin > yet. To anchor a run to a prior run's exact subset today, use > `--replay-from`. > **Replay subsetting (`--only-failed` / `--indices` / `--task-id`) > is not honored yet.** The API replays the *full* parent subset, so > the CLI refuses these flags (rather than silently running — and > billing — the whole subset). Replay without them re-runs everything; > scoped replay lands when the launcher reads `replay.{only_failed, > indices,task_ids}`. Common flags (all modes): - `--model model@provider` — the model. - `--driver ` — the run's Driver, the LLM that executes it (default: server's choice). `--agent` was the old name for this flag and has been removed; it now errors. See [Driver vs SUT vs Operator](./concepts.md#8-driver-vs-sut-vs-operator). - `--harness inspect|lmeval|lighteval|harbor|benchwright` — eval harness (default `benchwright`; aliases like `lm-eval`, `scratch`, `native` are canonicalized). - `--harness-task ` — harness task id, e.g. `inspect_evals/mmlu_0_shot`. When omitted, the server resolves the task from the request text. - `--limit-to .=` (repeatable) — every other ceiling, in one spelling. See [Limits](#limits) below. - `--metadata k=v` (repeatable) — arbitrary tags. - `--public` — auto-publish the run to public when it finishes successfully. Runs are private by default and nothing else flips that for you, so this is the flag that makes a score visible to everyone. - `--proxy-linger-seconds N` — keep the LiteLLM proxy sandbox alive this many seconds after the run finishes; `0` tears it down immediately. - `--idempotency-key K` / `--no-idempotency` — see above. - `--no-wait` — print the run id and exit; don't stream. - `--verbose` — stream every event type, not just phase + task results. - `--gate pass_rate=0.90` — exit non-zero if the gate fails. Currently `pass_rate` is the only key. The gate reads the finished run's tasks, so it can't be combined with `--no-wait`; the CLI rejects the pair up front rather than after you've paid for the run. ### Limits Every ceiling is spelled `--limit-to .=`, repeatable. ```bash benchwright run "..." --limit-to run.cost=5 \ --limit-to task.attempts=1 \ --limit-to attempt.duration=600 ``` **Money is in whole dollars and time is in seconds.** The unit belongs to the quantity, never to the name — there is no `_cents` or `_ms` suffix to get backwards. | Level | Quantities | What it stops | |---|---|---| | `run` | `cost`, `model_cost`, `duration`, `tokens`, `tasks` | the whole run | | `task` | `cost`, `tokens`, `duration`, `attempts` | that task only | | `attempt` | `duration` | that attempt | | `phase` | `calls`, `duration` | the whole run | - **`run.cost` vs `run.model_cost`** — `cost` is everything (sandbox + driver + model); `model_cost` is only what your own provider key pays for. On a BYOK connection that is the one that matters, and it can also be declared on the connection itself so it applies to every run without a flag. - **`run.tokens`** is the ceiling to reach for when a connection declares no rate: spend is then unknowable and a money ceiling can never fire, but tokens are still counted. - **Task ceilings fail that task and never retry it**, and the run carries on. A task over its own budget is a failed task, not a failed run. They bound the whole retry chain, so `task.cost=0.50` means fifty cents including retries — a per-attempt money cap would still allow `task.attempts` x that. - **`attempt.duration` multiplies.** Worst case per task is `attempt.duration x task.attempts`, so `600` alone is 40 minutes on the default of 4, not 10. A hard 10-minute cap is `--limit-to task.attempts=1 --limit-to attempt.duration=600`. - **`phase` may name one phase**: `--limit-to phase:atomize.calls=250` overrides `--limit-to phase.calls=100` for that phase alone. Useful for giving a build-heavy phase *more* room. Phases: `find`, `fetch`, `analyze`, `setup`, `attach`, `atomize`, `materialize`, `report`, `fork`. A run stopped by a ceiling halts with `halt_reason=limit_exceeded:.`, so the reason says which one fired. **A limit the platform cannot enforce is refused**, not accepted and ignored — `attempt.cost` is an error today, because spend is attributed per task and an attempt has no cost of its own. A ceiling that silently does nothing is worse than no ceiling. A ceiling is where teardown *starts*, not a figure the run is guaranteed to stay under: the check runs every 2s and teardown keeps accruing after it fires. The older flags — `--max-cost`, `--max-duration-ms`, `--max-tasks`, `--task-attempts`, `--task-timeout-sec` and `--limit` — have been **removed**. There is one spelling now. A script still passing them fails with an unknown-flag error rather than running uncapped, which is the failure you want: the units were in those names (`--max-cost` took *cents*), so a silent fallback would have been a 100x ceiling. The raw API still accepts the old body keys (`constraints.max_cost_cents`, `metadata.task_attempts`, …) and converts them, so an un-upgraded client keeps its ceilings rather than losing them quietly. Subcommands: ``` benchwright run launch [request] … # explicit form of bare `run` benchwright run show # full run JSON benchwright run ls [--status … --benchmark … --visibility … --limit N] benchwright run tasks [--passed true|false --limit N --full] benchwright run fork -m "what changed" [--from-phase atomize] benchwright run halt [--reason …] # stop it; the run stays readable benchwright run kill [-y] [--reason …] # stop it now (eager teardown) benchwright run delete [-y] # DESTROY it — irreversible benchwright run publish [--public=false] ``` **Publishing a run publishes its benchmark.** Benchmark visibility is derived from having at least one public run rather than stored on the benchmark, so the two move together and there is no flag to separate them. If you want the benchmark to stay private, keep the run private. **The list subcommand is `ls`, and only `ls`.** `run` itself takes an optional request argument, so `benchwright run list` is not an error and not a listing: it launches a run whose request is the word "list", and bills you for it. Same for any other word that isn't one of the subcommands above. **Stopping a run never destroys it.** `halt` and `kill` both tear the sandboxes down and finalize the run with a kill reason — `run show` and `run tasks` keep working afterwards, so a run you stopped is still a run you can investigate. `--reason "…"` is recorded on the run and is what `run show` and `watch` report on its `halted:` line afterwards; without one, the record says only who stopped it. Only `run delete` removes anything, and it removes everything: the run record, its event stream, its per-task results and its registry summary. That cannot be undone, so its prompt asks you to type the run id rather than `y`. **`delete` removes one run, not a fork chain.** Runs forked from the one you delete are kept. Whatever they were inheriting from it — the phase outputs and artifacts a fork reads from its ancestors instead of re-running — is copied onto them first, and they re-point at the deleted run's own parent, so a fork keeps working and keeps showing the same results. The prompt tells you how many will be kept before you confirm, and the response lists them under `forks_adopted`. Each one then reports `forked_from_deleted` in `run show`, because a fork whose parent is gone should say so rather than quietly present an unbroken lineage. `publish` flips the run to `visibility=public` by default (`--public` defaults to true); pass `--public=false` to record the publish without the visibility flip. Note the run object returned by `show` / `ls` carries the resolved model pin as `{identifier, source, provenance}` (e.g. identifier `qwen/qwen3-32b`); the `{model, provider}` shape is launch-body only. `halt` flips a database flag the running pipeline polls every ~2s. `kill` is the immediate one — it tears down the agent JVM and any spawned sandboxes synchronously, asks for confirmation on a TTY, and prints the teardown summary. Use `halt` first. `run` (and `run launch`) accept `--harness …` (default `benchwright`) and `--harness-task` to pin the harness task id; compute backend selection is account-level, not a CLI flag. ### `watch` ``` benchwright watch [run_id|@last] [--compact] [--timeout 20m] [--verbose] ``` Follow a run until it finishes, then print its outcome: final status, the halt reason if it was halted, the score, the cost and the dashboard link. Use it after launching with `--no-wait`, or to re-attach from another shell. The run id is optional: with no argument `watch` follows `@last`, which is the run you just launched. This is the supported way to monitor a run — you shouldn't need to write a polling loop. `watch` blocks until the run genuinely reaches a terminal state, reconnecting by itself (resuming from the last event it saw) if the connection drops, so a proxy timeout or a brief network blip doesn't end the wait early. By default it renders a live event feed as the run progresses — phase boundaries, task scores, errors; `--verbose` shows every event type. `--compact` drops the feed and prints only the final outcome block. That's the form to use in scripts and automation: one command, one answer, nothing to parse mid-stream. `--timeout` (`-t`) bounds the wait, e.g. `-t 20m`. If the run is still going when the clock runs out, `watch` reports where it got to and exits non-zero — the wait didn't finish, so a `watch && …` chain must not proceed. Watch it again to keep waiting. Without `--timeout` it waits indefinitely. Exit status: `0` when the run finishes successfully, non-zero when it ends in `status=error`, when it was halted (the halt reason is in the error message), or when `--timeout` expires with the run still in flight. ### `open` ``` benchwright open [--print] ``` Open the run's dashboard URL in your default browser. `--print` writes the URL to stdout instead. ### `bench` ``` benchwright bench ls benchwright bench range # show existing runs that covered an ad-hoc task range, e.g. 0-5 or 0,2,4 (no stored subset needed) benchwright bench create [--description … --category … --tag t] benchwright bench subset create --slug … --file tasks.jsonl [--grader-file grader.json --render-kind product_sandbox --name … --description … --rolling] benchwright bench subset append --file new_tasks.jsonl benchwright bench subset show # the subset, grader recipe included benchwright bench recipe show benchwright bench recipe set [--set k=v --json-set k= --unset k --file recipe.json --note …] benchwright bench samples set [--upsert-file fix.jsonl --delete task_id --file tasks.jsonl --note …] benchwright bench impl ls benchwright bench impl create [--file task_impl.py --language python --label … --benchmark … --description …] benchwright bench impl from-run --from-run --benchmark … --subset … --label … # not yet available ``` `subset create` / `append` read JSONL — one task object per line with fields `{task_id, input, expected, …}`. `--grader-file` takes a JSON file holding the subset's grader recipe; without it the subset is catalog-only and cannot run as a product benchmark, so pass it unless you only want the tasks catalogued. `subset show` reads a subset back with its grader recipe, which is how you tell which of the two you ended up with. On `impl create`, `--file` defaults to `task_impl.py` and `--language` defaults to `python`, so both are optional. The recipe file may be written either flat (the runtime keys — `base_image`, `boot`, `run`, `setup`, `capture`, `display`, `judge_model`, … at the top level, the way `recipe show` prints them) or already wrapped as `{"type": …, "config": {…}}`. Both work: the flat form is wrapped for you, and `render_kind` is inferred as `product_sandbox` when the recipe carries sandbox keys, so `--render-kind` is only needed to override that inference. `grader_type` is accepted as a spelling of `type`, because that is the name `recipe show` prints. A file with no recipe keys at all is rejected outright, rather than published as a subset that reports `runnable: true` and then fails partway through a paid run. There is no `bench publish`. A benchmark has no visibility of its own: it is public exactly when one of its runs is public. To put a benchmark on a public leaderboard, publish the run you want listed with `benchwright run publish `; the registry derives the benchmark's visibility from its runs. #### Fixing a built benchmark: `recipe` and `samples` A benchwright-BUILT (product) benchmark carries a **recipe** — the sandbox scripts that boot the product, drive it, capture its output and judge the result, plus `trials` and the timeouts. `recipe` and `samples` edit that authored definition. Both take a subset id, a benchmark id, or a benchmark **name** (the last two resolve to that benchmark's live subset), and both require you to **own** the benchmark. A subset id copied from the registry works too: catalog ids and authored ids are minted by different fingerprints, and the catalog one is mapped across for you. A name that matches two benchmarks is refused with both ids rather than guessed at. ```bash benchwright bench recipe show my-bench # read it back first benchwright bench recipe set my-bench --set run_timeout=900 --set trials=3 benchwright bench recipe set my-bench --set 'boot=mint --match ^clip-[0-9]+$' \ --note 'mint missed multi-digit clips' benchwright bench samples set my-bench --upsert-file fix.jsonl \ --note 'q2 pointed at the wrong frame' ``` **The recipe IS the implementation.** A product run records `bench_task_product.py` plus its recipe header as that run's `task_impl.py`, so `impl_id` is a hash of the two — edit the recipe and the next run registers a **new impl**, with the previous ones kept as history. That is why a subset carries one live recipe rather than a menu of them: to compare two ways of running the same tasks, publish them as separate benchmarks (or separate subsets with their own task sets), not as two recipes on one subset. `recipe set` is a merge: keys you pass replace theirs, `--unset k` deletes one, and anything you don't mention is left alone. Values type themselves the way their JSON counterpart reads (`900` is a number, `true` a bool, everything else a string); `--json-set k=` is the escape hatch for objects, arrays, or a string that looks like a number. `samples set` takes `--upsert-file` (edit or add, matched on `task_id`) with `--delete` (remove by `task_id`), or `--file` to replace the whole set. `--file` is a whole-set replacement, so it can't be combined with the surgical flags. **Both edits are versioned, not silent.** A recipe change is content-addressed into `task_impl.py`, so the next run records a new impl id and runs from before the edit keep pointing at the scorer that actually graded them. A sample change forks a new content-addressed subset (the response names both ids) and retires the old one, because a subset id is a hash of its tasks. An edit that changes nothing answers `changed: false` rather than forking. > **`impl from-run` is not yet available.** The API only accepts a > raw-code impl, so `from-run` exits with a "not available yet" > message. To register the scorer the pipeline atomized for a run, > grab its `task_impl.py` from the run's dashboard (Atomize artifact) > and submit it as raw code with > `benchwright bench impl create --file task_impl.py`. ### `secrets` ``` benchwright secrets ls benchwright secrets add [value] --rule host:kind[:target] (repeatable) [--description …] benchwright secrets rotate [value] benchwright secrets rm [-y] ``` The plaintext secret can be passed as the second positional, via `--value`, or piped on stdin. After creation values are never returned. `rotate` only changes the value — id, name, scopes, host rules carry over. In-flight runs keep using the old value (rotation is non-disruptive). `add`'s `--rule` flag is repeatable; the host pattern supports `*.foo.com` suffix wildcards. ### `policy` ``` benchwright policy ls benchwright policy allow # e.g. *.openrouter.ai benchwright policy block benchwright policy default # mode for unmatched hosts benchwright policy starter-pack # github / pypi / huggingface / … benchwright policy blocked [run_id] # hosts the proxy denied ``` The proxy enforces this list at the mitmproxy layer in the proxy sandbox. `default block` + `policy allow github.com pypi.org …` gives you a tight outbound allowlist; `default allow` + a small blocklist is the looser default. ### `egress` {#egress} Runs normally reach the internet from the datacenter. Some sites will not serve a datacenter address, and some pages only look right from a real home connection. `egress` lets a run exit the network from your own machine instead. ``` benchwright egress up [--mode hosts|all|except] [--allow HOST]… [--except HOST]… [--label NAME] [--max-bytes 500MB] [--max-duration 30m] [--max-runs N] [--run RUN_ID] [--save] benchwright egress ls benchwright egress rm ``` **`up`** serves runs from this machine until you press Ctrl-C. It prints a fingerprint like `eg_7k2m9x4p3q`, which is how a run asks for this machine by name. The fingerprint is safe to paste into a run command or share in a recipe. Three modes decide what actually crosses your connection: | Mode | Rides your connection | Takes the normal path | |---|---|---| | `hosts` (default) | only the hosts you name with `--allow` | everything else | | `except` | everything | only the hosts you name with `--except` | | `all` | everything | nothing | `hosts` routes nothing until you say so, on purpose: ```bash benchwright egress up --allow example.com --allow '*.example.com' ``` `*.example.com` covers the subdomains and the apex. Passing `--except` on its own selects `except` mode, so the common case is one flag rather than two. **What the host list means.** It chooses which hosts leave through your machine. It does not restrict what the run may reach: a host you did not name still works, it just takes the normal path. To limit what a run can reach at all, use [`policy`](#policy). That is a separate control and it applies whether or not you are serving egress. **The same host list means opposite things in different modes.** This is worth seeing rather than reading. Four runs of one benchmark, same model, same machine, changing only the declared policy: | Run | Policy | Reached your machine | Took the normal path | Score | |---|---|---|---|---| | control | no local egress | nothing | everything | 0.0 | | A | `hosts`, `example.com` + `*.example.com` | 2 example.com hosts | the other 7 | 1.0 | | B | `except`, `image.tmdb.org` | 8 hosts, example.com included | `image.tmdb.org`, only that | 1.0 | | C | `all` | all 9 | nothing | 1.0 | Runs A and B are the pair to look at. `image.tmdb.org` is named in B and not in A, and it is the one host that does *not* cross your connection in B. The seven hosts that never reached your machine in A all reached it in B, without either list changing meaning by accident: `hosts` says carry these, `except` says carry everything but these. The control run is why the rest means anything. Without local egress the site served a block page and the run scored 0.0, so a passing run is evidence the tunnel did something rather than evidence it merely existed. One thing you will notice comparing runs: the exit address can differ between sessions on the same machine. That is normal and not a fault. What a declared policy controls is which hosts cross your connection, not which address they emerge from. Budgets end a session on your terms. All three are unlimited unless set: ```bash benchwright egress up --allow example.com \ --max-bytes 500MB --max-duration 30m --max-runs 3 ``` `--run ` serves exactly one run and ignores every other. `--save` writes the current settings to `~/.config/benchwright/egress.toml` so later sessions need no flags, and any flag you pass still wins over the file for that session. **`ls`** lists the machines you have registered, and always shows the mode alongside the hosts, because a bare host list cannot tell you which of two opposite policies is in force. `rm ` forgets one. ``` FINGERPRINT LABEL STATE MODE HOSTS eg_7k2m9x4p3q laptop online hosts only: example.com, *.example.com ``` **Your declared policy is not changed by anything except you declaring it.** `up` both registers the machine and keeps it alive, so it repeats while it serves. Those keep-alives only prove the machine is still there. They cannot alter the mode or the host list, including from an older build of the CLI that does not know about modes. Only an explicit `--mode`, `--allow` or `--except` changes what is in force. If a second copy of the agent starts serving the same fingerprint, `up` and `ls` both say so. Both keep working, since the usual cause is a restart rather than a mistake, but you are told, because two agents on one fingerprint each declare their own policy and a run captures whichever was in force when it launched. **Using it in a run:** ```bash # exit from whichever of your machines is serving benchwright run "screenshot the example.com landing page" --local-egress # exit from one specific machine benchwright run "…" --egress eg_7k2m9x4p3q # queue the run until that machine comes online instead of failing now benchwright run "…" --egress eg_7k2m9x4p3q --wait-for-egress ``` `--egress` implies `--local-egress`. Without `--wait-for-egress` a run whose machine is not serving fails straight away rather than hanging. **Checking where the traffic went.** The run page has a Network routes card listing every host the run reached, one row each, with the exit it used: "your machine" or "datacenter", plus how many requests and when it was last seen. It opens with the summary you usually want, for example `3 via your machine, 7 direct`, so the question "how much of this crossed my connection?" is answered without reading the table. An empty card says no routing was recorded, rather than claiming everything went direct. Absence of a record is not a record of absence. The card sits under an Egress card that names the policy the run actually used: the mode, the host list labelled by what it means in that mode, the exit address observed, and the machine's fingerprint. That is read from what the run captured when it launched, so it keeps telling the truth about that run even if you re-register the machine differently afterwards. The reproduce block on the same page pins both halves, the machine and its policy, so pasting it reruns the run you are looking at rather than whatever your machine happens to be serving later: ```bash # 1. on your machine, first: benchwright egress up --allow 'example.com' --allow '*.example.com' # 2. then: benchwright run '…' --egress 'eg_7k2m9x4p3q' --limit-to run.tasks=1 ``` The same record is available at `GET /v1/runs/{id}/egress-routes` ([api.md](./api.md)). ### `billing` ``` benchwright billing balance benchwright billing topup [--flow hosted] [--open=false] # min $5 benchwright billing ledger [--kind trace_debit|stripe_topup|… --limit N] # alias: billing transactions benchwright billing export [--type transactions|runs] # streams CSV to stdout ``` `topup 50` makes a Stripe Checkout session and opens it in your browser (`--open` defaults to true; pass `--open=false` to just print the URL). Top-ups must be between $5.00 and $50,000; the CLI checks the minimum client-side and the server enforces both bounds. `export` writes CSV directly to stdout for easy `> file.csv` redirection. > **`--flow payment_intent` is not yet available.** Only the hosted > Checkout flow is wired up server-side; the CLI rejects any other > `--flow` value. The client-secret flow (for embedding in your own > UI) lands when the API exposes it. ### `hooks` > **Not yet available.** The webhooks API isn't implemented > server-side, so every `hooks` subcommand currently exits with > "the webhooks API is not available yet". The commands are listed > here for the shape they'll take once `/v1/webhooks` ships. Until > then, poll `run show`/`watch` or gate on `run`'s exit code. ``` benchwright hooks ls benchwright hooks add [--events run.finished,run.failed --description …] benchwright hooks test [--event run.finished] benchwright hooks rotate ``` Webhook deliveries are signed; `rotate` mints a new signing secret and returns it once. `test` fires a synthetic delivery so you can verify your endpoint's signature check before a real run. ### `operator` {#operator} Alias: `operators`, `op`. Message your operators and read their replies from the shell — same API surface as [api.md §12a](./api.md). ``` benchwright operator ls benchwright operator msg "text" [--wait 5m] benchwright operator log [--limit N] [--since i:N|epoch] [--follow] benchwright operator create [name] [--wait 5m] benchwright operator start benchwright operator shutdown [-y] ``` - **`ls`** — your operators (`id`, `name`, `status`, `placement`, `mode`). Needs `operators:read`. Table on a TTY; YAML or JSON via `--output`. - **`msg`** — queue one chat turn on a running operator. `` resolves a unique case-insensitive name via `ls` (or pass the UUID). Needs `operators:write`. Exits non-zero on **409** when the operator is not running. `--wait 5m` blocks until the reply completes and then prints it, so a conversation is one command per turn instead of a `log --follow` loop. The wait is server-side, nothing is re-requested on a timer, and it needs `operators:read` on top of `operators:write`. - **`log`** — recent turns + any still-queued messages. `--limit` (default 50) and `--since` match the API. `--follow` / `-f` polls with the index cursor until interrupted (Ctrl-C). Needs `operators:read`. - **`create`** creates a new operator and boots it. The name is optional and one is picked for you if you leave it off. Names are unique per account. The operator starts unbriefed, so tell it what you want with `msg`. `--wait 5m` blocks until it is running instead of returning straight away. Needs `operators:write`. `--help` also lists `--super`, `--harness` and `--placement`; those configure super operators on non-production deploys and aren't part of the normal flow. - **`start`** brings an operator back. Needs `operators:write`. - **`shutdown`** releases the operator's machine. Needs `operators:write`. **When chat stops answering, use `start`.** An operator can keep running while its chat stops responding: replies stop arriving and messages come back as errors, even though the operator itself is alive and its status still reads running. `start` repairs that in place, keeping the machine, its working directory and the open session: ```bash benchwright operator start yara ``` It is safe to run against a healthy operator, and safe to run when you are not sure which case you are in. That is deliberate. From the outside a stopped operator and a mute one look the same, so one verb covers both: a stopped operator gets a fresh machine under the same id and name, and a mute one gets its chat restored without losing anything. Reach for `shutdown` only when you actually want the machine gone. It keeps the operator (it stays in `ls` as stopped, and `start` brings it back with the same id), but the machine and anything left in its working directory are discarded. If chat has simply gone quiet, `start` is the one you want. There is no `rm`. Nothing deletes an operator record today, and a command named `rm` that does not remove would be worse than not having one. ```bash benchwright operator ls benchwright operator msg yara "please tighten the landing hero copy" benchwright operator msg yara "now make it shorter" --wait 5m # send, then print the reply benchwright operator log yara --limit 10 benchwright operator log yara --follow ``` Mint a PAT with both `operators:read` and `operators:write` for the full send-and-poll loop (`token create --full` includes them). ### `token` Alias: `tokens`. `token revoke` also answers to `token rm`. ``` benchwright token ls benchwright token create --name