An operator is not a wrapper around an API. It's an autonomous agent that writes code and runs it, pulling a dataset, authoring a scorer, fanning hundreds of tasks across a shared cluster, on your behalf, and often against your paid keys. That is enormous leverage sitting on top of an obvious hazard, and turning the hazard into something you never have to think about is its own discipline. Call it containment. It has three boundaries: the operator can't reach your keys, can't run away with the cluster, and can't leak what you didn't choose to share.

The trust boundary: keys never touch the code

Running LLM-written scorer code against your real API keys should make you nervous. It makes us nervous, so the sandbox that runs scorers never sees them. Your keys live in a server-side vault, and a separate proxy sandbox holds them and stamps the real credential onto outbound requests per host, per request. The scorer's own sandbox holds only a fake key and exactly one route, to the proxy wall. A host your policy allows gets the real key; a host it blocks gets no route, no credential, and a log line (hostname only) you can review. Run once, see what was reached, decide what to allow, rerun.

Isolation is per run and per kernel. The proxy lives in a different sandbox from the scorer, so even though the agent has full sudo inside its own work sandbox, it can't read the proxy's memory or the host-policy config that governs it: those sit on another machine. The full threat model, the sandbox boundary, and its known gaps are documented at docs/security.

The resource boundary: no run stampedes the cluster

A single run can fan out hundreds of task sandboxes. Ten runs doing that at once would trample the cluster, and each run lives in its own process, blind to the others. So every spawn passes through one shared lease queue. It enforces a cluster-wide cap, reserves floors of capacity for free and background work so a paying whale can't starve everyone, and grants within a priority class round-robin: the user who has waited longest goes next. No run has to know about the others. The queue is the referee they all share.

Inside a run, tasks don't go out in lock-step batches. A streaming pump keeps the pipe full, and a controller tunes the launch rate on its own: it eases up a step at a time while tasks are succeeding, and cuts the rate in half the moment the model starts returning rate-limit errors. The agent doesn't babysit throughput; it watches for the patterns the controller can't see: everything failing at once, a suspicious score distribution, cost creeping up per task.

A run still breaks a hundred ways: rate limits, dead sandboxes, a grader that chokes on one weird sample. The pipeline catches these mid-flight and requeues, and what comes back is success-only, never a partial number. Every task lands in one of three buckets: right, wrong (the scorer ran, the model missed), or error (the pipeline itself couldn't produce a score). Errors stay out of the pass-rate denominator, so infrastructure noise never masquerades as model quality. When the number disappoints, the failure pattern names the lever:

Fork and replay are first-class operations, not "run it again and hope." A replay reuses the pinned benchmark and scorer, holding everything constant except the one thing you changed.

The data boundary: a public commons, private by choice

The registry is a public good on purpose, but public by default can't mean public by accident. A run marked private stays private: any surface that pools runs across users, the landing feed, trending, anything served with elevated privileges, filters to public visibility, so one account's private benchmark can never surface in another's view. The commons grows from what people choose to share, and nothing else.

There's a quieter guarantee here too. Because operator-written scorer code is untrusted, the measurement definition is pinned separately from it: the fingerprint is a hash over the dataset, the scorer's source, the config, and the seed. A scorer that's been swapped or has silently drifted is detectable, because the hash moves. Containing the code and keeping tamper-evidence on what it measures are two different locks, and you want both.

The economics that make it repeatable

Containment is also what makes the price hold, and the cost that matters is calibration. Standing up a brand-new benchmark for the first time is the expensive part: the operator breaks the dataset into tasks, writes the scorer and the harness around it, then re-runs samples and revises that setup over and over until the scoring provably matches what the benchmark intends. That trial-and-error can burn through millions, sometimes billions, of tokens before the setup is trustworthy, and on a frontier model that is real money.

It's paid once, in the open. The moment a benchmark has been calibrated by public runs, its settled setup belongs to everyone: anyone can replay it on the model they care about and skip calibration entirely, just the tasks and the scoring, for a tiny fraction of the first-time cost. Calibrating a benchmark from scratch versus replaying a calibrated one can be the difference between billions of tokens and a routine run.

And calibration keeps improving. Because runs are public, several people can be bringing up the same unsupported benchmark at once, and every result sharpens the shared calibration, which samples are flaky, which are stable, what a correct answer actually looks like, and what to expect from each sample across different models. Later runs inherit all of it. That's why a gate can run on every commit instead of once a quarter: the costly work was done once, in public, and everyone gets to stand on it.

Let it run

The same autonomy that makes an operator a workforce, and the registry a commons that compounds, is only comfortable because it's boxed on all three sides: it can't reach your keys, it can't run away with the cluster, and it can't leak what you didn't choose to share. Contain those, and there's nothing left to babysit: you can just let it run.