CLI & file reference
The author-facing controls, in one place.
On this page
Eval file contract
| File | Required | Used for |
|---|---|---|
| benchmark.ts | Yes · project | Models, judge, repetitions, concurrency |
| evals/<id>/prompt.md | Yes · eval | Verbatim candidate task |
| evals/<id>/judge.md | One or both judge files | A rubric with named criteria and scoring rules |
| evals/<id>/judge.ts | One or both judge files | A plain function receiving JudgeContext and returning JSON |
| evals/<id>/eval.ts | Optional | Workspace preparation and early stopping |
| evals/<id>/workspace/ | Optional | Candidate-visible starting files |
Commands
| openeval … | Action |
|---|---|
| image | Build the isolated candidate runtime |
| plan | Preview missing / changed work and cost estimates |
| run | Execute the current plan |
| view | Serve the bundled results viewer |
| retry <run> <eval-run> | Retry a candidate execution |
| rejudge <run> <eval-run> | Judge existing evidence again |
| snapshot <run> <name> | Freeze the active scores into a named snapshot |
| add-models <run> --model <ref> | Extend an idle aggregate with more models |
| merge-runs <target> <source> | Merge results into one aggregate |
Flags
| Flag | Applies to | Purpose |
|---|---|---|
| --benchmark <dir> | All benchmark commands | Benchmark directory; default current directory |
| --run <dir> | run / plan | Use a specific result |
| --new | run | Start a separate result |
| --model <ref> | run / plan / add-models | Add a model |
| --only-model <ref> | run / plan | Limit execution to a model; repeatable |
| --only-eval <id> | run / plan | Limit execution to an eval; repeatable |
| --only-repetition <n> | run / plan | Limit execution to a repetition; repeatable |
| --max-cost <usd> | run / plan | Scheduling budget for this invocation |
| --final-only | run | Disable monitoring for this invocation |
| --port <n> | view | Viewer port; default 4173 |
Benchmark settings
| Setting | Default | Meaning |
|---|---|---|
| name | Directory name | Benchmark name in viewer headings and exported images |
| suite | None | Label for one benchmark.ts among several that share a name, such as Frontier |
| models | Required | Unique provider/model references; optional #variant |
| judge.model | Required with judge.md | Model used by the Markdown judge |
| repetitions | 3 | Candidate executions per eval / model |
| concurrency | 10 | Shared worker budget |
| candidate.timeoutMs | 45 minutes | Agent session limit, at most 12 hours; eval.ts candidate.timeoutMs overrides it for one eval |
| candidate.maxCostUSD | None | Per-attempt model cost limit in USD; the session is interrupted when it is reached |
| candidate.network | Public internet | { allow: [hosts] } limits the agent session to HTTPS through an SNI-checked proxy for those hosts; requires candidate.websearch: false and disables webfetch. The agent can still read its own model credential, so this and maxCostUSD bound a cooperative agent |
| judge.timeoutMs | 10 minutes | Judge execution timeout |
| candidate.websearch / judge.websearch | exa | Search provider, or false |
| container.engine | docker | docker or podman |
| container.cpus / memoryMiB | 2 / 4096 | Per-container resources |