CLI & file reference
The author-facing controls, in one place.
On this page
Eval file contract
| File | Required | Used for |
|---|---|---|
| benchmark.ts | Yes · project | Models, judge, repetitions, concurrency |
| evals/<id>/prompt.md | Yes · eval | Verbatim candidate task |
| evals/<id>/judge.md | Yes · eval | Declared metrics and grading criteria |
| evals/<id>/eval.ts | Optional | Workspace preparation and early stopping |
| evals/<id>/workspace/ | Optional | Candidate-visible starting files |
Commands
| openeval … | Action |
|---|---|
| image | Build the isolated candidate runtime |
| plan | Preview missing / changed work and cost estimates |
| run | Execute the current plan |
| view | Serve the bundled results viewer |
| retry <run> <eval-run> | Retry a candidate execution |
| rejudge <run> <eval-run> | Judge existing evidence again |
| snapshot <run> <name> | Freeze the active scores into a named snapshot |
| add-models <run> --model <ref> | Extend an idle aggregate with more models |
| merge-runs <target> <source> | Merge results into one aggregate |
Flags
| Flag | Applies to | Purpose |
|---|---|---|
| --benchmark <dir> | All benchmark commands | Benchmark directory; default current directory |
| --run <dir> | run / plan | Use a specific result |
| --new | run | Start a separate result |
| --model <ref> | run / plan / add-models | Add a model |
| --only-model <ref> | run / plan | Limit execution to a model; repeatable |
| --only-eval <id> | run / plan | Limit execution to an eval; repeatable |
| --only-repetition <n> | run / plan | Limit execution to a repetition; repeatable |
| --max-cost <usd> | run / plan | Scheduling budget for this invocation |
| --final-only | run | Disable monitoring for this invocation |
| --port <n> | view | Viewer port; default 4173 |
Benchmark settings
| Setting | Default | Meaning |
|---|---|---|
| models | Required | Unique provider/model references; optional #variant |
| judge.model | Required | Model that grades the recording |
| repetitions | 3 | Candidate executions per eval / model |
| concurrency | 10 | Shared worker budget |
| candidate.timeoutMs | 45 minutes | Candidate limit; cannot exceed 45 minutes |
| judge.timeoutMs | 10 minutes | Judge execution timeout |
| candidate.websearch / judge.websearch | exa | Search provider, or false |
| container.engine | docker | docker or podman |
| container.cpus / memoryMiB | 2 / 4096 | Per-container resources |