Run your benchmark
Plan a small batch, control cost, and keep one aggregate.
On this page
Start with a scoped plan
terminal
bunx --bun @hona/openeval plan \
--only-eval ask-dialect \
--only-model provider/candidate-model \
--only-repetition 1
bunx --bun @hona/openeval run \
--only-eval ask-dialect \
--only-repetition 1
| Scope | What stays in the result |
|---|---|
| --only-eval | Other evals and their active scores |
| --only-model | Other models and their active scores |
| --only-repetition | The configured repetition count |
| No scope flags | All missing or changed work |
Repeat a scope flag to select multiple values. Scope controls execution, not the benchmark's denominator.
Reuse evidence deliberately
| What changed? | Next run |
|---|---|
| Nothing | Reuse the selected candidate and judgment |
| Prompt, workspace, or candidate runtime | Collect new candidate evidence for affected inputs |
| Rubric or shared judge instructions | Rejudge the retained recording |
| A new model | Add its eval / repetition slots |
| The runner stopped mid-run | Close its open sessions as interrupted and collect them again |
| --new | Create a separate benchmark result |
Bound candidate and judge work
terminal
bunx --bun @hona/openeval run --max-cost 5
bunx --bun @hona/openeval run --final-only
| Control | Meaning |
|---|---|
| concurrency | Shared candidate / judge worker limit; default 10 |
| --max-cost | Admission budget using reported spend and reservations |
| candidate.timeoutMs | Agent session limit: 45 minutes by default, at most 12 hours. Set it in benchmark.ts, or in eval.ts for one eval |
| candidate.maxCostUSD | Stops an attempt once its recorded model cost reaches the limit; the run is recorded as cost limited and judged like a timeout. One in-flight step can overshoot |
| --final-only | Judge after the candidate finishes |
A scheduling budget stops admitting new work. Active sessions finish normally, so final billing can exceed the admission budget. Work without a cost estimate can be deferred.
eval.ts · opt-in early stopping
import type { Eval } from "@hona/openeval";
export default {
earlyStop: { minIntervalMs: 45_000, maxChecks: 12, maxCostUSD: 1 },
} satisfies Eval;
Early stopping requires every criterion score to be irreversible and non-null. Monitoring cannot steer the candidate. If monitoring pauses or exhausts its budget, final grading still follows execution. If you turn early stopping off, the next run collects new sessions only where it stopped one. Sessions that finished normally stay reusable.