Run your benchmark
Plan a small batch, control cost, and keep one aggregate.
On this page
Start with a scoped plan
terminal
bunx --bun @hona/openeval plan \
--only-eval ask-dialect \
--only-model provider/candidate-model \
--only-repetition 1
bunx --bun @hona/openeval run \
--only-eval ask-dialect \
--only-repetition 1
| Scope | What stays in the result |
|---|---|
| --only-eval | Other evals and their active scores |
| --only-model | Other models and their active scores |
| --only-repetition | The configured repetition count |
| No scope flags | All missing or changed work |
Repeat a scope flag to select multiple values. Scope controls execution, not the benchmark's denominator.
Reuse evidence deliberately
| What changed? | Next run |
|---|---|
| Nothing | Reuse the selected candidate and judgment |
| Prompt, workspace, or candidate runtime | Collect new candidate evidence for affected inputs |
| Rubric or shared judge instructions | Rejudge the retained recording |
| A new model | Add its eval / repetition slots |
| --new | Create a separate benchmark result |
Bound candidate and judge work
terminal
bunx --bun @hona/openeval run --max-cost 5
bunx --bun @hona/openeval run --final-only
| Control | Meaning |
|---|---|
| concurrency | Shared candidate / judge worker limit; default 10 |
| --max-cost | Admission budget using reported spend and reservations |
| Candidate timeout | At most 45 minutes per candidate |
| --final-only | Judge after the candidate finishes |
A scheduling budget stops admitting new work. Active sessions finish normally, so final billing can exceed the admission budget. Work without a cost estimate can be deferred.
eval.ts · opt-in early stopping
import type { Eval } from "@hona/openeval";
export default {
earlyStop: { minIntervalMs: 45_000, maxChecks: 12, maxCostUSD: 1 },
} satisfies Eval;
Early stopping requires every metric decision to be irreversible and non-null. Monitoring cannot steer the candidate. If monitoring pauses or exhausts its budget, final grading still follows execution.