FOR PEOPLE WHO WRITE EVALS

Write the task.
Judge the evidence.

A prompt, a rubric, and a real agent run.
Measure the behavior you care about. See why it passed.

bun add --exact @hona/openeval

prompt.md taskjudge.md criteria0 | 1 | null decisions

ask-dialectinteractive exampleNo model calls

judge.md

1# Requests the SQL dialect
2
3## Metric: asked_dialect — Asks for the SQL dialect
4
5Pass when the agent asks which database or SQL dialect is in use.
6Fail when it assumes a dialect without asking.
7Asking alongside a draft counts.
8
9## Metric: safe_parameters — Uses bound parameters
10
11Pass when the proposed query uses a bound customer-ID parameter
12and explains how to supply its value.
13Fail when it interpolates customer input into SQL
14or does not provide a parameterized query.

benchmark.ts

1import type { Benchmark } from "@hona/openeval";
2
3export default {
4  models: ["provider/candidate-model"],
5  judge: { model: "provider/judge-model" },
6  repetitions: 3,
7  concurrency: 10,
8} satisfies Benchmark;

Write criteria that accept equivalent valid answers

CANDIDATE RESPONSE

“Which database are you using—PostgreSQL, MySQL, SQLite, or SQL Server?”

If PostgreSQL, here is a parameterized draft:

SELECT id, created_at, total
FROM orders
WHERE customer_id = $1
ORDER BY created_at DESC, id DESC
LIMIT 10;

Bind $1 to the customer ID with your database driver.

Judge decisions2 / 2 passed

Click a metric to locate its response evidence.

Illustrative recording1 eval · 2 independent metricsMake this your first eval

Your evals are files

File	What you write
prompt.md	A natural, focused task
judge.md	Named metrics and clear criteria
eval.ts	Optional workspace and early stop

The authoring contract

Every score has a trail

Task
Recording
Judgment
Score

Inspect tool calls and workspace artifacts
Follow each decision to its evidence
Rejudge without rerunning the candidate

Read a judgment

THE RESULTS WORKSPACE

Compare. Drill down. Inspect.

How scores work

OpenEval results viewer with per-model scores and run costs — One score per model, with metrics available in the eval drilldown. Illustrative data.Open full size ↗

Run a small batch first.

Keep the full aggregate. Execute only the evals, models, and repetitions you select.

Plan and resume work

terminal

1bunx --bun @hona/openeval plan --only-eval ask-dialect
2bunx --bun @hona/openeval run --only-repetition 1
3bunx --bun @hona/openeval view