Your first eval
From two Markdown files to a recorded, scored agent run.
On this page
Start with two files
my-benchmark/
benchmark.ts
evals/
ask-dialect/
prompt.md # the task
judge.md # the criteria
evals/ask-dialect/prompt.md
Write a SQL query for the ten most recent orders for a customer.
evals/ask-dialect/judge.md
# Requests the SQL dialect
## Metric: asked_dialect — Asks for the SQL dialect
Pass when the agent asks which database or SQL dialect is in use.
Fail when it assumes a dialect without asking.
Asking alongside a draft counts.
## Metric: safe_parameters — Uses bound parameters
Pass when the proposed query uses a bound customer-ID parameter
and explains how to supply its value.
Fail when it interpolates customer input into SQL
or does not provide a parameterized query.
Choose your models
| You need | Why |
|---|---|
| Bun 1.4.2+ | Runs TypeScript declarations and the SDK |
| Docker | Provides isolated candidate workspaces |
| OpenCode connections | Supplies access to your candidate and judge models |
terminal · inside my-benchmark
bun add --exact @hona/openeval
benchmark.ts
import type { Benchmark } from "@hona/openeval";
export default {
models: ["provider/candidate-model"],
judge: { model: "provider/judge-model" },
repetitions: 3,
concurrency: 10,
} satisfies Benchmark;
Replace both provider/model placeholders with models connected in your OpenCode installation. Choose a candidate to evaluate and a judge to read its recorded work.
Build. Plan. Run. Inspect.
terminal
bunx --bun @hona/openeval image
bunx --bun @hona/openeval plan
bunx --bun @hona/openeval run
bunx --bun @hona/openeval view
- Build the image
- Inspect the plan
- Record the run
- Read the judgment
| Output | Where to find it |
|---|---|
| Local viewer | http://127.0.0.1:4173 |
| Run and evidence store | results/ in your benchmark directory |
| Next invocation | Resumes the same aggregate and reuses unchanged work |