# Your first eval

One SQL task. Two criteria. Choose a model, run it, and inspect the work.

<a id="setup"></a>

## 1. Check the tools

### terminal

```shell
bun --version
docker info
opencode --version
```

| Tool | Ready when |
| --- | --- |
| Bun | Version 1.4.2 or later |
| Docker | The daemon is running with Linux containers |
| OpenCode | Version 2; the SDK runtime is pinned to 2.0.22 |

[Install Bun](https://bun.com/docs/installation) · [Install Docker](https://docs.docker.com/get-started/get-docker/) · [Install OpenCode V2](https://opencode.ai/v2/docs/)

If your V2 executable is named opencode2, use that name for the OpenCode commands below.

<a id="directory"></a>

## 2. Pick a directory

Use an existing benchmark package or an empty project. For a new one:

### terminal

```shell
mkdir my-benchmark
cd my-benchmark
bun init -y
bun add --exact @hona/openeval@0.5.3
bunx --bun @hona/openeval --version
```

Keep the following commands in this directory. In an existing package, keep its manifest and add the dependency with its package manager.

### .gitignore · add these entries

```text
node_modules/
results/
```

<a id="skill"></a>

## 3. Add the writing skill

[Download eval-writing.zip](https://openev.al/eval-writing.zip)

Extract it into your project root. Keep an existing customized skill if you already have one.

### project files

```text
.opencode/skills/eval-writing/
  SKILL.md
  README.md
  references/
    openeval.md
    examples.md
    research.md
```

Open this project in OpenCode. Use /eval-writing when you want help with your own task or rubric.

### terminal

```shell
opencode .
```

<a id="files"></a>

## 4. Write one eval

An eval is a task plus its grading rules. The candidate does the task; the judge grades its recording. Each criterion is one scored requirement.

Try this SQL demo: does the candidate ask which database you use, and handle the customer ID safely? Create these files:

### my-benchmark/

```text
benchmark.ts
evals/
  ask-dialect/
    prompt.md    # the task
    judge.md     # the criteria
```

### evals/ask-dialect/prompt.md

```markdown
Write a SQL query for the ten most recent orders for a customer.
```

### evals/ask-dialect/judge.md

```markdown
# Requests the SQL dialect

## Criterion: asked_dialect — Asks for the SQL dialect

Pass when the agent asks which database or SQL dialect is in use.
Fail when it assumes a dialect without asking.
Asking alongside a draft counts.

## Criterion: safe_parameters — Uses bound parameters

Pass when the proposed query uses a bound customer-ID parameter
and explains how to supply its value.
Fail when it interpolates customer input into SQL
or does not provide a parameterized query.
```

| Example response | Asks for dialect | Bound parameters |
| --- | --- | --- |
| Asks which DB; gives a bound-parameter draft | 1 | 1 |
| Assumes PostgreSQL; gives a bound-parameter draft | 0 | 1 |
| Only asks which database | 1 | 0 |

> **Use your own task?**
>
> Start with one real request, one acceptable result, and one failure. Write criteria that distinguish them. Keep grading rules out of prompt.md. For an exact check, use a judge.ts function instead.

[Task prompts](https://openev.al/docs/prompts/index.md) · [Rubrics](https://openev.al/docs/rubrics/index.md) · [Code judges](https://openev.al/docs/code-judges/index.md) · [Workspace files](https://openev.al/docs/workspaces/index.md)

<a id="models"></a>

## 5. Choose models and variants

In OpenCode, connect your provider and pick a candidate and a judge:

### inside OpenCode

```text
/connect
/models
```

Replace both placeholders below with connected model IDs. A benchmark groups your evals and model choices.

### benchmark.ts

```typescript
import type { Benchmark } from "@hona/openeval";

export default {
  models: ["provider/candidate-model"],
  candidate: { websearch: false },
  judge: { model: "provider/judge-model", websearch: false },
  repetitions: 1,
  concurrency: 1,
} satisfies Benchmark;
```

Use provider/model#variant for a listed variant, such as #high. Omit the suffix for the default. Start with one candidate and one repetition. This demo needs no web search.

Using only judge.ts files? Omit judge.model; you only need the candidate model.

[Models & variants](https://opencode.ai/v2/docs/models/) · [Benchmark type](https://github.com/Hona/openeval/blob/v0.5.3/packages/openeval/src/types.ts)

<a id="run"></a>

## 6. Plan, then run

### terminal

```shell
bunx --bun @hona/openeval image
bunx --bun @hona/openeval plan --only-eval ask-dialect --only-repetition 1
```

Check the plan: ask-dialect, your candidate model, repetition 1. Planning previews work and its cost estimate. When you are ready to call the models:

### terminal

```shell
bunx --bun @hona/openeval run --only-eval ask-dialect --only-repetition 1
```

Running later? Keep the files and use the same command when ready.

[Run scopes & budgets](https://openev.al/docs/running/index.md) · [CLI reference](https://openev.al/docs/reference/index.md)

<a id="inspect"></a>

## 7. Watch and inspect

To watch live, open a second terminal in the same directory. You can also open the viewer after the run.

### second terminal

```shell
bunx --bun @hona/openeval view
```

[Open the results viewer](http://127.0.0.1:4173/)

1. Select ask-dialect
2. Select your model
3. Read Session / Judge
4. Follow the evidence

| Result | Read it as |
| --- | --- |
| Criterion score | 1 = pass, 0 = fail, null = unresolved |
| Benchmark score | Mean credit; this demo's [1, 0] gives 50% |
| Cost and tokens | Recorded metrics, separate from the score |
| results/ | Retained runs, judgments, and evidence |

Next: add your other model IDs or increase repetitions in benchmark.ts, then plan and run again. OpenEval resumes the same result and keeps completed work.

[Read the evidence](https://openev.al/docs/evidence/index.md) · [Scoring](https://openev.al/docs/scoring/index.md) · [Definitions](https://openev.al/docs/terminology/index.md)

[Documentation index](https://openev.al/llms.txt)