# Write your own eval in 30 seconds.

Built on OpenCode. Out of the box support for all models, and consistent data from every run.

[Agent prompt](https://openev.al/agent-start.md) · [Human quick start](https://openev.al/docs/quickstart/index.md)

<a id="guide-task"></a>

## 1. Task

### evals/ask-dialect/prompt.md

```markdown
Write a SQL query for the ten most recent orders for a customer.
```

### evals/exact-answer/prompt.md

```markdown
Reply with exactly APPLE.
```

<a id="guide-judge"></a>

## 2. Judge

### evals/ask-dialect/judge.md

```markdown
# Requests the SQL dialect

## Criterion: asked_dialect — Asks for the SQL dialect

Pass when the agent asks which database or SQL dialect is in use.
Fail when it assumes a dialect without asking.
Asking alongside a draft counts.

## Criterion: safe_parameters — Uses bound parameters

Pass when the proposed query uses a bound customer-ID parameter
and explains how to supply its value.
Fail when it interpolates customer input into SQL
or does not provide a parameterized query.
```

### evals/exact-answer/judge.ts

```typescript
import type { JudgeContext } from "@hona/openeval";

export default ({ response }: JudgeContext) => ({
  scores: { correct_answer: response.text === "APPLE" },
});
```

<a id="guide-run"></a>

## 3. Run

### terminal

```shell
bunx --bun @hona/openeval run
```

<a id="guide-inspect"></a>

## 4. Inspect

Example data · 2 evals · 6 models · 9 repetitions

Benchmark

| Model | Thinking | Score |
| --- | --- | --- |
| GPT-6 Astra | high | 91.67% |
| Claude Fable 5.1 | high | 75% |
| Big Pickle | default | 72.22% |
| DeepSeek V4.1 Flash | high | 69.44% |
| GLM-5.3 | high | 55.56% |
| Gemini 3.8 Flash | high | 44.44% |

ask-dialect

| Model | Thinking | Score | Asks for the SQL dialect | Uses bound parameters |
| --- | --- | --- | --- | --- |
| GPT-6 Astra | high | 94.44% | 8/9 | 9/9 |
| Claude Fable 5.1 | high | 83.33% | 7/9 | 8/9 |
| Big Pickle | default | 77.78% | 7/9 | 7/9 |
| DeepSeek V4.1 Flash | high | 72.22% | 6/9 | 7/9 |
| GLM-5.3 | high | 55.56% | 5/9 | 5/9 |
| Gemini 3.8 Flash | high | 44.44% | 4/9 | 4/9 |

exact-answer

| Model | Thinking | Score | Correct answer |
| --- | --- | --- | --- |
| GPT-6 Astra | high | 88.89% | 8/9 |
| Claude Fable 5.1 | high | 66.67% | 6/9 |
| DeepSeek V4.1 Flash | high | 66.67% | 6/9 |
| Big Pickle | default | 66.67% | 6/9 |
| GLM-5.3 | high | 55.56% | 5/9 |
| Gemini 3.8 Flash | high | 44.44% | 4/9 |

| Model | Cost | Tool calls | Context | Duration |
| --- | --- | --- | --- | --- |
| GPT-6 Astra | $0.91 | 14 | 41k tokens | 5m 12s |
| Claude Fable 5.1 | $1.34 | 18 | 52k tokens | 6m 41s |
| Gemini 3.8 Flash | $0.22 | 9 | 28k tokens | 2m 28s |
| DeepSeek V4.1 Flash | $0.18 | 22 | 61k tokens | 8m 50s |
| GLM-5.3 | $0.05 | 11 | 33k tokens | 3m 25s |
| Big Pickle | Not run | Not run | Not run | Not run |

<a id="guide-compare"></a>

## 5. Compare

### benchmark.ts

```typescript
import type { Benchmark } from "@hona/openeval";

export default {
  models: [
    "opencode/gpt-6-astra#high",
    "opencode/claude-fable-5-1#high",
    "opencode/gemini-3.8-flash#high",
    "opencode-go/deepseek-v4.1-flash#high",
    "opencode/glm-5.3#high",
    "opencode/big-pickle",
  ],
  judge: { model: "opencode/gpt-6-astra#high" },
  repetitions: 9,
} satisfies Benchmark;
```

[Documentation index](https://openev.al/llms.txt)