# Write judge rubrics

Small, independent decisions with evidence you can inspect.

<a id="criteria"></a>

## Declare each criterion

### judge.md

```markdown
# Requests the SQL dialect

## Criterion: asked_dialect — Asks for the SQL dialect

Pass when the agent asks which database or SQL dialect is in use.
Fail when it assumes a dialect without asking.
Asking alongside a draft counts.

## Criterion: safe_parameters — Uses bound parameters

Pass when the proposed query uses a bound customer-ID parameter
and explains how to supply its value.
Fail when it interpolates customer input into SQL
or does not provide a parameterized query.
```

| Part | Contract |
| --- | --- |
| ## Criterion: id — Label | Declares a stable criterion ID and display label |
| Scoring rules | State the observable behavior, including omissions |
| Domain references | Supply version-correct facts and relevant sources |
| Multiple criteria | All declared criteria are graded from the same recording |

<a id="categories"></a>

## Group criteria into categories

### judge.md

```markdown
## Criterion: asked_dialect — Asks for the SQL dialect
Categories: misalignment
```

### judge.ts

```typescript
export const criteria = {
  correct_answer: { name: "Correct answer", categories: ["general"] },
} satisfies CodeCriteria;
```

A category is any string. Put the optional Categories: line directly below a heading, or export criteria from judge.ts. OpenEval removes category lines before the judge reads the rubric, so adding or changing categories never rejudges recorded evidence.

The viewer adds a Categories tab with a radar chart and a models-by-categories heatmap, and a filter that rescores any view to the selected categories. Benchmark results also show a scorecard: the benchmark on the left, one row per category, and one column per model. Export image saves that table or the totals chart as a PNG. Set categories in benchmark.ts to compose a benchmark from them, or pass --only-category to run a subset.

| Kind | Published set |
| --- | --- |
| Capability (Artificial Analysis) | agents · coding · general · scientific-reasoning |
| Agent failure (MAST, NeurIPS 2025) | specification · misalignment · verification |

<a id="outcomes"></a>

## Decide what counts

| SQL response | Asks for dialect | Bound parameters |
| --- | --- | --- |
| Asks which DB; supplies a parameterized draft | 1 · pass | 1 · pass |
| Assumes PostgreSQL; supplies a bound $1 query | 0 · fail | 1 · pass |
| Only asks which database | 1 · pass | 0 · fail |
| Recording is unavailable | null · unknown | null · unknown |

> **Missing advice and missing evidence are different**
>
> An observed omission can fail a criterion. If the necessary recording is unavailable, the judge can return null. An unresolved criterion score keeps the final benchmark percentage unresolved.

<a id="shared-judge"></a>

## Let the shared judge handle mechanics

1. Read the rubric
2. Inspect the recording
3. Cite evidence
4. Submit every criterion score

The native openeval-judge agent supplies the common evidence, citation, and submission instructions. Your rubric supplies task-specific criteria. Structured tools validate the judge's submission and return errors for correction.

| Author responsibility | OpenEval responsibility |
| --- | --- |
| Define intended behavior | Validate criterion IDs and normalized 0–1 or null scores |
| Accept equivalent correct approaches | Validate citations against the recording |
| Explain borderline cases | Keep recordings and judgments immutable |
| Calibrate your criteria | Aggregate criterion scores consistently |

[Documentation index](https://openev.al/llms.txt)