Write judge rubrics
Small, independent decisions with evidence you can inspect.
On this page
Declare each metric
judge.md
# Requests the SQL dialect
## Metric: asked_dialect — Asks for the SQL dialect
Pass when the agent asks which database or SQL dialect is in use.
Fail when it assumes a dialect without asking.
Asking alongside a draft counts.
## Metric: safe_parameters — Uses bound parameters
Pass when the proposed query uses a bound customer-ID parameter
and explains how to supply its value.
Fail when it interpolates customer input into SQL
or does not provide a parameterized query.
| Part | Contract |
|---|---|
| ## Metric: id — Label | Declares a stable metric ID and a display label |
| Pass and fail criteria | State the observable behavior, including omissions |
| Domain references | Supply version-correct facts and relevant sources |
| Multiple metrics | All declared metrics are graded from the same recording |
Decide what counts
| SQL response | Asks for dialect | Bound parameters |
|---|---|---|
| Asks which DB; supplies a parameterized draft | 1 · pass | 1 · pass |
| Assumes PostgreSQL; supplies a bound $1 query | 0 · fail | 1 · pass |
| Only asks which database | 1 · pass | 0 · fail |
| Recording is unavailable | null · unknown | null · unknown |
Let the shared judge handle mechanics
- Read the rubric
- Inspect the recording
- Cite evidence
- Submit every metric
The native openeval-judge agent supplies the common evidence, citation, and submission instructions. Your rubric supplies task-specific criteria. Structured tools validate the judge's submission and return errors for correction.
| Author responsibility | OpenEval responsibility |
|---|---|
| Define intended behavior | Validate metric IDs and 0 / 1 / null values |
| Accept equivalent correct approaches | Validate citations against the recording |
| Explain borderline cases | Keep recordings and judgments immutable |
| Calibrate your criteria | Aggregate metric values consistently |