Write task prompts
Give the candidate a real task. Put the scoring rules in the rubric.
On this page
Write what the user would ask
prompt.md
Write a SQL query for the ten most recent orders for a customer.
| Put in the prompt | Put in judge.md |
|---|---|
| The user's goal and constraints | What qualifies as pass or fail |
| Context the agent should have | Domain facts used to check its answer |
| Requested output or deliverable | Accepted alternatives and edge cases |
OpenEval sends prompt.md verbatim. If you want to measure whether an agent notices missing information, preserve that ambiguity in the task.
Make the starting conditions explicit
evals/query-from-schema/
prompt.md
judge.md
workspace/
schema.sql
README.md
workspace/schema.sql
CREATE TABLE orders (
id BIGINT PRIMARY KEY,
customer_id BIGINT NOT NULL,
created_at TIMESTAMP NOT NULL,
total DECIMAL(12, 2) NOT NULL
);
Use workspace files for project context. Keep evaluator references and answer keys in judge.md, outside the candidate environment.
Review the behavior you are measuring
| Question | Useful distinction |
|---|---|
| Is this a task or a grading hint? | Ask for the outcome; let the candidate choose its method. |
| Would another valid workflow pass? | Keep command choices out of the rubric unless they are the intended behavior. |
| What happens if it only asks a question? | Make that outcome explicit in each metric. |
| Is the task reproducible? | Supply the starting files and pin external inputs. |