Write and run evals

An eval is the smoke test for an agent: given a prompt, does the final answer contain what it should? maiden’s evals are deliberately simple — a prompt and a list of expected substrings — so they are deterministic and need no second model to judge them.

Write a case

Each case is one file under evals/:

my-agent/evals/echo.toml

prompt = "echo the phrase hello"
expect = ["echoed"]

Key	Type	Description
prompt	string	The message sent to the agent.
expect	string[]	Substrings the final answer must contain. Omit or leave empty to only assert the run succeeds.

A case passes when every string in expect appears somewhere in the answer. The file stem is the case name (and the thread it runs on).

Run them

maiden eval my-agent --mock

  ok   echo
[maiden] 1/1 eval(s) passed

A failing case prints what was missing and the answer it got, and the command exits non-zero — so maiden eval slots directly into CI:

  FAIL echo — missing ["echoed"]
       answer: I'm not sure what you mean.
[maiden] 0/1 eval(s) passed

Keep them deterministic

Evals run against ephemeral state — a fresh temporary directory per invocation — so a case never resumes a previous run. For reproducible checks that don’t spend tokens or depend on a live model, run them with --mock: the scripted provider calls the agent’s first tool and quotes its output, which is stable across runs. That’s why the example above asserts echoed — it’s the marker the echo tool always returns under the mock.

Run evals against a real model (drop --mock) when you specifically want to check live behavior, but expect more variance and design the expect substrings to be robust — assert on stable markers a tool emits, not on the model’s exact phrasing.

Why substrings, not a judge

Substring matching has no dependency on a model, no non-determinism, and no per-run cost. It won’t grade nuance — but for the job an eval actually does (prove the agent boots, wires its tools, and returns the shape you expect) it’s the honest, fast choice.