Five-Eval Improvement Loop
Turn expected outputs into tests, then iteratively tune agent behavior
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 99%
The Five-Eval Improvement Loop converts an agent's intended behavior into a small initial test suite. For each representative input, specify the expected output or an explicit grading rule. Run the agent, record which cases fail, quantify the failure rate, and inspect patterns rather than reacting to one anecdotal output. Modify the prompt or other relevant system component, then rerun the complete set to detect improvement and regression. Starting with five evaluations makes the practice accessible while creating a foundation that can grow from production failures and edge cases. The loop provides an empirical standard for quality and prevents teams from declaring success merely because a few demonstrations looked convincing.
Origin
Extracted from Marketing Against The Grain during Tina Huang's explanation that evaluation constitutes the other half of agent development.
Core principles
- 01Define success before judging agent quality
- 02Represent expected behavior with repeatable input-output evaluations
- 03Begin with a small evaluation set rather than none
- 04Quantify failures instead of relying on impressions
- 05Use failures to guide iterative prompt and system improvements
How to run it
- 1
Define Success
Write down the behavior, output quality, restrictions, and business result that would make the agent acceptable.
Pro tip Use criteria a reviewer can apply consistently.
Watch out Vague goals such as 'be helpful' are difficult to test reliably.
- 2
Create Five Initial Evals
Choose at least five representative inputs covering normal cases, important variations, and a likely failure mode.
Pro tip Use anonymized examples from the real workflow when available.
Watch out Five nearly identical easy cases create false confidence.
- 3
Set Expected Results
For each input, define the exact expected answer or a grading rule that determines pass or fail.
Pro tip Use deterministic expectations where the task permits them and rubrics where outputs can vary.
Watch out Do not rewrite the expected result after seeing the model's answer merely to make it pass.
- 4
Run and Quantify
Execute the complete evaluation set and record individual failures, categories, and failure percentage.
Pro tip Keep model and system settings stable when comparing iterations.
Watch out A single successful demo is not evidence of consistent behavior.
- 5
Diagnose and Adjust
Identify the shared cause of failures and revise the prompt, context, tool use, guardrails, or architecture accordingly.
Pro tip Change one major factor at a time so the effect remains interpretable.
Watch out Not every failure is a prompting problem; some require better data or system design.
- 6
Rerun for Regression
Run all evaluations again after each material change and add newly discovered edge cases to the suite.
Pro tip Preserve previously passing cases to catch regressions.
Watch out Do not test only the failed example after modifying the agent.
In the wild
An evaluation supplies the input two plus two and expects four. If the agent produces five, the case fails and contributes to the measured failure rate, giving the builder a concrete problem to investigate.
→ Correctness becomes measurable rather than a matter of intuition.
A churn agent receives five representative cancellation emails. Each evaluation checks whether it identifies the reason, uses only an approved offer, acknowledges the concern, and escalates unsupported cases.
→ The team can quantify whether prompt and workflow changes improve safe personalization.
Common mistakes
Running Zero Evals
Without repeatable tests, builders are left guessing whether changes improve or degrade the agent.
Testing Only Happy Paths
An agent can pass obvious examples while failing ambiguity, prohibited actions, and real-world edge cases.
Tweaking Without Rerunning
A change that fixes one case can silently break behavior that previously worked.
Is it for you?
Best for
Agent builders moving from a plausible demonstration toward reliable and measurable behavior.
Not ideal for
Workflows whose desired outcomes have not been defined well enough to grade consistently.
From the transcript
“So that's why evaluations is like the other 50%.”
“Five evals is much better than zero evals, for example.”
“And this is how you can go back and iteratively improve the results of your agent.”
From the episode
I Used ChatGPT & n8n to Stop Customers from Leaving
Tina Huang