Reproducible AEO Experiment Loop
Test uncertain tactics, reproduce effects, and share the evidence
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 94%
This loop treats AEO guidance as provisional until a controlled intervention produces an observable and repeatable effect. The team begins with a specific hypothesis, records a baseline over multiple runs, changes one meaningful factor, and monitors the same prompts, citations, or conversion outcomes. It then repeats the intervention or reproduces it in another comparable setting. Only effects that survive repetition become operational guidance; null results and failures are documented as well. Sharing the hypothesis, setup, observations, and limitations allows other teams to reproduce the work. The loop is particularly important in AEO because testing is harder than in mature channels and plausible ideas can become accepted facts merely through repetition.
Origin
Extracted from Marketing Against The Grain
Core principles
- 01Plausibility is not proof
- 02A visible effect must be reproduced
- 03Stable prompts and measurements make comparisons useful
- 04Shared experimental data accelerates collective learning
How to run it
- 1
Form a falsifiable hypothesis
Define one proposed mechanism and the observable result that would support or contradict it.
Pro tip Use a statement such as: adding a mention to a cited URL will increase brand inclusion for this prompt set.
Watch out Vague goals such as improving AEO cannot be meaningfully tested.
- 2
Establish a repeated baseline
Run the target prompts or measurements multiple times before changing anything and record the natural variability.
Pro tip Keep engine, account state, location, and prompt wording as stable as practical.
Watch out A single baseline observation may be an outlier.
- 3
Change one variable
Implement the specific content, technical, or off-site intervention being tested while holding unrelated factors steady.
Pro tip Timestamp the intervention and preserve the before state.
Watch out Simultaneous changes make attribution difficult.
- 4
Measure the effect
Repeat the baseline procedure and compare citations, brand inclusion, placement, traffic, or conversions.
Pro tip Use both immediate and delayed measurement windows when crawling may take time.
Watch out Do not interpret one favorable output as proof.
- 5
Reproduce the result
Repeat the intervention, reverse it where safe, or test it on a comparable page or prompt group.
Pro tip Seek reproduction across more than one prompt when the mechanism should generalize.
Watch out An effect that cannot be reproduced should remain a hypothesis.
- 6
Publish the evidence
Share the setup, data, outcome, and limitations so others can evaluate and reproduce the result.
Pro tip Include null results to reduce duplicated effort and survivorship bias.
Watch out Do not convert an anecdote into a universal rule.
In the wild
A company records whether it appears across repeated runs of ten purchase-intent prompts. It earns a truthful mention on one frequently cited comparison URL, waits for recrawling, and reruns the same observations. It then repeats the tactic on a second cited URL to see whether any improvement reproduces.
→ The team gains evidence about whether citation-page mentions affect its target answers rather than relying on industry hearsay.
Common mistakes
Mistaking crawling for impact
A bot fetching a new file or page does not prove that the content influences retrieval or answers.
Changing several variables together
Concurrent content, markup, and outreach changes prevent the team from identifying the active mechanism.
Publishing only successful tests
Omitting null and negative outcomes creates survivorship bias and encourages repetition of ineffective tactics.
Is it for you?
Best for
Teams evaluating emerging optimization tactics for which reliable industry evidence is still limited.
Not ideal for
Situations where no stable outcome can be observed or where several uncontrolled changes must happen simultaneously.
From the transcript
“The way to know what works is to set up an experiment and to see an effect and to reproduce that.”
“So we don't have the answers to many things.”
“Everyone should be doing experiments and sharing their experiment data so that we can all learn and figure out the answers faster.”
From the episode
‘My Data Proves SEO is NOT Dead’ + How to Rank #1 on Google & AI