Curiosity-Test-Revise SEO Loop
Test uncertain search beliefs, observe outcomes, and revise before scaling.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 90%
Search platforms are black boxes whose behavior changes, so this framework treats every optimization theory as a falsifiable hypothesis. Start with a clear claim about what should affect ranking, then create the smallest controlled implementation that could reveal whether the claim is useful. Observe ranking, traffic, and user outcomes long enough to distinguish signal from noise. If evidence contradicts the theory, update the model rather than defending sunk effort. Only scale a change after the mechanism survives testing. The episode's PageRank-sculpting story illustrates the cost of reversing this order: a broad rearchitecture took months before external testing showed that the tactic did not work as assumed. The repeatable advantage is not always being right; it is learning faster and limiting the cost of being wrong.
Origin
Extracted from Marketing Against The Grain through Kieran Flanagan's reflection on SEO experimentation and a failed PageRank-sculpting implementation.
Core principles
- 01Search assumptions remain hypotheses until tested.
- 02Curiosity creates questions; experiments create evidence.
- 03Unexpected results should change the operating model.
- 04Small tests are safer than site-wide architectural bets.
How to run it
- 1
Form a falsifiable hypothesis
Define the specific change, predicted effect, and evidence that would disprove the idea.
Pro tip Write the hypothesis before examining results.
Watch out Vague claims can be rationalized regardless of the outcome.
- 2
Limit the blast radius
Select a representative but contained set of pages, templates, or queries for the test.
Pro tip Use matched cohorts where possible.
Watch out Do not begin with a six-month site-wide rearchitecture.
- 3
Establish a baseline
Record relevant rankings, traffic, engagement, and technical conditions before changing anything.
Pro tip Include user outcomes so ranking gains do not hide experience losses.
Watch out Seasonality and concurrent releases can confound the comparison.
- 4
Run and observe
Implement the test and monitor it for an appropriate period without repeatedly changing the treatment.
Pro tip Keep an experiment log with dates and external events.
Watch out Premature conclusions can turn ordinary volatility into false learning.
- 5
Revise and scale
Update the underlying belief from the evidence, then expand only validated changes.
Pro tip Retain failed tests as institutional knowledge.
Watch out A past result may decay when the platform changes.
In the wild
A publisher suspects that a subfolder will outperform a subdomain. Instead of moving the entire archive, it migrates one comparable content section, preserves redirects and templates, records a control group, and watches search and engagement results before deciding whether to continue.
→ The team gains actionable evidence without risking the whole property.
Common mistakes
Scaling before learning
Large implementations magnify the cost of an incorrect ranking theory.
Changing several variables
Bundled changes make it impossible to identify the mechanism behind the result.
Defending sunk effort
Past implementation cost should not prevent a team from rejecting a disproven tactic.
Is it for you?
Best for
Search teams operating under incomplete information about ranking systems.
Not ideal for
Changes involving legal, security, or reputational risks that experimentation cannot safely contain.
From the transcript
“You have to be really curious. You have to iterate, and you can't actually learn until you do the thing.”
“we've realized that PageRun sculpt and this doesn't work. We've tested it, it doesn't really work the way you think it works.”
From the episode
How To Rank #1 With Google’s Secret Algorithm (Google Leak Explained)