Step-Function Experiment Prioritization
Favor experiments with transformational upside over marginal optimization
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 5
- Confidence
- 94%
Compare initiatives by their plausible improvement range and opportunity cost, not merely by how predictable or familiar they appear. Traditional optimization often pursues one-to-five-percent gains in existing processes. AI may permit new systems, routing mechanisms, or automation that can produce much larger changes in core metrics. The framework does not require blindly choosing the riskiest idea; experiments should still have bounded scope, acceptable failure consequences, and measurable outcomes. However, once those guardrails exist, a team should question whether incremental work justifies the capacity it consumes. Build an experiment portfolio around hypotheses with step-function upside, release them before they are perfect, and use live evidence to decide which deserve further investment. Preserve incremental work only where reliability, compounding gains, or mandatory obligations make it strategically necessary.
Origin
Extracted from Marketing Against The Grain during Kip Bodner and Emmy Jonathan's discussion of AI's opportunity cost for marketing teams.
Core principles
- 01Team capacity should follow potential impact, not familiarity
- 02AI can enable new systems rather than merely optimize old ones
- 03A small reliable gain may still be inferior to a bounded high-upside experiment
- 04Imperfect early versions can produce the evidence needed for improvement
How to run it
- 1
Inventory Competing Work
List the incremental optimizations and AI-enabled experiments competing for the same people, time, and budget.
Pro tip Include maintenance and recurring optimization work that normally escapes prioritization reviews.
Watch out Hidden business-as-usual work can consume most experimentation capacity.
- 2
Estimate Improvement Ranges
Assign each initiative a conservative and optimistic outcome range using available evidence.
Pro tip Express outcomes in core KPIs rather than feature completion or activity volume.
Watch out Do not present speculative maximum upside as an expected result.
- 3
Apply Risk Boundaries
Assess reversibility, customer exposure, data quality, and the consequences of failure.
Pro tip Shrink a promising experiment's scope rather than rejecting its hypothesis outright.
Watch out High upside does not excuse unbounded operational or customer risk.
- 4
Compare Opportunity Cost
Ask what larger experiment cannot happen if the team continues pursuing a marginal gain.
Pro tip Convert long-running small optimizations into explicit capacity costs.
Watch out A familiar initiative can feel inexpensive because its opportunity cost remains invisible.
- 5
Launch and Reallocate
Fund the best bounded, high-upside tests, measure their outcomes, and shift resources toward hypotheses that prove out.
Pro tip Stop weak experiments quickly while expanding validated ones.
Watch out Do not continue an AI initiative solely because its theoretical upside was large.
In the wild
Instead of concentrating only on small marketing optimizations, HubSpot invested in an AI chat system. The experiment reportedly produced a 43% increase in qualified-lead conversion and more than 50% improvement in value per chat on some pages.
→ A new operating system produced a larger KPI change than the one-to-five-percent gains associated with conventional optimization.
Common mistakes
Treating Every AI Idea as Transformational
An AI label does not establish large upside; the hypothesis still needs a mechanism connecting the intervention to a core outcome.
Abandoning Essential Incremental Work
Security, reliability, compliance, and compounding conversion improvements may remain mandatory despite smaller headline gains.
Requiring Perfection Before Evidence
Delaying a bounded experiment until it is flawless sacrifices learning and increases opportunity cost.
Is it for you?
Best for
It is best for growth and marketing teams choosing between familiar incremental work and bounded AI-enabled experiments with substantially larger upside.
Not ideal for
It is not ideal when foundational reliability, regulatory compliance, or severe customer defects require immediate incremental work.
From the transcript
“Now in the world of AI, you have the ability to build new systems and programs and automation that can get you 50%, 100%, 300%…”
“So the opportunity cost of going out there and working on something that might be a five percent improvement gain is just far too expensive.”
“So, you know, get it to a certain place, get it out, and then you really start getting your learnings and you can see the…”
From the episode
How We Hacked Hubspot With Ai To Make Free Money