Experimental Input-Impact Scorecard
Judge early experiments by execution and impact before demanding ROI.
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 94%
The scorecard replaces premature ROI pressure with two comparisons: planned inputs versus completed inputs, and expected impact versus delivered impact. The first comparison tests execution quality by asking whether the team performed the work it committed to perform. The second tests the underlying hypothesis by comparing the anticipated effect with observed evidence such as attention, adoption, engagement, or qualified demand. Leaders can then decide whether to expand, revise, or terminate the experiment without treating uncertainty as failure. ROI can return as the governing measure once the channel and its conversion path become sufficiently understood. This preserves creative exploration while still imposing explicit accountability.
Origin
Extracted from Marketing Against The Grain, drawing on Anu El Turu's lessons from building Clubhouse.
Core principles
- 01Premature ROI targets kill promising experiments.
- 02Controllable inputs reveal whether a team executed the test properly.
- 03Expected impact provides a benchmark without pretending outcomes are predictable.
- 04Delivered impact determines whether an experiment deserves further investment.
How to run it
- 1
State the hypothesis
Describe the initiative, the audience behavior it should change, and why that change could eventually create business value.
Pro tip Keep the hypothesis narrow enough to test in one bounded cycle.
Watch out Do not disguise an open-ended project as an experiment.
- 2
Define controllable inputs
List the actions, assets, outreach, or production volume the team can directly control.
Pro tip Use observable commitments rather than vague effort measures.
Watch out Inputs alone do not prove that the underlying idea works.
- 3
Estimate expected impact
Choose leading indicators and record the magnitude of impact expected before running the test.
Pro tip Use a range when uncertainty is high.
Watch out Do not retroactively lower the expectation to make the result appear successful.
- 4
Measure delivered impact
Collect the actual leading and lagging indicators produced by the initiative.
Pro tip Include qualitative evidence when quantitative tooling is immature.
Watch out Avoid substituting vanity activity for audience or business impact.
- 5
Make a portfolio decision
Compare inputs and impact, then continue, modify, scale, or stop the initiative.
Pro tip Introduce ROI targets gradually as the conversion path becomes repeatable.
Watch out Do not scale a well-executed experiment whose impact consistently misses expectations.
In the wild
A startup funds six short founder videos. Instead of demanding immediate pipeline, it scores whether all videos were published, whether the intended audience watched them, and whether branded searches and direct visits increased as expected. Strong attention but weak site traffic prompts a revised call to action rather than immediate cancellation.
→ The team preserves a promising channel while fixing the weak transition from influence to demand.
Common mistakes
Demanding immediate ROI
An immature initiative may not yet have the tooling or conversion path needed to demonstrate financial return, so an early ROI gate can terminate learning prematurely.
Measuring inputs without impact
Completing the planned work is evidence of execution, not evidence that the idea deserves continued investment.
Moving the target after launch
Changing expected impact after observing results makes the comparison meaningless and encourages self-serving conclusions.
Is it for you?
Best for
It is best for early-stage channels, creative initiatives, and tests whose monetization path is not yet mature.
Not ideal for
It is not ideal for established programs with reliable attribution and a clear obligation to produce near-term revenue.
From the transcript
“sometimes ROI isn't the best measure for success.”
“you should measure inputs versus outputs, measure impact expected versus impact delivered.”
“if you hold them accountable to ROI at the start, you're going to just kill them.”
From the episode
How to Build A World-Class Distribution Engine