Conversion Lift Incrementality Study
Use controlled market exposure to estimate the customers advertising truly adds.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 99%
A conversion lift study replaces click-level attribution with a controlled comparison. The marketer chooses comparable geographic markets, runs a campaign in an exposed group, and leaves another group dark. Total signups, customers, demand, or revenue are then compared across those groups, including conversions attributed to direct, organic, and other channels. The measured difference estimates the campaign's incremental contribution. That lift is converted into a ratio that supports an indirect ROAS calculation and budget decisions. Because audience behavior, channel effectiveness, and market conditions change, the estimate must be refreshed periodically. The method may require temporarily withholding effective advertising from some markets, sacrificing short-term demand to obtain a more trustworthy measurement.
Origin
Extracted from Marketing Against The Grain through Kieran Flanagan's explanation of incrementality measurement and Zapier's state-level social-video studies.
Core principles
- 01Incrementality asks what happened because the campaign ran.
- 02Untreated markets provide the counterfactual for exposed markets.
- 03Influenced conversions may appear through direct, organic, or unrelated channels.
- 04Lift ratios must be recalibrated as channels and markets change.
How to run it
- 1
Choose the Outcome
Define the total business result the study will measure, such as free signups, customers, or incremental revenue. Include conversions from every channel through which influenced users may arrive.
Pro tip Use an outcome close enough to revenue to support investment decisions.
Watch out A narrow platform-reported conversion metric can omit the indirect behavior the test is designed to detect.
- 2
Create Comparable Groups
Select similar geographic markets and divide them into exposed and control groups. Establish baseline differences before campaign activation.
Pro tip Match markets on historical conversion volume, seasonality, and customer profile.
Watch out Large pre-existing differences can be mistaken for campaign lift.
- 3
Control Campaign Exposure
Activate the campaign in exposed markets while keeping it off in control markets for a defined period.
Pro tip Avoid unrelated regional campaign changes during the test window.
Watch out Contaminating control markets weakens the counterfactual.
- 4
Calculate Incremental Lift
Compare the change in total outcomes between exposed and control markets. Attribute the defensible excess in exposed markets to the campaign.
Pro tip Use data-science support for uncertainty estimates and adjustments where available.
Watch out Do not assign every exposed-market conversion to the campaign.
- 5
Convert Lift Into an Investment Ratio
Relate incremental customers or revenue to campaign spend to derive an indirect return measure. Use that ratio to guide optimization and budgeting.
Pro tip Report direct and indirect return separately before presenting a combined view.
Watch out The estimate is a model, not a permanent universal attribution constant.
- 6
Recalibrate Periodically
Repeat conversion lift studies every three to nine months to detect changes in channel contribution.
Pro tip Schedule recalibration before major budget cycles.
Watch out Refreshing the estimate may require turning campaigns off in markets where they normally run.
In the wild
Zapier ran social video through TikTok in selected US states and compared free-user signups with states where the activity was not running. The team correlated the relative signup increase with campaign exposure and applied a ratio to estimate how many additional signups the advertising generated.
→ The company obtained an incrementality estimate for video advertising whose conversions appeared through other channels.
Common mistakes
Counting Every Exposed Conversion
Incrementality is the difference caused by exposure, not the total number of conversions occurring in exposed markets.
Ignoring Cross-Channel Conversions
Customers influenced by video may later arrive through direct, organic, or another paid channel, so platform clicks alone undercount impact.
Using a Stale Lift Ratio
Channel effects change over time; an old coefficient can misallocate current spending.
Is it for you?
Best for
It is best for brands running video, social, brand-response, or other campaigns with weak click-level attribution.
Not ideal for
It is not ideal for very small campaigns whose expected lift is too weak to separate from geographic or seasonal noise.
From the transcript
“An incrementality model is basically how many additional customers am I getting indirectly from this spend.”
“So you go dark in some states, if you use the US, and you turn them on in other states, and then you look in…”
“So like three to six months, maybe nine months. You have to then figure out has anything changed by doing more conversion lift studies for…”
From the episode
ROAS Is a Trap: How Smart Marketers Really Drive Growth