Incentive Distortion Check
Separate authentic product value from behavior purchased by rewards
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 96%
The incentive distortion check treats every reward as an intervention that changes the meaning of observed behavior. A team first identifies an action that appears to predict conversion or retention, ideally using unincentivized historical data. Before rewarding that action, it states two competing explanations: users perform it because it reveals product value, or users perform it because they want the reward. The team then uses a control group, downstream conversion, post-reward retention, and continued voluntary use to separate those explanations. If activity rises while durable adoption does not, the incentive bought motion rather than value. This prevents onboarding dashboards, quest completions, and token-driven user counts from being misread as authentic demand.
Origin
The hosts developed this diagnostic while comparing product-led-growth activation analysis with Web3 systems that pay users to perform supposedly predictive actions.
Core principles
- 01Incentives create behaviors and alter the signal being measured.
- 02Rewarded activation does not prove intrinsic product value.
- 03Correlations discovered before an incentive may fail after rewards are introduced.
- 04Post-incentive retention is stronger evidence than rewarded completion.
- 05Healthy growth survives the removal of the reward.
How to run it
- 1
Define the target behavior
Name the action believed to predict customer conversion, retention, or successful product use. Document the evidence for that belief before introducing a reward.
Pro tip Choose one behavior at a time so the causal question remains understandable.
Watch out A correlation is not automatically a causal activation mechanism.
- 2
Capture the baseline
Measure how often users perform the behavior voluntarily and what happens afterward. Segment by acquisition source and relevant user type.
Pro tip Use a pre-incentive cohort or randomized holdout as the comparison group.
Watch out Without a baseline, increased activity can be mistaken for increased product value.
- 3
Introduce the incentive experimentally
Offer the reward to a bounded cohort while retaining a comparable control group. Keep other onboarding changes stable where possible.
Pro tip Predefine the evaluation period and success metrics.
Watch out A universal rollout can erase the counterfactual needed to interpret results.
- 4
Measure downstream outcomes
Compare conversion, retention, meaningful usage, and unit economics—not just completion of the rewarded action. Look for differences after the reward is exhausted.
Pro tip Pay special attention to voluntary return behavior.
Watch out Reward redemption is not evidence of customer intent.
- 5
Classify and adjust
Decide whether the incentive accelerated authentic discovery, merely rented behavior, or damaged the original signal. Retain, redesign, or remove it accordingly.
Pro tip Reduce the reward gradually to test whether the behavior remains.
Watch out Continuing a vanity-growth program can hide deteriorating product-market fit.
In the wild
A software company finds that users who install its mobile app during the first week convert at a higher rate. It offers a discount for installation, causing installations to surge. A holdout group reveals that rewarded installers do not retain or upgrade more often than other users, while voluntary installers still do. The company stops labeling incentivized installation as an activation signal.
→ The team preserves a meaningful activation metric and avoids scaling an expensive vanity campaign.
Common mistakes
Optimizing the rewarded action
An action becomes a poor proxy when the team maximizes it without checking the downstream outcome it was meant to predict.
Skipping the control group
Without comparable unincentivized users, the team cannot tell whether rewards accelerated value discovery or simply purchased compliance.
Stopping measurement at completion
The decisive evidence often appears after the payment, discount, or token distribution ends.
Is it for you?
Best for
It is best for product and growth teams using discounts, cash, tokens, or gifts to influence onboarding behavior.
Not ideal for
It is not ideal when the reward is itself the permanent product benefit and cannot meaningfully be separated from normal usage.
From the transcript
“when you start to incentivize someone to do something, you've completely changed the dynamics of that thing, right?”
“Because it's unclear now if that is still the thing that will correlate to usage that becomes customers, or you've just incentivized that.”
“Incentives create behaviors. Behaviors don't create incentives.”
From the episode
How DAOs Will Change Media In The Future with Peter Yang
Peter Yang