Scrape-to-Visit Ratio
Measure whether an aggregator returns enough traffic for the content it consumes
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 93%
The Scrape-to-Visit Ratio evaluates the economic exchange between a content publisher and an aggregator. The input is the amount of content a platform crawls or otherwise consumes and the number of referral visits it returns. Dividing scraped pages by referred visitors produces a simple indicator of how much content must be supplied to earn one visit. Tracking that ratio over time exposes deterioration that raw audience-growth figures can conceal. Comparing platforms also shows where content is being summarized without meaningful traffic in return. Marketers can use the result as a channel-risk signal: a rapidly worsening ratio suggests reducing reliance on conventional search traffic, testing direct audience acquisition, and adopting metrics suited to AI-mediated discovery.
Origin
Extracted from Marketing Against The Grain while analyzing Cloudflare data on the collapsing traffic exchange between publishers, Google, and AI assistants.
Core principles
- 01Content distribution depends on a measurable value exchange
- 02Scraping without referral traffic weakens publisher economics
- 03Ratio trends matter more than raw traffic totals
- 04Platform dependence becomes riskier as the exchange deteriorates
How to run it
- 1
Select the platform
Choose one search engine, AI assistant, or aggregator whose value exchange you want to evaluate. Keep platforms separate because their referral behavior can differ dramatically.
Pro tip Start with the platform responsible for the largest share of your discovery traffic.
- 2
Measure content consumption
Estimate how many pages or content units the platform crawled, scraped, or retrieved during a defined period. Use the same measurement method in every comparison period.
Pro tip Record both the source and date range for the crawl estimate.
Watch out Do not mix page crawls with unrelated bot requests or internal traffic.
- 3
Measure returned visits
Count attributable referral visits from that platform during the same period. Apply consistent attribution rules so the ratio remains comparable.
Pro tip Separate ordinary search referrals from paid traffic where possible.
Watch out Dark or unattributed AI referrals may cause returned visits to be understated.
- 4
Calculate the ratio
Divide consumed pages by returned visits to determine how many pages the platform takes for each visitor it sends. A larger number represents a weaker exchange for the publisher.
Pro tip Express the result as pages scraped per one visit.
- 5
Compare the trend
Compare the ratio with earlier periods and with other platforms. Look for abrupt changes associated with features such as AI-generated answers or zero-click results.
Pro tip Annotate product launches and algorithm changes on the timeline.
Watch out A short-term spike may reflect measurement noise rather than a durable shift.
- 6
Rebalance acquisition
Treat a persistently worsening ratio as a signal to diversify acquisition and update success metrics. Test direct outreach, owned audiences, paid channels, and AI share-of-voice measurement instead of assuming referral traffic will recover.
Pro tip Set a threshold that triggers a formal channel-allocation review.
Watch out Do not abandon a profitable channel solely because its ratio worsened; assess customer value and total economics too.
In the wild
The episode compares three snapshots of Google's exchange with publishers: two pages scraped per visit ten years ago, six pages per visit six months ago, and eighteen pages per visit after the expansion of AI overviews. The rising ratio indicates that publishers supply substantially more content for each referral they receive.
→ The comparison identifies a ninefold deterioration in the traffic exchange and highlights growing dependence risk.
A publisher applies the same ratio to an AI assistant and finds that it consumes far more pages per referral than conventional search. The team retains AI visibility as a brand objective but stops forecasting meaningful website traffic from that platform.
→ The publisher separates brand exposure from traffic acquisition and adjusts its channel forecasts accordingly.
Common mistakes
Tracking traffic without the exchange
Raw referral totals do not show how much content a platform consumed to produce those visits. Measure both sides of the exchange.
Comparing inconsistent periods
Using different crawl definitions, attribution rules, or date ranges creates a misleading trend. Keep measurement boundaries stable.
Treating the ratio as the only metric
A deteriorating ratio signals platform risk but does not capture customer quality, revenue, or brand exposure. Use it alongside business-outcome metrics.
Is it for you?
Best for
It is best for publishers and marketers evaluating SEO economics, platform dependence, and shifts toward zero-click discovery.
Not ideal for
It is not ideal when crawl data, referral attribution, or comparable historical periods are unavailable.
From the transcript
“For every two pages they scraped, they sent you one visitor on average across the entire internet.”
“for every 18 pages that Google takes from you, you get one visitor.”
“What's the ratio for Open AI 6 months ago? 250 to1. What is it today? 1,500 to1.”
From the episode
The Great Content Collapse