TL;DR
The crawl-to-referral ratio divides the pages an AI operator fetches from you by the referral visits its platform sends back. Cloudflare Radar publishes it per operator, and it has become the standard evidence in every block-or-allow argument.
It was designed for ad-funded publishers deciding whether to license, block or charge, and for that job it is a good metric. Almost nobody quoting it in 2026 is an ad-funded publisher.
Decompose it and your own performance cancels out. Crawls per referral equals crawls per citation multiplied by citations per referral, and your citation count sits in both terms. That is why the published spread is between engines — roughly 1,917:1 against about 5:1 in the same month — and never between two publishers on the same engine.
Both terms are mismeasured, in opposite directions. Cloudflare notes that native-app referrals often carry no Referer header, and The Digital Bloom estimates 70.6% of AI traffic arrives with no referrer at all — the slice that converts best.
Apply the Absence Test: any metric you can score perfectly by ceasing to exist is a cost metric. Cost metrics are useful, and they belong to finance. On a marketing dashboard they do what cost metrics always do — minimise the activity being measured.
The usable data is in the numerator you were told to shrink. In July 2026 Claude-User was the second-busiest bot on the internet at 8.31% of all bot requests, and it fetches pages when a person asks a question.
What the crawl-to-referral ratio is, and what it was built to do
Every argument about AI crawlers now ends the same way. Somebody opens Cloudflare Radar, points at a column and says: they took nearly two thousand pages from us and sent back one visitor. The room goes quiet, and the decision is effectively made. That number is doing more work than any other statistic in the field, and almost nobody has asked what it is made of.
What the number actually is
What is the crawl-to-referral ratio?
The crawl-to-referral ratio divides the number of HTML pages an AI operator’s crawlers fetch from a site by the number of referral visits that operator’s platform sends back over the same window. Cloudflare Radar publishes it per operator, normalised to a single referral, so a ratio of 250:1 means 250 fetches for every visitor returned.
The published spread is wide and it moves fast. Cloudflare’s original June 2025 table put Anthropic at 70,900:1. The same measure read 23,951:1 across Q1 2026, 11,122:1 for the single week to 1 June 2026, roughly 3,386:1 in June and 1,917:1 for the full month of July 2026. OpenAI travelled the same direction, from around 903:1 in May to 251:1 in July, dropping below Perplexity at 289:1 for the first time in the series. Google has sat near 5:1 throughout.
Two housekeeping rules before anyone quotes those figures. Ratios are window-specific and must never be averaged across windows; a 31-day month is a noisier sample than a 91-day quarter, and the most careful series say so on every chart. And Cloudflare notes that native-app referrals frequently arrive with no Referer header, which means the published ratios may overstate the imbalance by an amount nobody has quantified.
The job it was designed for
It is worth being fair about the metric’s origins, because the criticism that follows is not that it is badly built. It is well built. Cloudflare produced it so that site owners could make an informed allow-or-block decision, and so that publishers negotiating licensing terms had a defensible number to put on the table. For a business whose product is the page, whose revenue is the pageview and whose counterparty is manufacturing a substitute from its inventory, crawls-per-referral is close to the perfect summary of the bargain. The UK playbook for AI content licensing and pay-per-crawl is exactly the context this number was drawn for.
Why it escaped that job
Then it spread, and it spread for a reason that has nothing to do with fitness. The crawl-to-referral ratio is free, universal, already computed and sitting in a dashboard the infrastructure team opens anyway. Every rival measure of AI visibility costs money, requires a prompt panel, and returns a number somebody can argue with. A metric’s adoption is governed by the cost of computing it, not by the cost of being wrong about it — which is how a licensing instrument built for newspapers ended up as a board KPI at software companies, retailers and professional-services firms that do not sell a single pageview.
The transplant is easy to miss, because the ratio sounds like a fairness measure and fairness feels like everybody’s business. A number is only a fairness measure if both parties are trading the same good, and a newspaper and a software company are not: one is selling the page, the other is using the page to be found. Anyone who has worked through the wider 2026 link building statistics will recognise the shape of the problem — a headline figure travels much further than the sampling frame it was drawn from, and the frame is where the meaning lives.
Two numbers from your property, one decision on someone else’s
Both terms are counted at your own edge: requests arriving, and visits arriving. Neither touches the event that decides whether the exchange was worth it — the moment a model composes an answer and chooses whose name goes in it. That happens on a third party’s infrastructure, and nothing computed from your own logs observes it directly. The ratio is a two-terminal reading of a three-terminal circuit, and the terminal it cannot reach is the one where the value is settled.
What the numerator is really counting
The fetch count is not one population. It is at least three, with opposite economics. Training fetches feed a corpus and carry no referral mechanism by design. Search-purpose fetches build an index that can later produce a cited link. Agent fetches happen in real time because a person asked something. Cloudflare’s May 2026 breakdown put training at 51.8% of AI crawler requests and search-purpose at only 9.3%, so the headline number is dominated by the category least able to refer anybody. More than half of AI crawler requests re-fetch pages that have not changed, which means a meaningful share of your numerator is somebody else’s caching inefficiency. And user-agent strings are trivially forged — anyone can send a request that claims to be a major crawler from a laptop — so an unverified numerator counts impostors alongside operators.
The volume behind those fetches is climbing fast enough that composition matters more every quarter. DataDome counted 17.7 billion AI agent requests across its network in Q2 2026 against 12.2 billion in Q1, a 45% rise in three months, and Fastly measured AI request growth of roughly 30% between January and May. Anything that shifts the mix — a new agentic product, a caching improvement, one operator’s training run — moves your ratio without a single thing changing on your side of the wire. Teams that treat this as a technical problem at the infrastructure layer at least hold the logs to see the composition shift. Teams reading one headline number do not.
What the denominator cannot see
The referral count is worse. Cloudflare’s own caveat about missing Referer headers on native-app traffic is the conservative version. The Digital Bloom estimates that 70.6% of AI traffic may arrive without referrer data at all, and reports that this dark slice converts at 10.21% against 1.66% for the visible AI traffic and 0.15% for organic. The best-performing visits are the ones the denominator cannot count. Agentic browsers make it structurally worse: Comet, Atlas, Claude for Chrome and Copilot Actions all drive a real browser session and emit the underlying browser’s user agent, so the fastest-growing form of machine-mediated visiting appears in your logs as an ordinary human on Chrome. Anything that changes what a click is worth in an agentic session is invisible to a metric built on the referrer header.
There is a third leak, and it is the one that should end the argument. Seer Interactive found that brands cited inside AI Overviews earn 35% more organic clicks and 91% more paid clicks on the same queries than brands that are not cited. The measurable payoff from being crawled and cited lands in the organic and paid rows of somebody else’s report. It never passes through the referral line at all.
Key takeaway
The numerator over-counts, because it mixes training, search and agent fetches with re-fetches and forgeries. The denominator under-counts, because most AI-sourced visits carry no referrer and the largest measured effect of a citation shows up in other channels. Both errors are real, both are unquantified, and they run in opposite directions.
The decomposition: why your own performance cancels out
Set the measurement problems aside for a moment and grant the ratio a perfect count of both terms. It still cannot tell you how you are doing, and the reason is arithmetic rather than instrumentation.
Let C be the fetches an operator makes against your site in a period, A the answers in which it cites you, and R the referrals it sends. Then the headline ratio is an identity:
C / R = (C / A) × (A / R)
The first term is fetch overhead: how many pages the operator spends per answer that names you. The second is answer completeness: how many citations it takes to produce one click. Read them separately and the ownership is obvious — and so is the fact that A cancels. Your citation count, the one variable in the system you actually control and the only one that represents your success, appears in both terms and disappears from the product. Double your citations and you raise C and R together; the ratio barely moves.
The Crawl-to-Referral Decomposition
| Term | What it actually measures | Who owns the lever |
| C / A — fetch overhead | How efficiently the operator converts crawling into answers. Dominated by its caching, refresh cadence and training schedule. | The crawler operator, almost entirely. |
| A / R — answer completeness | How often a citation still produces a click. Set by the surface, the query mix and how completely the answer resolves the question. | The answer engine and the intent profile of the query. |
| C / R — the headline | The product of two variables you do not own, with your own performance algebraically removed. | Nobody on your side of the transaction. |
Read the third column downwards. Every lever in the ratio belongs to the counterparty.
The empirical tell
If the decomposition is right, the variance in published ratios should sit between operators and not between publishers. That is exactly what the data shows. In July 2026 the spread ran from roughly 1,917:1 for one operator to about 5:1 for Google on the same network in the same window — a difference of several hundred times, entirely explained by which company was doing the crawling. No published series reports a comparable spread between two publishers on the same engine, because there is not one to report. The 2026 collapse from 70,900:1 to under 2,000:1 was likewise not publishers getting better at earning referrals; it was operators getting better at caching and shipping consumer products that link out.
The practical consequence is worth stating plainly. Two brands in the same category, one cited four times as often as the other, will report near-identical crawl-to-referral ratios for the same operator over the same window. Nobody publishes a competitive benchmark for this metric the way they publish one for referring domains, because there is no signal in the comparison. A number that cannot separate the leader from the laggard is not measuring the contest — it is measuring the venue. The strategies that genuinely move citation share have always had to be judged on different evidence, and this is why.
What is a good crawl-to-referral ratio?
There is no publisher-side benchmark, because the number is a property of the engine rather than of your content. Comparing your ratio for a given operator against another site’s ratio for the same operator will produce two similar numbers regardless of which site is winning the citations.
The Absence Test: telling a cost metric from a performance metric
There is a one-question diagnostic that sorts this out faster than any argument about attribution, and it generalises well beyond crawlers. Ask what score the metric returns if the organisation ceases to participate entirely.
The Absence Test
Question: if you went dark tomorrow — blocked every crawler, published nothing, removed yourself from the market this metric describes — would the metric improve?
Crawl-to-referral ratio: yes. Block everything and the numerator falls to zero. Perfect score, no business.
Bot share of traffic: yes. Same mechanism, same perfect score.
Blocked-request rate: yes, trivially. It is a measure of your own enforcement, not of any outcome.
Citation share on a fixed question set: no. Going dark drives it to zero.
Cost per cited query: no. Absence makes it undefined, not excellent.
Verdict: any metric you can perfect by disappearing is a cost metric. Cost metrics are genuinely useful and they belong to finance. Put one on a marketing dashboard and it will do what cost metrics do — minimise the activity being measured.
This is not a hypothetical failure mode. Enforcement against AI bots reached a 9.64% block rate across July 2026 and 12.92% in the final week of the month, and the pattern is familiar to anyone who has watched a channel measured wrongly and then killed because the dashboard looked flat. The blocking is a rational response to the only number on the table. The number is the problem.
The asymmetry underneath it matters more than the arithmetic. A bandwidth bill arrives monthly, itemised, and can be cut this week. Absence from a retrieval set is never billed, never notified, has no equivalent of a disavow file to reverse it and no control arm to measure it against. When one error is measurable and the other is not, the dashboard chooses the measurable one.
The numerator is a demand dataset
Here is the inversion the ratio hides. In July 2026 Claude-User was the second-busiest individual bot on the internet at 8.31% of all bot requests, behind only Googlebot at 12.88% and ahead of Meta-ExternalAgent at 6.72% and GPTBot at 5.26%. Claude-User is not a training crawler. It fetches a page because a person asked a question and the model went to look. The single fastest-growing line in the numerator is live demand, filed as an extraction cost.
An agent-purpose fetch is a timestamped record that a real user’s question routed a machine to a specific URL of yours. Nothing else in the stack gives you that. Citation trackers sample a prompt panel and read the answers, which tells you where you appeared; they cannot see what was fetched and then discarded. Your own edge logs can. It is closer to an impression log for the answer layer than anything the current AI-visibility monitoring tools are able to sell you, and it is already in your possession.
It also changes what a crawl means. For twenty-five years the case for being crawlable was indexation, and what backlinks do for a page was understood almost entirely through that lens: get fetched, get indexed, rank later. An agent fetch is a different animal. It is not preparation for a future query; it is a live one. Fetch frequency by URL behaves far more like a demand curve than like a crawl budget, and it responds to the same events that move link velocity across a real campaign — a publication, a mention, a spike in category interest — usually within days rather than months.
The near-miss set
The most valuable thing in that log is the pages that get fetched repeatedly and never cited. A URL that is never fetched has a discovery problem. A URL that is fetched and passed over has cleared discovery and failed selection — it was retrieved, read and judged not worth quoting. Those are different diagnoses with different budgets attached, and the crawl-to-referral ratio collapses both into the same undifferentiated numerator. Sites running listicle and roundup placements as a citation tactic see the same effect from the other side: the placement is retrieved constantly and quoted rarely, and only a fetch log shows the gap.
What the log honestly cannot tell you
- No query. You see a URL, a timestamp and a user agent. The prompt that caused the fetch is not transmitted, so you get demand by page, never demand by question.
- A subset, not the whole. Many answers are composed from an index or a cache with no live fetch, so agent-purpose traffic is a lower bound on retrieval, biased towards queries that need freshness.
- Agentic browsers are silent. Sessions driven through a real browser carry no bot signal at the HTTP layer and will not appear in this population at all.
- Forgery. Unverified user-agent strings are worthless. Verify by signed request or published IP range before a single figure reaches a report.
Should I block AI crawlers with a bad crawl-to-referral ratio?
Not on the strength of the ratio alone, because the ratio is set by the operator’s crawling efficiency rather than by your content’s value. Decide per purpose instead: training access is a licensing question for legal and the board, while agent and search access are visibility questions whose cost is paid in citations you will never see disappear.
Running the audit properly
The replacement is not a better ratio. It is a purpose-split read of the same log, run on a schedule, with the cost line sent to finance and the demand line kept by marketing.
The seven steps
- 1. Pull edge logs, not origin logs. A managed edge blocks before robots.txt is read, so requests refused upstream never reach your origin — and an origin log gives a clean bill of health for traffic that was turned away. It is the most common misdiagnosis in this exercise.
- 2. Verify every agent. Match against signed requests or published IP ranges. Discard everything unverified into a separate bucket and report it as a security line, not a demand line.
- 3. Classify by declared purpose. Split training, search and agent user agents into three populations and never merge them again. Each answers a different question and carries a different decision.
- 4. Join to URL and template. Aggregate agent-purpose fetches by page and by page type. The concentration curve is usually brutal and always instructive.
- 5. Overlay the citation panel. Run a fixed question set on a schedule and mark which of your URLs appear. Fetched-and-cited, fetched-and-not-cited, and never-fetched are your three working segments.
- 6. Aggregate over weeks, not days. Research on measuring visibility in generative search finds that cited source sets overlap by only 34–42% between consecutive days, so a single reading is noise. Roll two to four weeks before drawing any conclusion.
- 7. Send the cost line elsewhere. Attribute bandwidth and origin compute to bot traffic honestly, put it on the infrastructure budget, and take it off the marketing dashboard permanently.
The four numbers that replace the ratio
Agent-fetch coverage: the share of your estate receiving at least one verified agent-purpose fetch in a rolling 28-day window. Near-miss rate: the share of fetched URLs that never surface in the citation panel. Cost per cited query: programme spend divided by the questions on which you are named, the only figure here that connects the work to a business outcome. And the honest infrastructure cost, reported quarterly to the people who pay it. Teams already running a disciplined competitor backlink analysis will recognise the shape — the value is in the segmentation, not the headline.
Two practical notes on the panel. Keep the question set fixed across quarters even when it stops flattering you, because a moving panel produces a moving baseline and destroys the comparison you built it for. And resist the urge to score every page: the concentration curve means a small fraction of URLs absorbs most of the demand, so a well-run audit produces a short list rather than a spreadsheet. Teams already publishing interactive tools and calculators as linkable assets tend to find those pages sitting near the top of the fetch list, which usually settles the internal argument about whether they were worth building.
One by-product is worth naming. Nearly every widely quoted figure in this article came from somebody publishing their own log analysis. Server-log studies of AI crawler behaviour get cited and linked at a rate most content in this field never approaches, because almost nobody has the data and everybody wants it. An audit you were going to run anyway becomes a proprietary dataset worth building a link programme around, which is a rare case of measurement paying for itself twice.
Worked example: two readings of the same log
Verewood Corporate Services is invented and the figures are illustrative, but the mechanics are not. It is a Bristol company-formation and statutory-compliance provider: about £24M revenue, 190 staff, roughly 41,000 SME clients, and 1,430 guide pages covering confirmation statements, PSC registers, dormant accounts and share transfers. Its 2027 measurement and content budget is £190k.
In January 2027 the edge dashboard shows a crawl-to-referral ratio of 4,100:1 against its largest AI counterparty, 61% of HTML requests automated, and AI-referred sessions at 1.2% of total. Citation share on a 60-question panel across four engines, rolling 28-day, is 9.7%. The finance director asks why the company is subsidising an AI lab. Two plans go to the board.
| Line | Plan A — manage the ratio | Plan B — instrument the numerator |
| Diagnostic spend | £30k crawler reporting and ratio dashboards | £38k edge log pipeline: capture, signature verification, purpose classification, URL join |
| Access policy | £40k bot management; block training and agent purposes for the two worst-ratio operators, keep search | £0; keep access open, verify everything, decide per purpose after three months of data |
| Content spend | £120k redirected to paid search to replace expected lost traffic | £72k rebuilding the pages agents actually fetch; £45k quarterly first-party filing-rejection dataset |
| Measurement | Monthly crawl-to-referral ratio as the board KPI | £35k question panel; coverage, near-miss rate and cost per cited query |
Both plans spend £190k. Only one of them can observe what it bought.
Months 1 to 6
Plan A works immediately on its own terms: the ratio falls from 4,100:1 to 190:1, bot bandwidth drops 22%, and about £9k a year comes off the infrastructure bill. Plan B’s ratio gets worse — 4,600:1, because agent fetches keep growing — and it spends month three defending a number heading the wrong way. Then the purpose split lands. Seventy-one per cent of Plan B’s verified agent-purpose fetches concentrate on 96 URLs, 6.7% of the estate; 34 of those were on a pruning list for low organic traffic. Twenty-two are fetched repeatedly and appear in no answer on the panel. Over the same window Plan A’s citation share on the blocked engine slides from 9.4% to 6.1%, which appears on no dashboard it owns.
Months 7 to 12
Plan A cannot attribute the slide, because the instrument that would have shown it was the one it switched off. It unblocks in month eight, and recovery is slow: caches expire on their own schedule, re-crawling takes weeks, and by then the answers have settled on two rivals. Plan B publishes its rejection-reason dataset drawn from 41,000 filings, and the accountancy trade press picks it up. At month twelve Plan A reports a ratio of 210:1, citation share of 6.8%, £9k saved and 12 new referring domains. Plan B reports 3,900:1, citation share of 17.4%, 96 rebuilt pages and 61 new referring domains. AI-referred sessions are still only about 2% of Plan B’s traffic, which is the honest part — the win never showed up in the referral line.
What the board was told
“Plan A perfected a ratio by leaving the market that produces it. Plan B kept the worse ratio and took a larger share of the answers.”
What this example does not claim: Plan A’s saving was real, its bot bill was real, and for an image-heavy or paywalled estate the arithmetic can genuinely favour blocking. The failure was not the block. It was governing a marketing decision with a cost metric, and then having no instrument left that could see the consequence.
Where this argument is weakest
The strongest objection is not that the ratio is imperfect. It is that it is exactly right for the people who built it, and that this whole argument is a marketer moving the goalposts because the honest number is unflattering. Put at full strength: for an ad-funded publisher the pageview is the product, the referral is the revenue and the crawl is a real marginal cost. One publisher recorded €42,000 a month in CDN bandwidth savings after blocking unauthorised crawlers. Large estates run one to ten terabytes a month in bot egress. The output being assembled from that inventory is a direct substitute for the product. Crawl-to-referral was built for that business and for the licensing negotiation that follows, and telling a publisher their extraction metric is a cost metric is like telling a landlord that rent is not a growth KPI. They know.
That is correct, and I am not going to soften it. Four things bound it rather than refute it.
- The error is the transplant, not the metric. Almost every organisation now quoting the ratio in a marketing meeting does not sell pageviews. The number did not get worse; it got moved.
- It is a pricing number, not a management number. The cancellation applies to publishers too. Use the ratio to set a price with a counterparty; it cannot tell you what to publish, because it does not move when you get better.
- The two errors are not equally reversible. Bandwidth is billed, itemised and recoverable. Absence from a retrieval set is unbilled, unnotified and has no control arm.
- The exception is real and worth naming. For media-heavy estates, unverified scraper floods, or a business whose product genuinely is licensed content, the arithmetic can favour blocking outright. The right instrument there is still a cost line owned by finance.
What would falsify this?
If site-level benchmarking shows large variance in the ratio between publishers on the same engine in the same window, and that variance correlates with citation share, then the cancellation claim is empirically dead and the ratio does carry publisher-performance information. Separately, if operators begin passing a query or a citation identifier on the fetch, the near-miss set becomes directly observable and this stops being a technique and starts being a product. Watch for both.
What survives the deadline
Strip out the dates, the operator names and the September default, and a durable pattern is left. Every time a new intermediary inserts itself between your content and its audience, the first metric to appear is the one the intermediary can compute cheaply — and that metric describes the intermediary’s economics, not yours. Server logs, referrer headers, impressions, average position: each arrived as somebody else’s accounting and was adopted as our scorecard. The crawl-to-referral ratio is the current instance, and it will not be the last.
The defensive habit is to keep two ledgers and never let them merge. A cost ledger for what machine traffic costs to serve, owned by the people who pay the bill. A demand ledger for what machine traffic reveals about who is asking for you, owned by the people who can act on it. The characteristic failure of this period is one number attempting both jobs and doing neither.
And the deeper answer to a metric you do not control is not a better metric. It is owning an input that no re-pricing of access can take away. Corroboration earned on other people’s domains is re-established on every query, sits outside any single operator’s crawl policy, and is the one asset whose value does not depend on which header happens to be transmitted this quarter. That is as true for a specialist working across European markets as it is for a single-market brand, and it is why what link building is actually for has survived every measurement fashion that came before this one.
It is also why the placements least sensitive to any of this keep quietly outperforming their reputation. Sponsorships, registers and institutional listings were never priced on referral traffic in the first place, so nothing about a missing header or a changed crawler default alters what they are worth. The assets that look dull on a referral report are frequently the ones with the least exposure to how the referral report is computed.
The Monday checklist
- Pull last month’s edge logs, not origin logs, and confirm which requests were refused before they reached you.
- Verify agent identity by signature or published IP range, and quarantine everything unverified into a security bucket.
- Split the verified population into training, search and agent purposes, and commit to never reporting them as one figure again.
- Aggregate agent-purpose fetches by URL and identify the top 5% of pages absorbing most of the demand.
- Stand up a fixed 40–60 question panel across your priority engines and run it weekly, aggregating over four weeks before reading anything into it.
- Cross the two lists to produce your near-miss set, and treat fetched-not-cited pages as a content brief rather than a pruning candidate.
- Move the crawl-to-referral ratio and bot bandwidth share off the marketing dashboard and onto the infrastructure budget, where they are genuinely useful.
