Pay-Per-Crawl vs Pay-Per-Use: Choosing a 2027 AI-Access Model

TL;DR

Cloudflare retired the crawl as the unit of payment on 1 July 2026 and proposed the citation in its place, on the reasoning that a page might be fetched once and used in thousands of answers, or fetched constantly and never used at all. That reasoning is correct. The conclusion drawn from it is not automatic. Pay-per-crawl and pay-per-use do not price the same good — one sells a copy, the other sells an outcome — and they differ far more in

where the meter sits than in what the rate is. You can count fetches at your own edge. You cannot count uses inside someone else’s inference stack. This piece gives you the Use-Per-Fetch Multiplier, the freshness-versus-canonicity sorting rule, the Meter Questions to ask before signing anything, and the part that actually changes your outreach: pay-per-use pays your

hosts, not you — and it changes what they will accept from you.

1. Two models, and the question nobody is asking about either

On 1 July 2025, Cloudflare told AI companies they would have to pay each time a crawler fetched a page. On 1 July 2026, the same company declared that model insufficient and proposed paying publishers when their content appears inside an answer instead. Twelve months, two units of account — and an industry debating which rate is better without noticing that the two schemes are not selling the same thing.

What is pay-per-use AI crawl pricing?

Pay-per-use pays a publisher when their content is actually used to produce an AI output, rather than when a crawler fetches the page. It is the successor to pay-per-crawl, which charged a set fee per request using HTTP 402 Payment Required — a dormant status code revived for the purpose, with Cloudflare acting as merchant of record and publishers choosing one of three settings per crawler: allow free, charge, or block. Pay-per-use launched with two named partners running two different mechanics. Ceramic.ai operates a pay-per-query model, paying opted-in publishers when their content appears in its search results and returning query-level reporting — the queries that surfaced the content, the exact page and snippet shown, and the average ranking position within its output. You.com does something else again: it lets an AI agent pay on demand for a specific piece of premium content at the moment it needs it, with no upfront commitment from either side — the same on-demand fetch pattern behind AI browsers and agent-driven retrieval. No pricing structure was disclosed for either.

Why the unit of account changed at all

The pressure behind both models is a measured collapse in the traffic that used to pay for content. The most rigorous number available comes from a randomised field experiment by Saharsh Agarwal of the Indian School of Business and Ananya Sen of Carnegie Mellon’s Heinz College, posted to SSRN in April 2026 and revised that June: across 1,065 desktop Chrome users and 68,089 searches, suppressing Google’s AI Overviews raised outbound organic clicks from 0.37 to 0.62 per search and cut zero-click searches from 0.73 to 0.54 — a 39.8% reduction established by random assignment rather than by before-and-after comparison. Ahrefs put the correlation at 34.5% across 300,000 keywords in April 2025 and at 58% in a February 2026 follow-up; Seer Interactive measured a 61% fall in organic click-through on AI Overview queries across 42 client organisations between June 2024 and September 2025. Our read of what the data actually shows about AI Overviews and backlinks covers the SEO side of the same collapse.

The regulatory track is moving in parallel. The European Commission opened a formal investigation into Google’s AI content practices in December 2025, examining whether publisher content was used without appropriate compensation or a viable opt-out, and the UK Competition and Markets Authority has proposed publisher opt-out rights for AI summaries. If you run campaigns across European markets, assume compensation rules diverge by jurisdiction before they converge — one more reason a single global access policy is the wrong default, as anyone running multi-country link building programmes already knows from hreflang and everything downstream of it.

The reasoning behind the shift is published and, on its face, unanswerable. More than half of the crawl traffic from bots Cloudflare classifies as legitimate goes to re-fetching pages that have not changed. Research Cloudflare published with ETH Zurich in April 2026 found that over 90% of pages processed by large-scale crawlers such as Common Crawl are unique by content, which defeats caching designed on the assumption that popular pages get requested repeatedly. So crawling is a poor proxy for value. A page might be fetched once and cited in thousands of answers, or fetched every day and never cited once.

That gives us the counterparty claim, and it is worth stating in its strongest form because almost everyone holds it: “crawling is a bad proxy for value, so paying per citation is strictly better — take the aligned model.” The first half is true and I will not argue with it. The second half does not follow. Replacing a bad proxy you can verify with a good proxy you cannot verify is not automatically an improvement, and whether it is depends on a variable nobody in this debate has named.

2. They are not two prices. They are two goods.

Strip both schemes to what changes hands.

Pay-per-crawl sells a copy

The transaction completes at your server. A request arrives, you supply bytes, a fee is owed. The good is delivery of a file. It is discrete, it happens once, and — the property that matters — it happens at your end of the wire. Your logs contain the complete record of everything you are owed for. If the invoice and your logs disagree, you know, and you can prove it. This is ordinary server-log territory that any technical SEO team already works in.

Pay-per-use sells a share of an outcome

The transaction completes somewhere you cannot see: inside a retrieval pipeline and a model, at the moment an answer is assembled for a user you will never meet. The good is not a file. It is a proportion of the value of an output, determined by a process you have no access to, counted by the party who owes you the money, and reported to you on a dashboard they built.

Key takeaway

Every argument for pay-per-use is an argument about alignment. Every risk in pay-per-use is a risk about measurement. Alignment without auditability is not a better deal — it is a better-sounding rate card, and those are not the same product.

This is not a claim that anyone is cheating. Ceramic.ai’s query-level reporting is genuinely more transparency than most licensing arrangements offer, and Cloudflare has been careful about what it claims — it describes the whole effort as an experiment, states there is a lot to learn about how the approach holds up at internet scale, and positions itself as plumbing rather than as a party setting prices. That is honest framing. It is also a precise description of a market where, as PPC Land put it in its 2 July 2026 write-up, whether a citation is worth more or less than a crawl remains a question the market has not priced.

The point is structural. A report is the counterparty’s account of the counterparty’s count; an audit is your ability to check it. That is the relationship between a bank statement and the right to inspect the ledger, and no amount of good faith converts one into the other.

3. The Use-Per-Fetch Multiplier

Cloudflare’s justification for the shift contains the variable that decides the whole question, and then walks past it. “Crawled once and cited in thousands of answers, or crawled repeatedly and never cited” describes a ratio: the number of answer-uses each fetch generates. Call it the use-per-fetch multiplier, or U/F. The arithmetic is trivial and the implication is not.

• Pay-per-crawl revenue = fetches × your price. You set the price; they control the volume. Raise the price and they fetch less, or leave. But you can count the fetches yourself.
• Pay-per-use revenue = uses × their rate. They control the volume, the rate, and the count. You can independently verify none of the three.

Which model pays you more is therefore decided entirely by U/F — and U/F is a number only the buyer can compute. You can measure F precisely, at your own edge, today. You cannot measure U at all. The variable that determines which contract is better for you sits exclusively on the other side of the table.

Reading your own corpus

U/F is not uniform across a site. It is a property of content type, and it is roughly predictable from one question: does the content have to be re-fetched in order to be used again?

Key takeaway

Read the table’s colour banding as a risk gradient, not a quality ranking. The higher your U/F, the more money is theoretically at stake — and the less able you are to check whether you received it. Your measurement exposure rises with exactly the content quality you were trying to monetise.

4. Freshness versus canonicity: the sorting rule

Compress the table into one line and you get the rule worth remembering after the specific programmes are renamed. Pay-per-crawl monetises freshness — the buyer has to come back. Pay-per-use monetises canonicity — the buyer has to cite you. Those are different assets, they are produced by different editorial strategies, and almost no publisher holds only one.

For example: a UK trade title running a weekly tariff table and a library of explainer guides owns both assets in the same CMS. The tariff table is a freshness asset — nobody can answer a pricing question from a copy taken in March. The explainer library is a canonicity asset — one fetch in March can furnish answers all year. Putting both under a single access policy prices one of them wrongly, and which one depends on which policy you picked. The same logic applies to reactive newsjacking content, which is freshness at its most extreme, and to interactive calculators and evergreen tools, which are canonicity at its most extreme.

The rule also explains which hosts will never engage with any of this. A club, charity or community site of the kind used in sponsorship and local community link building holds neither asset class in commercial quantity and has no reason to meter anything, which is why those placements stay simple while trade media gets complicated. Community platforms sit in a third position again: a Hacker News thread that earns you links is high-canonicity content whose value accrues to the platform, not to any contributor — including you.

Should I charge AI crawlers or let them in for free?

For most businesses reading this, the answer is let them in — and the money question is a distraction. Charging only makes sense if your content is the product. If your content is marketing for a product, a toll you collect on being read is trivial next to the cost of not being cited: the visibility is worth multiples of the licence fee, and you have no leverage to price it anyway. The exception is genuine proprietary data, where the calculation flips — which is exactly why original research and datasets outperform every other linkable asset format on both the earned-link and the licensing side.

5. Advertising already ran this experiment

There is no need to speculate about what happens when a payer counts the units it pays for. Digital advertising spent two decades on the question, and the arc is documented.

The impression started as a seller-counted unit, moved to buyer-counted, and became independently verified only after roughly fifteen years — and even now the gap between count and reality is wide. The ANA Programmatic Transparency Benchmark, cited in IAB Spain’s first dedicated SSP guide in April 2026, found only 41% of total programmatic investment reaches genuine, measurable, viewable impressions free of invalid traffic. DataBeat’s June 2026 Sellers Report found a 46% duplicated-domain rate among established supply-side platforms, meaning nearly half of publisher domains are reachable through more than one path and the same impression can enter an auction twice, competing against itself.

The asymmetry that decides whether verification arrives

Here is the part that should worry anyone signing a use-based agreement. In advertising, the party demanding verification was the party holding the money. Buyers could withhold spend until they got independent measurement, and eventually they did. In AI content licensing the roles are reversed: the party who needs verification is the publisher, who has no spend to withhold, no coalition, and — in most cases — no alternative buyer. Verification arrives when the powerful side wants it. That is not a cynical reading; it is just where the leverage sits.

One more figure makes the scale of the gap concrete. Cloudflare’s AI Crawl Control now produces more than a billion HTTP 402 Payment Required responses per day to AI crawlers across its customer base, with early adopters including Condé Nast, Time, the Associated Press, BuzzFeed, Reddit, Pinterest and Stack Overflow. A billion payment demands a day — and no published figure anywhere for what any of it collected. A price is not a transaction, and the distance between the two is the entire subject of this article.

6. The part that actually changes your link building

If you are a brand rather than a publisher, you may have read this far assuming the choice is yours to make. Mostly it is not, and the reason matters more than the choice.

Pay-per-use pays the host of the content, not the subject of it. When a trade publication’s article about your company is used to answer a question about your category, the publication is paid. You are not, and there is no mechanism by which you could be. So pay-per-use does not create a revenue line for you. What it creates is a change in your hosts’ incentives — and that is what will reorganise your outreach in 2027.

Your content stops being inventory and becomes a claim on an asset

Under advertising economics, an additional article was strictly good for a host. More pages, more impressions, more revenue. That is the economic fact underneath fifteen years of guest posting working as a link building tactic — your content was inventory, and inventory was the business. It is worth being precise that this is an economic change, not an algorithmic one: nothing about how a backlink passes authority has altered, and the fundamentals of link building are untouched. What changed is the host’s reason to say yes. Under corpus economics the sign flips. A page that is rarely cited does not earn; a corpus diluted with promotional filler is cited less often overall. An additional article is now a claim on a licensed asset, and the host has a direct financial reason to be more selective than they were.

The symmetric half is the opportunity. Content that makes a host more citable — original figures, named-credential expertise, data nobody else holds — moves from “we might run this” to “we want this.” The same shift devalues the commodity pitch and revalues the contributor who arrives with something checkable, which is why expert-source platforms of the Connectively, Featured and Qwoted generation matter more in 2027 than they did in 2024, not less.

How to tell whether a host has switched

There is no announcement, and most publishers will not describe the change in these terms even to themselves. Three observable signals, in ascending order of reliability. First, structural tells: a licensing or AI-terms page, RSL terms in robots.txt, a machine-readable feed offered separately from the RSS one. Second, editorial tells: fewer contributed posts, tightening author-credential requirements, a drift from listicles toward reference-format pieces that answer one question completely. Third, and most reliable, the outcome tell — put the questions their content answers to three or four engines and watch whether that domain gets cited more often across successive months. A host whose citation rate is climbing has a reason to protect it, and will behave accordingly whether or not they have signed anything.

This is a screening job and it belongs inside a process you already run. Fold it into your quarterly competitor backlink analysis by segmenting the intersect on whether each host shows the signals above, and track the result with the discipline described in measuring entity authority. Most of the standard link building tool stack cannot see any of this yet, so for now it is a manual column. It is also worth knowing which target hosts already shape how AI systems form product recommendations, because those are the ones that will price their corpus first.

The Pitch Inversion

What the ad-funded host wanted, against what the corpus-monetising host wants. Read the right column as your rewrite brief.

The bottom row is the one to sit with. Under corpus economics, attribution and the hyperlink come apart: a host who is being paid for citations has every reason to attribute a claim clearly and a diminishing reason to send the reader away. Plan for placements that name you without linking you, price them as real wins, and adjust your reporting before your board notices the referring-domain count flattening while your citation share rises. The same reasoning applies to link insertions into existing evergreen content, where you are now negotiating with a page that has its own revenue line.

7. Worked example: Barrowgate Energy, £185,000, twelve months

Barrowgate Energy is an invented Leeds B2B energy procurement and net-zero consultancy — roughly £24M revenue, 160 staff, around 900 mid-market clients. All figures are illustrative and built to show the mechanism. It runs a respected content hub with two very different halves: a quarterly UK business tariff benchmark, and a library of 60 evergreen explainer guides. Its 2027 earned-visibility budget is £185,000. In September 2026 it faced both decisions at once — what to do about access to its own hub, and what to do about a trade press that had started charging for its corpus.

The access decision

The instinct was a single site-wide policy, and the first draft was “pay-per-use everywhere, it’s the aligned model.” Running the multiplier split the answer. The benchmark data is worthless stale, so every use forces a fresh fetch — U/F near 1, and pay-per-crawl routes essentially the whole value through Barrowgate’s own logs. The explainer library is the opposite: one fetch can serve answers for two years. Final policy: metered access on the benchmark, and for the library, no toll at all plus a direct approach to two answer engines about a licence with reporting attached. Charging a toll on the library would have collected a few hundred pounds and cost the citations the guides existed to earn.

The outreach decision

Barrowgate’s standing tactic was six guest contributions a quarter to energy trade titles. In October its two best relationships declined pieces they would have run a year earlier, with near-identical feedback: useful, but nothing in it we don’t already have. The team rewrote the programme around the right-hand column of the Pitch Inversion — three pieces a quarter instead of six, each built on a cut of the benchmark data nobody else held, bylined to a named analyst with a verifiable professional registration.

• m0. 64 referring domains in the prior year, 7% citation share across four engines on fourteen procurement questions, zero licensing revenue.
• m1–m3. Output halves. 9 new referring domains against a run-rate of 22. The programme is queried at board level and defended on the strength of the acceptance rate: 3 pitches sent, 3 accepted, against 6-sent-2-accepted a year earlier.
• m4–m6. Two titles ask for the benchmark cut as a recurring quarterly feature. One offers a standing attribution credit but no link, which the old scorecard would have logged as a failure. Metered benchmark access bills £2,900 in its first full quarter — immaterial as revenue, decisive as evidence, because it is the first number in the building that came from a meter Barrowgate controls.
• m7–m9. The benchmark figures start appearing in answers attributed to Barrowgate via the trade titles rather than via its own site. Citation share reaches 16%. Referring domains are still running below the prior year.
• m12. 41 referring domains against 64 the prior year — a 36% fall. Citation share 22% against 7%. £11,400 of metered access revenue. Three recurring editorial slots that did not exist in m0, two of which carry attribution without a hyperlink.

The board line

“We published half as much, lost a third of our links, and tripled the number of answers that name us. The old report would have called that a bad year. It was the year the hosts changed what they were buying and we changed what we were selling.”

Two honest caveats. The link decline is real and it carries a real ranking cost that the citation gain does not automatically offset — these are different currencies and Barrowgate accepted a genuine trade, not a free lunch. And the £11,400 does not justify the metering work on its own — it earned its place as the only verifiable number in the programme, a smaller claim.

8. Where this argument is weakest

The strongest objection is not that pay-per-use is unauditable. It is that auditability is worthless if nobody buys.

Stated at full strength: pay-per-crawl ran for a year. Cloudflare’s network emits over a billion payment demands a day and there is no public evidence of meaningful collection. The AI companies simply routed around it, bought bulk licences from the largest publishers, or went to data brokers. Cloudflare replaced the model because it did not work, and it moved from letting publishers set a price to describing itself as plumbing precisely because per-crawl pricing found no market. A meter you control that reads zero is worth less than an unauditable royalty that reads something. And Ceramic.ai does publish query-level reporting, which is more transparency than the enterprise licensing deals this article implicitly holds up as the alternative.

All of that is correct, and the reporting point in particular is the best evidence against my case. Four things bound it rather than dissolve it.

• The failure was leverage, not metering. Pay-per-crawl collected little because the buyer could walk away. Pay-per-use does not repair that asymmetry — it relocates it, from “will they buy?” to “what will they report?” One failure is visible in your bank account within a month; the other is invisible indefinitely. Trading the first for the second is a real cost, even when the second pays better.
• Reporting is not audit, and the distinction is the whole history of ad measurement. Publishers received detailed impression reports for years while the ANA benchmark was reading 41%. Rich reporting and accurate accounting are not the same variable, and the industry that learned this lesson had far more leverage than publishers do now.
• The real answer is collective, not contractual. An individual publisher cannot negotiate audit rights into a programme offered on standard terms. This is precisely why the RSL Collective was constituted as a collective rights organisation rather than a specification body, and why its licence classes run all the way to pay-per-inference. Choose your metering model second. Choose whether you are negotiating alone first.
• For most readers here the choice is moot in the way that matters. You are not a publisher selling access; you are a brand trying to be in the corpus, and section 6 is your section. Your decision is not which model to adopt — it is how to pitch hosts who have adopted one. That reframing survives whichever way the metering argument goes.

What would falsify this?

Two checkable outcomes. First: if a pay-per-use programme publishes third-party-verified per-publisher use counts — the equivalent of what independent measurement eventually did for the impression — the measurement asymmetry closes and the core of this argument loses most of its force. Watch for an auditor’s name attached to a use count, not a vendor’s dashboard. Second: if published publisher earnings from use-based programmes come in above what access-based licensing produced for comparable corpora, then “better rate card, worse realisation” is simply wrong and the aligned model deserves the benefit of the doubt. Neither has happened as of August 2026, and both would move me.

9. What survives the next rename

Pay-per-crawl lasted twelve months as the headline model. There is no reason to think pay-per-use gets longer. Programmes will be renamed, partners will change, and the specific mechanics in section 1 will be historical detail by 2028. What does not change is the question underneath:

Any scheme that pays you for machine consumption of your content is a scheme in which somebody counts. Establish who that is before you evaluate the rate, because the rate is a number the counter can always make attractive.

The rest is a portfolio judgement: match freshness assets to models you can audit, and canonical assets to relationships you can negotiate.

The Meter Questions

Ask these three before signing any AI-access arrangement.

1. Who produces the count? If the answer is the party who owes the money, you are in a reporting relationship, not an accounting one.

2. Can you reconstruct the denominator independently? Not “can you see your number” — can you tell whether your number is a fair share of the total? If not, you cannot detect underpayment even in principle.

3. What happens in a dispute? If the escalation path terminates in the counterparty’s own dashboard, there is no dispute mechanism — there is a customer-support queue.

The Monday checklist

Seven items, none of which needs a licensing lawyer — though between them they add a licensing literacy to the link building specialist’s job description that was not there in 2025, sitting alongside rather than replacing the core tactical playbook.

1. Split your own content into freshness and canonicity halves. One spreadsheet column: does this have to be re-fetched to be used again? That column is your access policy.
2. Pull 90 days of AI-crawler hits per URL from your server logs. This gives you F. It is the only half of the multiplier you will ever be able to measure, so measure it properly.
3. Stop treating access policy as site-wide. A single toggle across a mixed corpus prices one half of your content wrongly by construction.
4. If you are considering any use-based programme, run the three Meter Questions and write the answers down. If question two has no answer, that is not a dealbreaker — it is a discount you should be pricing in.
5. Rewrite one live outreach pitch against the right-hand column of the Pitch Inversion. Send both versions to comparable targets and compare acceptance rates, not link counts.
6. Add unlinked attributed mentions to your reporting as a first-class outcome this quarter, with their own count and their own target. Under corpus economics they will grow faster than links, and a scorecard that cannot see them will report your best work as a decline.
7. Identify one dataset you could publish that nobody else holds. It is the only asset class in the table above with enough leverage to negotiate terms rather than accept them, and it is simultaneously the strongest earned-link asset you can own.

The unit of account is being rewritten in public, and the people writing it are the people who will do the counting. That is not a scandal — it is the ordinary shape of a market forming before its measurement standards exist. The advantage in 2027 goes to whoever reads the meter before they read the rate, and to whoever notices that their hosts have quietly started buying something different from what they have spent a decade selling them.