Verified, then logged.
Sun 22 Nov 2026 · 00:21 KST / 15:21 UTC한국어 | ENSearch /

01 / AI · Analysis

Per-page AI crawler fees would keep sites free for people

A DEV.to post proposes sharing the cost of delivering public content with the AI crawlers that collect it.

An unbranded laptop showing an unreadable article layout sits in front of web servers and dense network cabling in a small publisher’s server room. AI-generated image

A post on DEV.to, a developer publishing platform, proposes charging AI crawlers for each page they fetch while keeping access free for people. The proposal separates access for human readers from automated content collection.

The post argues that allowing crawlers in for free leaves website operators paying for bandwidth while the crawlers obtain the content. The issue it raises is who should bear the cost of delivering publicly available pages.

Its proposed arrangement preserves human access while asking AI crawlers to pay. That shifts the discussion beyond whether content is freely available to whether different types of visitors should bear different costs.

Who pays for free crawling

The post presents free crawler access and blocking through robots.txt, a file that tells crawlers which pages they may access, as the existing choices. Charging per page would add an option between allowing free collection and blocking it.

According to the post, AI crawlers fetch webpages to train models or answer questions. The proposed charges therefore target automated content collection, while people would continue to read pages for free.

Per-page charges as an alternative to blocking

The post argues that blocking crawlers through robots.txt saves bandwidth but causes a site to disappear from AI answers. This is its stated trade-off: blocking collection may reduce delivery costs while also reducing the site's presence in those answers.

The proposal focuses on distinguishing visitors who retain free access from crawlers asked to pay. Its central argument is that those collecting public content should share the cost of delivering it, with per-page charges offered as an alternative to blocking collection.