90% of What Your AI Agent Scrapes Is Wasted Tokens

A typical web page runs about 80,000 tokens once a model reads the raw HTML behind it. Over 90% of those tokens are CSS, JavaScript, and comments the model will never quote. Most AI agents are still paying for all of it, on every page they touch, for a reader who never asked for the navigation bar, the cookie banner, or the CSS.

That gap is not about which scraper an agent uses to read someone else's site. It is about what your own site serves at the origin, before any agent, proxy, or scraper gets involved. Every website on the internet was built for one reader: a human, with eyes, holding a mouse. That reader is no longer alone, and the second one does not read the way the first one does.

The traffic no one designed for

In 2025, AI bots other than Googlebot accounted for 4.2% of all HTML requests across Cloudflare's network, nearly matching Googlebot's own 4.5% share. That number alone is easy to dismiss as a rounding error. The growth rate is not. Cloudflare's 2025 Radar review recorded "user action" crawling, the category triggered when someone asks ChatGPT or a similar assistant a question and the model fetches a live page to answer it, growing more than 15 times over in a single year.

OpenAI's ChatGPT-User bot alone hit request volumes 16 times higher in December than it had in January. That is not a niche automation running in the background. It is a new reader showing up at the front door of every website at a pace no traffic forecast accounted for two years ago.

This reader does not browse. It requests a URL, receives a document, and has a fixed token budget to make sense of it before the rest of the conversation even starts. Every kilobyte of navigation markup is a kilobyte the model cannot spend on the sentence that actually answers the question it was asked.

What HTML actually costs a model

The token math is not a minor efficiency complaint. A study formalizing the case for structured retrieval, published as the HtmlRAG paper, measured that raw web HTML averages roughly 80,000 tokens per page, and over 90% of those tokens are CSS, JavaScript, comments, or other content with no semantic value. That is not an edge case. That is the median web page, the one every retrieval pipeline built on a generic scraper is currently feeding into a model's context window.

A model reading a raw web page is spending nine tokens out of every ten on markup it will never quote, never cite, and never reason about.

Anthropic's own web fetch tool documentation puts rough numbers on the same problem from the consumption side: a typical 10-kilobyte web page runs about 2,500 tokens once fetched, and a 100-kilobyte documentation page runs closer to 25,000. The tool now ships with dynamic filtering, letting Claude write code to strip content before it ever reaches the context window. That feature exists because the supply side never cleaned up the page in the first place.

Every dollar spent building a filtering step on the consumption side is a dollar spent compensating for a publishing decision made years before any of this mattered. The fix was always cheaper upstream.

The fix is architectural, not a better scraper

The instinct inside most engineering teams is to treat this as a tooling problem: swap scraping libraries, add a cleaning regex, tune the chunker. That misdiagnoses the layer where the decision actually lives. The question is not which library extracts text from a page. It is what a URL should return in the first place, and to whom.

Jina AI's open-source Reader project takes this approach. Prefix any URL with r.jina.ai/ and the service returns clean markdown instead of raw HTML, choosing between a headless Chrome engine for JavaScript-heavy pages and a lightweight curl-based fetch for static ones. No API key or account is required. The project has drawn 11.7k stars on GitHub, one signal that a meaningful number of engineers are reaching for a purpose-built conversion layer rather than solving this from scratch.

The underlying conversion model, ReaderLM-v2, scores a ROUGE-L of 0.72 on held-out HTML-to-markdown benchmarks and outperforms both GPT-4o and Gemini-1.5 at the same task, while running as a 1.5-billion-parameter model small enough to deploy cheaply at the edge. A dedicated, purpose-built conversion layer beats asking a general-purpose scraper, or a general-purpose frontier model, to do cleanup work it was never optimized for.

Both endpoints can be called directly, without any setup:

# Read: any URL becomes clean markdown
curl https://r.jina.ai/https://example.com

# Search: query the web, get markdown-formatted results back
curl "https://s.jina.ai/what is model context protocol"

The same conversion layer can also run self-hosted instead of calling the hosted endpoint:

git clone https://github.com/jina-ai/reader.git
cd reader
docker compose up

A conversion layer built for one job outperforms a frontier model asked to do it as a side effect.

Why "just add a file" already failed

The web has already tried a lighter-weight fix. In September 2024, Jeremy Howard proposed /llms.txt, a markdown file at the root of a domain that would give models a curated index of a site's most important content. The founding logic was sound: "language models generally like to have information in a more concise form" than a rendered page provides, and a site owner is better positioned than a crawler to say what matters.

It did not work. An analysis of nearly 300,000 domains found that just 10.13% had implemented an llms.txt file, and more importantly, no major AI provider, not OpenAI, not Anthropic, not Google, currently reads the file when their crawlers visit a page. Site owners who built one got nothing for the effort.

The failure is not the idea. It is the layer. llms.txt asked every publisher on the internet to perform a second, manually maintained act of curation with zero feedback loop telling them whether it worked. That is the same mistake as asking every site to hand-write a mobile stylesheet instead of building responsive design into the rendering pipeline itself. The fix that scales does not depend on publisher goodwill. It depends on infrastructure that produces the right output automatically, the way Reader's dual rendering engine does, without a single file anyone has to remember to update.

What technical executives should build instead

The stakes are not hypothetical. Gartner forecasts AI agent software spending will reach $206.5 billion in 2026 and jump to $376.3 billion in 2027, and separately estimates that $234 billion in existing enterprise application spend is now at risk from agentic AI displacing it. An organization getting agent-facing content wrong is not losing a rounding error. It is losing visibility into the fastest-growing category of software spend in the market.

The practical response does not require waiting for a standard to mature. It requires three decisions, made once, at the infrastructure layer:

• Generate a canonical markdown mirror for every page at build time, the same way a sitemap or RSS feed is generated automatically rather than hand-maintained.
• Serve that mirror on a dedicated agent-facing path or through content negotiation on the Accept header, so the decision happens at the server, not inside every consuming agent.
• Skip the manifest-file approach. A curated index that depends on a publisher remembering to update it will decay the same way llms.txt did.

The second point is the one teams skip because it sounds harder than it is. It is a routing decision, not a rewrite:

// middleware.ts: serve the markdown mirror to agents, HTML to everyone else
export function onRequest(context, next) {
  const accept = context.request.headers.get('accept') ?? '';
  const ua = context.request.headers.get('user-agent') ?? '';

  const wantsMarkdown =
    accept.includes('text/markdown') ||
    /GPTBot|ClaudeBot|PerplexityBot|ChatGPT-User/i.test(ua);

  if (wantsMarkdown) {
    const mirrorPath = context.url.pathname.replace(/\/$/, '') + '.md';
    return context.rewrite(mirrorPath);
  }

  return next();
}

The markdown mirror itself is generated once at build time, the same pass that already produces the HTML. No proxy, no third-party service, no publisher remembering to update a manifest. The decision happens once, at the server, and every agent that requests the page gets the format it actually wants.

None of this is exotic engineering. It is the same instinct that produced responsive design and RSS: build the second representation into the pipeline, do not ask every visitor to tolerate the first one. Teams that treat this as a content architecture decision, not a scraping inconvenience, will show up correctly formatted in every agent that fetches them. Teams that do not will keep paying for the 90% of every page that is markup, not meaning, multiplied across every request, for a reader that is already 4.2% of their traffic and growing 15 times a year.

Websites that only publish for eyes are already invisible to the fastest-growing category of visitor they have. The fix isn't a new file at the root of the domain. It's admitting the browser was never built for the reader you're getting now.

Sources

1. Cloudflare - Radar 2025 Year in Review - AI bot share of HTML traffic and user-action crawling growth data.
2. Anthropic - Web Fetch Tool Documentation - HTML-to-markdown conversion, dynamic filtering, and token sizing examples.
3. Jina AI - ReaderLM-v2: Frontier Small Language Model for HTML to Markdown and JSON - Benchmark accuracy data for HTML-to-markdown conversion.
4. Jina AI - jina-ai/reader GitHub Repository - Open-source URL-to-markdown service architecture and adoption.
5. Answer.AI - /llms.txt: A Proposal to Provide Information to Help LLMs Use Websites (2024) - Original llms.txt proposal by Jeremy Howard.
6. arXiv - HtmlRAG: HTML Is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems - Measurement of raw HTML token bloat and non-semantic content share.
7. ppc.land - llms.txt Adoption Stalls as Major AI Platforms Ignore Proposed Standard - Domain-sample analysis of llms.txt adoption rates.
8. Gartner - $234 Billion in Enterprise Application Software Spend Is at Risk From Agentic AI (2026) - Agentic AI software spending forecasts for 2026 and 2027.

Working through the challenges in this post? I help engineering leaders and CTOs navigate complex technical decisions and scale high-performing teams. Schedule a consultation →