Robots now send more requests to the average website than humans do — and a fast-growing slice of that traffic is AI crawlers deciding whether your content is worth training on, citing, or ignoring completely.
OpenAI publishes its own GPTBot documentation, which is worth bookmarking alongside your robots.txt rules.
Once these crawlers can reach your content, the next step is making it citable, which is what our Perplexity SEO services and ChatGPT SEO services focus on.
Quick answer: Bots accounted for 57.5% of HTML web traffic in June 2026, overtaking human visits for the first time. AI crawlers make up roughly 26.7% of that bot traffic, but they are not one thing: training crawlers like GPTBot and ClaudeBot scrape at massive scale for a tiny referral payoff (ClaudeBot’s crawl-to-referral ratio runs as high as 23,951:1), while answer bots like OAI-SearchBot and PerplexityBot send real, high-converting visitors. Blocking every AI bot in robots.txt protects your content, but it can also cut you out of AI search entirely.
AI crawlers 2026: the year bot traffic passed human traffic
As of June 2026, automated requests made up 57.5% of HTML traffic across the web, against 42.5% from human visitors — the first time bots have outnumbered people at this scale. Within that verified bot traffic, AI crawlers account for about 20.3%, with AI-search bots adding another 6.5%, for a combined AI-related share of roughly 26.7%. That is no longer a rounding error in your server logs; it is a meaningful share of every request your site receives, and it is growing every quarter.
Training bots vs. answer bots — they are not the same crawler
The biggest mistake in AI crawler management is treating “AI bot” as a single category. Data on crawler purpose shows 51.8% of AI crawler requests exist purely to build training data, 35.7% serve mixed purposes, and only 9.3% is genuine search-related retrieval done to answer a live user query. Those are functionally different visitors:
- GPTBot (OpenAI) — crawls to train future models. No live user is waiting on the other end.
- ClaudeBot (Anthropic) — same training purpose, and currently the least efficient crawler in terms of referral value.
- Google-Extended — feeds Gemini and AI training separately from classic Googlebot indexing.
- OAI-SearchBot and ChatGPT-User — fetch pages in real time to answer or browse for an actual ChatGPT user right now.
- PerplexityBot — retrieval for live Perplexity answers, with a much better referral payoff than the training crawlers.
The crawl-to-referral ratio problem
| Crawler | Crawl-to-referral ratio |
|---|---|
| Google (benchmark) | 4.9:1 |
| PerplexityBot | 111:1 |
| GPTBot | 1,276:1 |
| ClaudeBot | 11,122:1 to 23,951:1 |
Google sends you roughly one visitor for every 4.9 pages it crawls. ClaudeBot, at its worst, crawls upward of 23,000 pages for every single visitor it sends back. That gap is the entire training-versus-answering divide expressed in one number, and it is why so many publishers now treat these crawlers completely differently in robots.txt.
What publishers are actually doing about it
A September 2026 analysis of robots.txt files across Cloudflare’s network found a clear, deliberate pattern: publishers are not blocking “AI” as a category. They are blocking training and allowing answering.
| Bot | Block-to-allow ratio | Type |
|---|---|---|
| Bytespider | 5.80:1 | Training |
| ClaudeBot | 2.39:1 | Training |
| GPTBot | 2.33:1 | Training |
| Google-Extended | 2.14:1 | Training |
| ChatGPT-User | 1.12:1 | Answering |
| OAI-SearchBot | 0.94:1 | Answering |
Every training bot above sits above a 2:1 block ratio; every answer bot sits at or below 1.12:1. The report’s own conclusion is worth repeating: “there is no overlap” between the two groups. Publishers are making a deliberate, informed distinction rather than issuing a blanket AI ban — and by August 2025, more than 2.5 million sites had already fully disallowed AI training crawlers, with GPTBot alone appearing in 5.52% of all robots.txt DISALLOW rules, the single most-blocked AI crawler on the web.
Why blocking everything can cost you real revenue
Before you block on principle, look at what AI referral traffic is actually worth. AI-sourced visits to US retail sites grew 393% year-over-year in Q1 2026, and those visitors converted 42% better than non-AI traffic and spent 48% longer on-site. A blanket block that catches PerplexityBot, OAI-SearchBot, and ChatGPT-User alongside the training crawlers doesn’t just protect your content — it forfeits some of the best-converting traffic your site can get.
How to set up your robots.txt for the AI era
- Split your bot list into two groups — training crawlers (GPTBot, ClaudeBot, Google-Extended, Bytespider) and answer/search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot).
- Decide your training policy first. If you don’t want your content used to train foundation models, disallow the training group explicitly by user-agent — a generic wildcard rule won’t distinguish them.
- Keep the answer bots allowed if AI citations and AI-referred traffic matter to your business, since these are the crawlers that translate directly into visits and conversions.
- Monitor server logs, not just robots.txt. Rules only work if bots respect them, and log analysis is the only way to confirm which crawlers are actually hitting your pages.
- Revisit quarterly. New crawlers appear, existing ones change behavior, and 2026’s fast pace of AI product launches means last quarter’s robots.txt is already out of date.
Frequently asked questions
GPTBot crawls purely to train OpenAI’s future models — no live user is behind that request. ChatGPT-User and OAI-SearchBot fetch pages in real time to answer or browse for an actual ChatGPT user, and generally are not used for training. Most sites that want AI visibility without AI training data block GPTBot while allowing the other two.
That depends on how you feel about Gemini and AI Overviews training on your content, since Google-Extended is separate from the classic Googlebot that powers Search indexing. Blocking it will not hurt your regular Google rankings, which is exactly why 2.14 sites block it for every one that allows it.
Blocking training bots like GPTBot or ClaudeBot has no effect on classic Google Search rankings, since Googlebot is a separate crawler. But blocking answer bots such as OAI-SearchBot or PerplexityBot can remove you from citations in ChatGPT and Perplexity answers, which is a real visibility cost given how much AI referral traffic has grown.
Robots.txt only tells you what you’ve asked bots to do, not what they’re actually doing. Server log analysis (or a CDN-level bot report, if you’re on Cloudflare or similar) is the only reliable way to see real crawl volume by user-agent, and it’s worth checking monthly since new crawlers appear constantly.
No. Robots.txt is a decades-old, widely-respected standard for controlling crawler access. llms.txt is a newer, unproven proposal for describing your site to language models, and compliance across AI providers is still inconsistent — it’s worth understanding, but it doesn’t replace a properly configured robots.txt.
Know exactly which bots are crawling your site
Most sites are either blocking AI indiscriminately or not managing it at all — and both mistakes cost visibility or revenue. Get a free website audit and we’ll show you exactly which AI crawlers are hitting your site, which ones are worth allowing, and how to rewrite your robots.txt without accidentally shutting the door on AI search. Ready to get this right? Talk to our team and we’ll take it from there.
Related Reading
- Schema Markup in 2026: The Structured Data AI Search Engines Actually Use
- How to Optimise Your Website for ChatGPT: The Complete 2026 Guide
- How to Track AI Referral Traffic in GA4
Riya Bhardwaj
Lead Marketing Strategist
Leading content and growth initiatives with a focus on search visibility, audience engagement, and measurable business outcomes. Specialised in SEO, Generative Engine Optimization (GEO), AI search optimisation, and performance-driven content marketing. Passionate about transforming market insights into scalable content strategies that strengthen brand authority and drive sustainable digital growth.