Skip to content
Etalo

Technical

robots.txt and llms.txt: letting AI read your site

It is the one generative-visibility job that can be settled in an afternoon, and the one that makes every other job pointless if you get it wrong. Three files, a dozen agents, and a trap almost nobody sees: the most expensive block is not in robots.txt.

Founder of Etalo, SEO and generative visibility consultant

Read on 16 September 2026. Every study cited carries its date in the text.

The essentials in thirty seconds

Letting AI read your site is checked in three places: robots.txt, which declares your intentions agent by agent; your firewall or bot-protection service, which can refuse a crawler that robots.txt allows; and how your pages render, since content appearing only after JavaScript runs may never be seen. The llms.txt file costs one page to write but is read by no AI system to date.

  • Agents are not equivalent: blocking GPTBot refuses training, blocking OAI-SearchBot removes you from the pages ChatGPT can cite. Confusing the two is the most common mistake, and the quietest.
  • The market already draws that line: 17.6% of the top 10,000 sites block GPTBot, against 6.6% for OAI-SearchBot (June 2026 reading across 7,870 readable sites).
  • One site in five could not be read at all in that survey — bot protection, timeout or error. That is the real blind spot: a spotless robots.txt behind a firewall answering 403.
  • robots.txt is a sign, not a lock: in August 2025 Cloudflare documented undeclared crawlers attributed to Perplexity, up to 3-6 million requests a day across tens of thousands of domains.
  • llms.txt: "purely speculative for now", said Google in June 2026, "the file has existed for years, yet none of the AI systems use it". Publishing one costs an hour; presenting it as a solution would be dishonest.

robots.txt is a sign, not a lock

Start with what the file does not do, because it decides everything else. robots.txt stops nobody from reaching a page: it declares an intention that well-behaved crawlers follow. A crawler that decides to ignore it ignores it, and the server never notices.

This is not a theoretical worry. In August 2025 Cloudflare published an investigation attributing undeclared crawlers to Perplexity: an ordinary Chrome browser identifier in place of the official agent, IP addresses outside the published ranges, changing source networks. Cloudflare describes 3 to 6 million requests a day for that disguised agent alone, across tens of thousands of domains, including brand-new test domains whose robots.txt explicitly forbade reading.

The practical consequence cuts both ways. If you want to prevent reading, robots.txt is not enough: you need a server rule. And if instead you want to be read — the subject of this article — then robots.txt is only the first of three doors to open.

The agents that matter, and what blocking each one does

Ten agents, three families: those collecting for training, those feeding an answer index, and those fetching a page because a user just asked for it. Blocking them has entirely different consequences.

AgentVendorFamilyBlocking it meansWeight
GPTBotOpenAITrainingRefusing to let your content train future models. No effect on ChatGPT Search citations.High
OAI-SearchBotOpenAIAnswer indexDropping out of the pages ChatGPT can cite. The most expensive block in this table, and the least often intended.High
ChatGPT-UserOpenAIOn demandPreventing the reading of a page a user explicitly asked ChatGPT to go and look at.Medium
Google-ExtendedGoogleTrainingDropping out of the Gemini app and Vertex AI grounding. No effect on AI Overviews: Google treats AI as part of Search, so the only lever is Googlebot — and the price is your search ranking.High
ClaudeBotAnthropicRetrievalPreventing Claude from reading your pages to answer.High
Claude-WebAnthropicBrowsingClosing browsing triggered from a conversation.Medium
PerplexityBotPerplexityAnswer indexAsking Perplexity not to index your pages — a request its declared crawler honours; see the Cloudflare investigation above for the rest.Medium
CCBotCommon CrawlTrainingDropping out of the public corpus feeding a large share of models, including those that do not exist yet.Medium
Applebot-ExtendedAppleTrainingRefusing Apple Intelligence training.Low
BytespiderByteDanceTrainingRefusing ByteDance collection.Low

The "weight" column is the one Etalo applies in its AI Score: it follows each assistant's real audience, not the number of lines in the file. Blocking GPTBot does not cost the same as blocking Bytespider.

What the market actually does

Share of the top 10,000 domains disallowing each agent at the root of their site. June 2026 reading across the 7,870 sites whose robots.txt could be read and parsed the way a compliant crawler would.

CCBot — Common Crawl 17.8%
GPTBot — training 17.6%
ClaudeBot 16.0%
Google-Extended 14.5%
PerplexityBot 10.4%
OAI-SearchBot — answers 6.6%
Source: survey of the top 10,000 Majestic Million domains, June 2026. The author publishes the method and the number of sites excluded, which is why this figure is cited here rather than another.

What that ranking tells you

The gap between OpenAI's two agents is the most telling result: 17.6% block GPTBot, only 6.6% block OAI-SearchBot. In plain terms: most sites have understood the distinction and act on it — refuse training, accept being cited.

The three most-blocked agents (CCBot, GPTBot, ClaudeBot) are the ones associated with collecting for training. The least blocked is the one feeding an answer index. That is a coherent position, and probably the one you want to hold: do not feed a model for nothing, but stay citable.

The trap is in the writing. A Disallow: / under User-agent: * applies to every agent you have not named — so to OAI-SearchBot if you did not plan for it. And the reverse holds: as soon as an agent is named in a group, the * group is entirely ignored for it, including its permissions.

Three figures worth keeping

21%

of the top 10,000 sites returned no readable robots.txt during the survey: 2,130 excluded for bot protection, timeout or error. The study sets them aside; for an audit they are the interesting ones.

June 2026 reading

17.6 / 6.6%

the share of sites blocking GPTBot against OAI-SearchBot. Two agents from the same vendor, two opposite consequences, almost a threefold gap.

June 2026 reading

70,900 to 1

pages read by Anthropic's crawlers per visit sent back, late June 2025. Cloudflare notes that visits from the Claude app carry no referer: the real ratio is therefore more favourable, by an unknown amount.

Cloudflare Radar, June 2025

The block nobody decided

Here is the case most often found in an audit, and the one robots.txt will never show: the file allows everyone, and the server refuses anyway. A bot-protection service, a web application firewall rule, a protection switched on by a supplier two years earlier — and AI agents leave with a 403 or a 429.

Nobody in the company decided it, nobody sees it, and no interface reports it. The only way to observe it is to request the same page twice, once with an ordinary browser identifier and once with GPTBot's real identifier, and compare the two response codes. If the browser gets through and GPTBot is refused, your content is not the subject.

Etalo runs this check systematically, and it carries six points of the AI Score's Access axis on its own — more than any single agent, because a firewall that filters never filters just one crawler.

Check your site in ten minutes

Six checks, in order. The commands can be typed as they are, replacing the domain.

01

Read the file

Type curl -s https://yoursite.com/robots.txt and look for the names in the table above. If none appears, only the * group applies to them — read it carefully.

02

Check it is not an HTML page

Many servers answer 200 with the home page when the file does not exist. curl -sI https://yoursite.com/robots.txt should show content-type: text/plain — an HTML page is worth a missing file.

03

Compare browser and crawler

Two requests, two identifiers, one number to compare: the response code. This is the firewall check described above, and it reveals more unintended blocks than anything else.

04

Look at the page without JavaScript

curl -s https://yoursite.com/ | wc -c, then search it for a sentence you know is on the page. If it is not there, your content only exists after a script runs — and many agents run none.

05

Hunt down forgotten directives

A noindex or noai tag left on a template, an X-Robots-Tag header added for a staging site and never removed: both show up in the response headers and in the page source.

06

Declare the sitemap

One Sitemap: https://yoursite.com/sitemap.xml line in robots.txt. Without it a crawler is left guessing the address, and often only tries the convention.

The rules crawlers actually follow

The robots exclusion protocol has been standardised since 2022 by RFC 9309, and its four rules explain most robots.txt files that do not do what their author believed.

Only one group applies. The crawler looks for the group carrying its name, case-insensitively, and obeys that one only (section 2.2.1). The * group applies only if no named group exists for it. Writing a general permission under * therefore protects nothing if the agent is named further down with a Disallow: /.

The longest rule wins. "The most specific match found MUST be used", says section 2.2.2: specificity is counted in characters of the path, not in the order lines were written.

On a tie, allow beats disallow, and an empty Disallow allows everything: if no rule in the group matches, the URI is allowed. That is exactly what robots_verdict() does inside Etalo — a parser following line order instead of these rules would return wrong verdicts on perfectly valid files.

llms.txt: what is known, and what is not

The llms.txt file is a proposal from late 2024: a Markdown summary at the root of the site, listing the pages describing what you offer, sparing models the crossing of a cluttered site. The idea is sound. Adoption has not followed.

In June 2026 John Mueller, of Google, called it "purely speculative for now" and added that the file has existed for years without any AI system using it. His recommendation fits in one sentence: the day a platform that brings you customers asks for the file, take the time to write it.

So the honest position is this: it costs one page, the risk is nil, the benefit is unproven. Etalo checks for it and grants it three points on the Discovery axis — ten times less than rendering content without JavaScript. A supplier selling you llms.txt as the key to generative visibility is selling a convention nobody has yet confirmed reading.

Five mistakes found everywhere

All observed on production sites, all fixable in under an hour.

Blocking OAI-SearchBot believing you are blocking training
Both agents carry the same vendor name and do different jobs. It is the most expensive mistake, and nothing flags it: your site works perfectly, it has simply stopped being citable.
Believing Google-Extended closes AI Overviews
This is the mistake we made ourselves in a first version of this article. Google-Extended covers the Gemini app and Vertex AI grounding; it does not touch AI Overviews. Google said so back in 2023: "AI is built into Search, not bolted on, and integral to how Search functions, which is why robots.txt is the control". In other words, leaving AI Overviews means leaving Search.
A spotless robots.txt behind a refusing firewall
The file says yes, the server says 403. An audit stopping at reading the file concludes all is well.
A robots.txt that does not exist but answers 200
The server returns the home page instead of a 404. Crawlers receive HTML where they expect text, and many treat the case as a missing file — sometimes with consequences you did not choose.
Disallowing a directory holding your proof
Case studies, pricing and documentation often sit in directories closed by inheritance — a Disallow: /docs/ written back when the site was a staging build. Those are exactly the pages an assistant would cite.

Frequently asked questions

Does blocking AI crawlers cost me Google rankings?
It depends on the agent, and Google is the special case. Google-Extended covers the Gemini app and Vertex AI grounding: blocking it does not touch classic search. But it does not close AI Overviews either — for those, the only documented lever is Googlebot, which is search itself. There is no way out of AI Overviews that is not a way out of Search.
Should I allow every AI agent?
Not necessarily. The most widespread and most defensible position is to refuse training agents and allow the ones feeding answers — which is what most large sites already do.
Does robots.txt really prevent reading?
No. It is a declaration of intent that well-behaved crawlers follow. To actually prevent access you need a server rule or authentication. Cloudflare's August 2025 investigation into undeclared crawlers is the reminder.
Should I create an llms.txt?
If you have an hour, yes: the risk is nil and the convention may catch on. But no AI system reads it today, and making it a priority ahead of rendering without JavaScript would be a poor trade.
How do I know whether an agent visited me?
In the server logs, searching for the agent identifiers in the table above. Careful: an identifier can be faked. The major vendors publish the IP ranges of their crawlers, and that is the only way to verify a visit is genuine.
What if my site sits behind Cloudflare or an equivalent?
Then the setting that matters is there, not in robots.txt. Check the list of allowed agents in their interface — several services have blocked AI crawlers by default since 2025.
  • The business law firms AI assistants cite most

    When a company asks ChatGPT or Google which law firm to call, which names come back? We put 26 questions to five assistants and counted the citations of seven…

  • Why my brand doesn't appear in ChatGPT

    You asked the question, and a competitor came out instead of you. Before rewriting anything, work out which of the four causes applies: they share neither the…

  • How much does a GEO audit cost

    Two of the three most-cited tools no longer show any price at all. Here are the ones still public, what the API calls behind them really cost, and why the gap…

  • AI prompt volume does not exist

    Every AI visibility tool ends up printing a number next to each question: "2,400 searches a month". Nobody measures that. Here is where the figure actually…

  • Getting into Google AI Overviews

    This is the generative surface most people see without ever looking for it: it sits above the results, in front of users who have never opened an assistant…

  • When AI gets you wrong

    A brand that is highly visible in assistant answers, with a wrong price, does not have a visibility problem. It is the one case where a good score is worse…

  • Building your question corpus

    It is the part of a GEO audit nobody wants to do, and it decides everything else. A badly written corpus produces impeccable figures about questions nobody…

  • GEO audit: what it measures, how, and what makes an audit wrong

    Public figures on generative search contradict each other: AI Overviews appear on "more than 20%" of searches by one measurement, on 75.24% by another. Both…

  • How to get cited by ChatGPT

    There is no button. There is a technical door that many sites close without knowing it, three editorial levers that have actually been measured, and a source…

  • GEO or SEO: what really changes

    Three serious studies measured the same thing — the share of AI citations coming from pages that rank well on Google. They found 16.7%, 52% and a correlation…

  • AI visibility tracking tools in 2026

    Seven tools, read on the same day from their public pages. Two publish no price at all. And once you divide published prices by the number of answers actually…

  • Etalo.ai review

    You are looking for reviews of Etalo and you cannot find any. That is normal: the tool is young and has no clients to parade. Rather than manufacture…

  • Best Peec.ai alternative

    Peec.ai does one thing well: it tracks your visibility in assistant answers, daily, in a dashboard that is easy to read. If that is what you need, it is a good…

  • Best Promptwatch alternative

    Promptwatch has one real strength few tools have: crawler logs, that is, what AI robots actually come and read on your site. If that is your subject, keep it…

Sources

robots.txt interpretation rules: RFC 9309, sections 2.2.1 and 2.2.2.

Scope of Google-Extended: Google documentation on AI features, and Google's own statement reported on 9 October 2023: Google-Extended does not cover AI Overviews.

OpenAI crawlers: OpenAI documentation, read on 16 September 2026.

Block rates per agent: analysis of the top 10,000 Majestic Million domains, June 2026, 7,870 usable sites out of 10,000.

Undeclared crawlers: Cloudflare, 4 August 2025. Ratio of pages read to visits sent back: Cloudflare Radar, week of 19 to 26 June 2025.

Google's position on llms.txt: John Mueller quoted by Search Engine Journal, June 2026.

Have the three doors checked

Etalo's AI Score runs these checks on your domain: agents, firewall, rendering without JavaScript, forgotten directives. Ten minutes, and you know what assistants can read on your site.