Allow or block AI crawlers? Weighing it up
The decision usually gets argued as a matter of principle and is a business question: if you earn from content, allowing costs you. If you win projects through content, blocking costs you — and more.
In short
- There are two kinds of request: advance collectors and runtime fetches. They can be controlled separately — and that is where the most usable solution lies.
- If you sell projects, allow the runtime fetches in any case: they produce mentions in front of people mid-way through looking for a solution.
- If you sell content or earn from ad impressions, there are good reasons to block the collectors.
- robots.txt is a request, not a block. Enforcing it technically requires a rule at server level.
The two kinds of request
Advance collectors
Fetch content independently of any specific question — for training or for a search index of their own. Examples: GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot.
Benefit to you: indirect — your content sits available in an index. Cost: your content gets used without anyone visiting your page.
Runtime fetches
Fetch a page at the moment someone has asked a matching question. Examples: OAI-SearchBot, ChatGPT-User, Perplexity-User, Claude-User.
Benefit to you: direct — mentions and links in specific answers come from these. Cost: practically none.
Worth knowing
Blocking "all AI bots" blanket blocks both kinds — and thereby loses exactly the part that benefits you. Being named as a source in an answer requires the page to be fetchable at runtime.
For a company selling projects that is the worst possible trade: you prevent a recommendation reaching someone mid-way through looking for a solution, in order to prevent a model learning from a publicly accessible article.
Three strategies
| Strategy | Collectors | Runtime fetches | Suits |
|---|---|---|---|
| Open | allow | allow | service businesses, B2B, consulting |
| Mixed | block | allow | publishers, course providers, trade media |
| Closed | block | block | paid content, protected areas |
For most small and mid-sized companies, "open" is the right choice. The mixed strategy is the sensible middle way for anyone whose content is itself the product.
The robots.txt
For the open strategy with explicit naming — not strictly necessary, but it makes the decision visible and documents it:
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
For the mixed strategy, the collectors get blocked and the runtime fetches allowed:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-User Allow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Google-Extended controls only the use of content for Google's AI products — not ordinary search indexing. Block Googlebot by accident and you disappear from search. Those two names are different things and get confused regularly.
What robots.txt does not do
Three limitations worth knowing before relying on the file:
- It is a request. Reputable providers observe it, others do not. Enforcing it technically requires a block at server or firewall level based on the user agent.
- It does not work retroactively. What has already been collected does not disappear because of it.
- It does not protect against reproduction from other sources. If your content gets quoted elsewhere, your own file does not help.
Before making the decision, it is worth looking at the server logs: which of these identifiers appear at all, how often, and which pages do they fetch?
On small websites the answer is often sobering — few requests, few pages. The question is then practically secondary, and the actual task is a different one: getting found at all. A block against crawlers that are not coming solves no problem and merely prevents that changing later.
The legal frame
In the EU a text and data mining reservation exists: rights holders can expressly prohibit the use of their content for automated analysis, and that reservation has to be declared machine-readably. The robots.txt is the common form for that, sometimes supplemented by a note in the terms of use.
It does not apply directly in Switzerland; anyone also addressing EU users should nonetheless declare the reservation if they want it. That is a point for a legal professional to assess — this piece does not replace legal advice.
In closing
The question is not one of principle but hangs on the business model. If you sell content, block the collectors. If you sell projects, allow both — and gain mentions in front of people currently looking for a solution.
In either case: consider the runtime fetches separately. Blocking them along with the rest is the most common and most expensive mistake in this decision.
Common questions
Should you block AI crawlers?
It depends on the business model. Anyone earning from content itself — publishers, course providers, trade media — has good reasons to block the advance collectors. Anyone selling projects or services loses more by blocking than they gain.
What is the difference between collectors and runtime fetches?
Collectors such as GPTBot, ClaudeBot or PerplexityBot fetch content independently of any specific question, for training or an index of their own. Runtime fetches such as ChatGPT-User, Perplexity-User or OAI-SearchBot fetch a page exactly when someone has asked a matching question — and mentions and links come from those.
What is Google-Extended?
An identifier controlling the use of content for Google's AI products — not ordinary search indexing. It is a different thing from Googlebot; block that one by accident and you disappear from search.
Does every crawler observe robots.txt?
No. It is a request, not a technical block. Reputable providers observe it, others do not. Enforcing a block requires a rule at server or firewall level based on the user agent — and even then it does not work retroactively on content already collected.
How do you check which crawlers are even coming?
Through the server logs, filtered by the known identifiers. That shows which crawlers fetch which pages, how often. On small websites the result is often modest — the blocking question is then secondary to the task of getting found at all.
Marketing that sets itself up
The Studio Engine beta is live. Claim your spot and help shape it from the start.
Join the beta →