· 7 min read
Dutch news sites block ChatGPT. Google's AI answers they cannot block
We read the robots.txt of 40 Dutch websites. Ten block Google's AI training token, none blocks Googlebot or limits snippets, the two controls that govern AI Overviews.
Written with the help of AI, reviewed and added to by Kevin Bos.
Ten of the 31 Dutch websites whose robots.txt we could read tell Google’s AI training crawler to stay out. Not one of them blocks the crawler that feeds the AI answers above Google’s search results:
Google-Extended: fully blocked on 10 of 31, named in 13
Googlebot: fully blocked on 0 of 31, named in 0
[…]
home pages that limit the snippet: 0 of 29
The last line matters as much as the first two. Googlebot and the snippet directives are the only controls Google offers for its AI Overviews. The sites that built the longest block lists have left both untouched, and they have good reason to. That is the trap this article is about.
0 of 31Dutch sites in our survey that block the crawler behind Google's AI answersOur survey in the sandbox, 18 November 2026
Why this is on the front page
Two pieces about the same loss were on Hacker News on 18 November. RevScope’s analysis (382 points) collects the studies: in a randomised experiment by Saharsh Agarwal and Ananya Sen, people who saw AI Overviews made 0.37 clicks to outside websites per search against 0.62 for people who had them hidden, a reduction of 39.8%. We read RevScope’s account of that working paper, not the paper. Pew Research Center, which we did read, logged the browsing of 900 US adults in March 2025: a click on a normal result in 8% of visits to a page with an AI summary, 15% without one, and a click on a link inside the summary in 1%.
The second piece is one site’s bill. Nicolas Bouliane (320 points) has run All About Berlin since 2017 and lives from it. He writes that it has lost 75% of its traffic to AI Overviews.
So we wanted to know what site owners close to home are doing about it, and whether it can work.
What we did
A script fetched three things from 40 well-known Dutch websites, one site at a time: robots.txt, llms.txt and the home page. The list is our own choice: 15 news and trade titles, 10 shops and marketplaces, 4 banks and insurers, and 11 public and reference sites. Nine sites refused the request from a data-centre address, so the counts are over 31 robots.txt files and 29 home pages. The script and its full output, with a row per site, are in this site’s repository.
Three controls, used very differently
- Block CCBot (Common Crawl)
- 11 sites
- Block GPTBot (OpenAI training)
- 10 sites
- Block ClaudeBot
- 10 sites
- Block Google-Extended (Gemini training)
- 10 sites
- Block OAI-SearchBot (ChatGPT search)
- 8 sites
- Block Googlebot (Search, AI Overviews)
- 0 sites
- Limit the snippet on the home page
- 0 sites
The blockers are almost all publishers. Nine sites block GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot together: NOS, NU.nl, De Telegraaf, AD, de Volkskrant, Trouw, Het Parool, RTL and Het Financieele Dagblad. The government sites, Tweakers and Wikipedia name none of them. MediaMarkt names them in order to allow them, and blocks only CCBot. NU.nl’s file addresses 66 user agents and NRC’s 194.
| Control | What the vendor says it governs | Sites using it |
|---|---|---|
GPTBot |
Training of OpenAI’s models | 10 of 31 block |
OAI-SearchBot |
Appearing in ChatGPT’s search answers | 8 of 31 block |
Google-Extended |
Gemini training and grounding outside Search | 10 of 31 block |
Googlebot |
Google Search, including its AI features | 0 of 31 block |
nosnippet, max-snippet |
What Google may show from a page, AI features included | 0 of 29 limit |
llms.txt |
Nothing that anyone has measured | 2 of 40 have one |
Why the block list misses the product that takes the clicks
Google’s documentation for site owners is plain about it: “AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners”. To limit what is shown, it points to nosnippet, data-nosnippet, max-snippet or noindex. About the token that ten of these sites block, the crawler list says: “Google-Extended does not impact a site’s inclusion in Google Search”. It is not even a crawler. It has no user agent of its own and only tells Google what it may do with pages Googlebot has already fetched.
So the only switch for AI Overviews is the switch for Google itself. Blocking Googlebot removes a site from Search. A snippet limit shortens the normal result as well, and Google’s page says a page must be “eligible to be shown in Google Search with a snippet” to appear as a supporting link in an AI answer at all. A publisher who limits snippets gives up the citation and keeps the summary of everyone else’s reporting on the same news.
The home pages show which way publishers lean. Six carry a snippet directive and none of them limits anything. We looked at two by hand, de Volkskrant and RTL: both say max-snippet:-1, which means no limit. The same sites that list every AI crawler by name tell Google to quote as much as it likes. For a site that depends on visitors from search, that is a rational choice. It is also the opposite of a block.
The crawler worth a second look
OpenAI documents two separate tokens. GPTBot collects training data. OAI-SearchBot decides whether a site can appear in ChatGPT’s search answers, and “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”. The first costs you nothing to block. The second is the one that can send a reader.
Eight of the 10 sites that block GPTBot also block OAI-SearchBot. NOS and BNR do not: their files name the training crawler and leave the search crawler alone. Whether the other eight chose that or copied a list, we cannot tell from a file.
If the goal is “do not train on us, do send us readers”, the file looks like this. It follows the vendors’ documentation; we did not test its effect:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
What we do on this site, and why it differs
Our own robots.txt names twelve AI crawlers and allows every one, and the site publishes an llms.txt and a Markdown version of each page. That is the reverse of the news sites, and both can be right. A publisher sells the visit. We sell work, and a company that is quoted correctly in an answer has lost nothing when the reader never clicks.
We should be honest about llms.txt, since we have one. Google says “you don’t need to create new machine readable files, AI text files, or markup” for its AI features, and RevScope cites two studies that found no relation between the file and being cited. Two of the 40 sites in our survey have one. We keep ours because agents that visit can read it, and we would not sell it to anyone as a way to be found.
What we would do
Sort the crawlers by what they return, and decide per row.
| If your site lives from | Training crawlers | Search and answer crawlers | Snippet limit |
|---|---|---|---|
| Page views and subscriptions | Block | Allow, and measure what they send | No: it costs normal results too |
| Leads and reputation | Your call | Allow | No |
| Paid content behind a login | Block | Block | data-nosnippet on the paid parts |
Then check the file against the vendors’ pages once a quarter. Tokens are added faster than block lists are updated: 10 sites in our survey block ClaudeBot and 7 block Claude-SearchBot.
What would change our view: a control from Google that separates AI Overviews from the rest of Search. Google’s crawler page announces support for a shared training-preference signal in early 2027. That is about training again, and not about answers.
Limits of this survey
Forty sites that we picked, on one day, from one data-centre address. Nine did not answer, and a refusal can itself be a bot block that never shows up in robots.txt. We read home pages only, so a site may limit snippets on its articles and not on its front page. robots.txt is a request that a crawler may ignore, and we did not measure whether any crawler obeys it. We counted a crawler as blocked only when its own group, or the wildcard group, disallows the whole site.
Sources
- Google Search Central: AI features and your website
- Google: list of common crawlers, including Google-Extended
- OpenAI: overview of OpenAI crawlers
- RevScope: Google's AI Overviews cut clicks to websites by 40%
- Pew Research Center: Google users are less likely to click on links when an AI summary appears
- Nicolas Bouliane: AI is killing All About Berlin
- AI search
- robots.txt
- SEO
- AI answers