I found a way around the AI crawler blocklist, then read how it works

Last time I found that 16 of the news organisations with the strongest fact-checking reputations refuse my agent’s crawler. Reuters, AP, WSJ, FT, the New York Times, the Guardian, the BBC and nine more. Telling the model to prefer credible sources does nothing when those sources were never in the pool.

That left a question I couldn’t answer at the time. Is that a property of crawlers in general, or just of the one I was using?

Search APIs keep their own indexes. If Tavily or Exa or Serper have Reuters, the blocklist is a plumbing problem and the fix is to stop crawling and start querying an index.

So I tested it. It works. And then I read the documentation and decided not to use it.

It works

Free Tavily key, one request per blocked domain, asking for full page content rather than snippets.

13 of the 16 come back with real article text. Several over 10,000 characters. Ars Technica returned 48,000. Reuters, AP, the New York Times, the Economist, the Atlantic, Politico, Wired, The Verge, the BBC, Business Insider, ZDNet and the FT all returned actual article bodies.

Three didn’t. WSJ gave me 1,295 characters of Dow Jones reprints boilerplate. The New Yorker gave a paywall stub. The Guardian returned something short that might be a genuinely short article, I didn’t check.

Those three matter, because they rule something out. If Tavily were running browsers with paid subscriptions, WSJ would not come back as reprints boilerplate. There is no paywall bypass here.

I got the first number wrong

Worth saying, because it is exactly the mistake my tool is supposed to catch.

On the first pass all 16 domains returned results, all with text, and I wrote down “16/16”. Then I looked at what the text actually was. The default content field is a snippet of about 150 characters. Roughly what you see under a link on a search results page.

You cannot verify a claim from 150 characters. It tells you an article exists. It doesn’t tell you what the article says.

The real number is 13, and only when you explicitly ask for raw content. Most integrations don’t. The web research agent in a popular 500-project agent collection takes the top 5 results, keeps the first 500 characters of each, and asks the model to synthesise a report from that. That is not verification. That is summarising search snippets.

Then I read the docs

The content coming back is a fetched web page converted to markdown. I could tell because it still had “Skip to main content” at the top, a cookie banner, the word “advertisement”, and ad tracking parameters stuck to the end. Nobody licenses you a clean feed with tagmgr=gtm in it. Somebody loaded the page.

Their crawler documentation explains the rest, and to their credit it is stated openly:

“The Tavily Search crawler does not advertise a differentiated user agent because we must avoid discrimination from websites that allow only Google to crawl them.”

“If a domain or page is not crawlable by Googlebot, then Tavily Search’s bot will not crawl it either.”

That’s the whole trick. Publishers block ClaudeBot and GPTBot by name, because they don’t want AI crawlers. They keep allowing Googlebot, because they still want the search traffic. If you don’t say who you are, you get Googlebot’s permissions.

Why that’s a no for me

Earlier the same day, before any of this, I’d asked whether my tool could just present itself as something other than an agent to get past those blocks.

The answer I got back was no, and it was right. Dressing a script up as a browser to walk through a door a publisher deliberately closed is circumvention whatever the user agent string says. A fact-checking tool caught doing that has a much bigger problem than a missing source.

Going through a search API gets the same result with one more layer in between. The reason it can read Reuters is that it doesn’t say who it is.

I want to be fair here. This is legal. It respects Googlebot’s robots.txt. It is standard practice across AI search and plenty of serious products run on it. Tavily documents it plainly instead of hiding it, which is more than most.

But publishers who allow Googlebot and block AI crawlers by name have said something quite specific, and this walks through the gap between those two rules. For a tool whose entire argument is “check it yourself”, “my vendor does it” is not a line I want to defend.

So: yes, a search API reaches the blocked sources. No, I’m not using it for that.

What I’m doing instead

Same as before, and now I have a better reason.

The Guardian runs an Open Platform API. Free key, full article body, offered on purpose. That’s a door somebody opened rather than one left ajar. Wikipedia is reachable and its reference lists point straight at the reporting I can’t fetch, with the claim, the date and the outlet attached. Every dedicated fact-checker I tested is reachable, and they’re built for this. And primary sources, which my own rubric already ranks above reporting, were never blocked in the first place.

That last one is the interesting part. When I wrote the source hierarchy I put empirical and primary sources at the top because they’re epistemically better. It turns out they’re also the tier I can actually reach. The ranking that’s correct on principle is the same one that’s correct on availability.

The measurements

Full write-up with the per-domain numbers, the method, and what I’d do differently: experiments/2026-07-30-search-api-access.md.

One provider, one day, 16 domains, one query each. Serper and Exa might behave differently and I haven’t tested them. If you run it and get something else, I’d like to know.