The news outlets an AI fact-checker cannot read
I tried to make my fact-checking tool prefer reputable sources. Turns out 16 of the most reputable ones refuse to talk to it. Their reporting still comes back, wearing someone else's URL.
Someone on Hacker News gave me a good suggestion: tell the model to “use credible sources”. You can watch its reasoning and see it rule out random blogs and prefer outlets with a reputation for fact checking.
I said I’d test it and publish the comparison either way.
Test failed. What I found instead is more interesting, at least to anyone building tools that ask a model to check facts against the open web.
Sixteen of the news organisations with the strongest fact-checking reputations refuse the crawler. Telling the model to prefer them does nothing. They were never in the pool.
What I was doing
I maintain a small open-source fact-checking tool. It takes a video or an article, splits it into claims, searches for independent evidence on every claim, and gives a per-claim verdict with sources. That’s the whole trick. It is not clever.
Experiment was supposed to be simple. Run the same report twice, once with the extra instruction and once without, diff the verdicts.
The measurement
I tried to pin one arm of the test to reputable outlets. The search API refused and told me why:
API Error: 400 The following domains are not accessible to our user agent:
['ft.com', 'reuters.com', 'wsj.com']That error is a free measuring instrument. It lists exactly which domains in a batch are blocked, so a handful of calls maps a long list.
Of roughly 47 domains I tested on 30 July 2026, 16 were unreachable:
Reuters · AP · WSJ · FT · The New York Times · The Guardian · The Economist · The Atlantic · The New Yorker · Politico · Wired · The Verge · Ars Technica · BBC · Business Insider · ZDNet
These were reachable:
Every .gov I tried (SEC, FTC, DoJ, NASA, NIH) · WHO · europa.eu · gov.uk · Nature · Science · ScienceDirect · arXiv · PubMed · every dedicated fact-checker I tried, so Snopes, PolitiFact, FactCheck.org, Full Fact · Wikipedia · Washington Post · NPR · Bloomberg · CNBC · Forbes · Time · ProPublica · Axios · TechCrunch · The Information · 404 Media · The Register
Primary and official sources are fine. Scientific literature is fine. Dedicated fact-checkers are fine.
The one tier that got hollowed out is general news reporting with a corrections desk. Which is exactly the tier that “use credible sources” points at.
This is not a complaint about publishers. Blocking AI crawlers is a rational business decision in the middle of an industry-wide argument about licensing, and some of these organisations are in active negotiations or litigation. They are entitled to it. But it has a consequence for anyone building fact-checking on top of a general web search, and I hadn’t seen it written down anywhere.
What fills the gap
Something always ranks. In my baseline run, seven factual claims from a business-advice video, the number of reputable news sources retrieved was zero.
What came back instead: SEO affiliate blogs, vendor content marketing, a paid press release wire, a stock trading forum, an AI-generated wiki. For a claim about the speaker’s own revenue, the only sources were the speaker’s own podcast and website.
A model asked to prefer credible sources over that candidate set will do its best. Its best is picking the least bad SEO blog.
The part that’s worse than absence
I expected blocked outlets to be missing. They weren’t missing. They came back second hand.
One claim was about a startup’s user numbers. WIRED and Business Insider both reported figures. Both are blocked. The numbers arrived anyway:
“Business Insider reported about 200,000 signups in a week, while WIRED reported 518,284 registered humans.”
That came from an aggregator quoting them, next to the company’s own boast that it “was featured in Forbes, Wired, Futurism, and 30+ more publications”.
So the tool ends up holding WIRED’s number, with no access to WIRED’s article, and a citation pointing at whoever repeated it. To a naive source counter that looks like independent corroboration. It isn’t. It’s one newsroom’s reporting wearing an aggregator’s URL, and the original is unreachable by construction, so you can’t fall back to it.
Blocking doesn’t remove reporting from the pipeline. It removes the attribution and launders what’s left.
That’s a much more specific failure than “AI gets things wrong”. It produces confident, well-sourced-looking output on a citation graph that quietly collapsed to one node.
The other thing I found, which killed the experiment
I was setting up a control arm: same query, run twice, to see how much results move on their own. Same query, same day, minutes apart.
| First run | Second run |
|---|---|
| techinterview.org | bogleheads.org |
| rupakghose.substack.com | financhill.com |
| danielscrivner.com | quantifiedstrategies.com |
| wallstreetoasis.com | quartr.com |
| pyrfordfp.com | ofdollarsanddata.com |
| brokersdb.com | en.wikipedia.org |
| waylandz.com | medium.com |
| brunch.co.kr | moneysexnerd.com |
| brunch.co.kr |
One domain in common out of nine, marked in bold above.
Note what else moved. Second run surfaced Wikipedia, which is a reasonable source for this question. First run didn’t return it at all. So the churn doesn’t just swap one blog for another, it changes the quality of what you have to work with.
Worse, the second run lost the evidence the published verdict depended on. The claim needed a specific fact about two hedge funds. First run retrieved it. Second run came back with, in its own words, “the search results do not contain any information comparing… D.E. Shaw or Two Sigma.”
Same tool, same question, same afternoon. One run can support a verdict and the next can’t reach it. A different query in the same batch returned byte identical results both times, so this isn’t a uniform noise floor you can subtract. It’s per query and unpredictable.
Which means my original experiment was never going to work. Any difference between “with the instruction” and “without” would be indistinguishable from that.
What I don’t know
This is one crawler. I measured Anthropic’s user agent on one day. Other vendors have different agreements and different blocklists, so your numbers will differ. That’s a problem on its own: the same open-source tool gives different source quality depending on which agent runs it.
The blocklist finding is reproducible. Nothing else here is. Restrict a search to reuters.com and read the error, that’s it. The retrieval churn is an observation from a handful of queries, not a controlled study.
And I still can’t tell you whether “use credible sources” helps. That was the actual question and I failed to answer it. My tool has no instrument that can detect an effect that size, which is a statement about my tool, not about the advice.
What I changed
Three things, in the version I shipped today.
Source tiers, with primary sources on top. Not because primary sources are philosophically purer, but because they’re the tier I can actually reach. Putting reporting first would be ranking on availability I don’t have.
Origin counts instead of URL counts. Every claim backed by more than one link now records how many independent origins are behind it, like [8 URLs → 1 origin: vendor press release]. It’s mandatory, so a missing marker means “single source” and not “nobody checked”.
“Unreachable” as a separate outcome from “unverifiable”. A paywall and a claim nobody ever checked are different findings. Collapsing them lets a tool imply it looked when it couldn’t.
Honest summary of my own position: I built a thing that checks claims against sources, then measured that a large share of the best sources are closed to it. Worth publishing precisely because it’s inconvenient.
Reproduce it
Ask any agent with web search to restrict a search to reuters.com and read what comes back. Then try propublica.org. Difference takes about ten seconds to see. If you get a different answer than I did, I’d like to know.
Full write-up with the raw arms and per-claim results: experiments/2026-07-30-credible-sources.md.
Update: I tested whether a search API reaches these blocked sources instead of a crawler. It does, for 13 of the 16. Then I read how it manages that, and decided not to use it. Part two.