Seven news sites covering the Coachella Valley and Inland Empire ask at least some AI crawlers to stay out, according to public files we checked on October 13.
The restrictions differ by outlet. Some tell every crawler we examined to keep out. Others make exceptions for particular crawlers or sections of their sites.
Three outlets—KESQ, the Coachella Valley Independent and NBC Palm Springs—name no AI crawler in their files. The Press-Enterprise’s file returned “not found.”
These rules appear in robots.txt, a public file that tells automated visitors where a site asks them not to go. The file is a request, not proof that a crawler stayed out. It also cannot tell us whether an AI company has used a newsroom’s reporting.
Which sites restrict access?
The Inland Valley Daily Bulletin, San Bernardino Sun and Redlands Daily Facts each disallow all 12 crawlers we examined across their entire sites.
The Desert Sun also disallows all 12, but makes narrow exceptions. Five—GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot and Google-Extended—are allowed into branded and contributor content paths.
The Desert Review and Hi-Desert Star disallow 11 of the 12. Amazonbot is the exception.
Palm Springs Life disallows nine: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Applebot-Extended, Bytespider, meta-externalagent and Amazonbot. It does not block ChatGPT-User, OAI-SearchBot or PerplexityBot.
The full set we examined was:
- OpenAI’s GPTBot, ChatGPT-User and OAI-SearchBot.
- Anthropic’s ClaudeBot and anthropic-ai.
- Google-Extended, Common Crawl’s CCBot, PerplexityBot and Applebot-Extended.
- ByteDance’s Bytespider, Meta’s meta-externalagent and Amazonbot.
These are the 12 we chose to examine. A site could restrict other crawlers.
Four sites had no named AI restrictions
KESQ, the Coachella Valley Independent and NBC Palm Springs name no AI crawler in their robots.txt files. NBC Palm Springs’ KMIR address leads to the same site.
The Press-Enterprise’s robots.txt returned “not found,” so that file states no crawler rules.
That does not establish why an outlet has taken a particular approach. We did not ask the publishers to explain their files.
A new way for AI to report its use of journalism
Access rules are one part of the relationship between publishers and AI. A new standard addresses another: how AI systems can report their use of a publisher’s content.
SPUR, a publisher coalition, released Content Telemetry 1.0 on October 2. It defines how AI agents can report when they retrieve content, use it to support an answer, cite it, display it or send readers to it. Our Intelligence brief explains the background.
The standard describes a small public file, called a manifest, that lets a publisher specify where those reports should be sent. Its address is /.well-known/content-telemetry.json.
We checked that address at all 11 outlets. Ten returned “not found.” Palm Springs Life refused the request, so we could not determine whether it had the file.
The standard was only 11 days old when we checked. This finding records what was available that day; it does not establish that an outlet without the file is behind.
What this check can—and cannot—tell us
The files show the outlets’ published crawler rules. They do not show whether AI companies obey those rules or whether a site’s journalism has been used.
The 11 sites were outlets we could identify that report on the region. This was not a census.
For disclosure, AICV.news’s own robots.txt disallowed nothing at the time of the check and explicitly allowed OpenAI’s, Perplexity’s and Anthropic’s crawlers. AICV.news did not publish a content-telemetry manifest either.
How we checked
A script identifying itself as an AICV News check requested each site’s robots.txt and manifest address twice on October 13. The results matched for the nine sites that answered it.
The Desert Sun and NBC Palm Springs refused scripted requests, so we read their files once in a browser.
We counted a crawler as blocked when a group naming it contained Disallow: /. The Desert Sun’s exceptions are noted above.
The manifest’s description appears in section 8 of SPUR’s specification.
Update, October 14: Revised the headline and opening, reorganized the article for clarity and added key points. The facts and sources are unchanged. The findings remain a snapshot of October 13.