Plenty of B2B sites block AI crawlers by accident. The cause is usually an old robots.txt rule, a CDN security setting or pages that only render through JavaScript, not a deliberate decision.
Know Which Bot Does What
AI companies run separate bots. Training bots such as GPTBot collect content for models, while search and retrieval bots such as OAI-SearchBot and ChatGPT-User put pages into answers. Google works the same way: Googlebot powers Search and AI Overviews, while Google-Extended only covers Gemini training.
Blocking the Wrong Bot Backfires
Block a search bot and you disappear from the answers you want to appear in. Block a training bot and you only opt out of training.
Check All Three Gates
Robots.txt is one gate. Your CDN’s bot management is another, and it can override robots.txt. The third is what a bot actually receives: if content loads only through scripts, many crawlers see an empty page.
Access Is Only the Start
Once crawlers can reach you, the content still has to be useful and quotable to earn a citation.
Where to Start
- Read your robots.txt for rules aimed at AI bots.
- Check your CDN for a setting that blocks AI bots.
- View the source of key pages and confirm the text is in the raw HTML.
- Look for AI bot user agents in last month’s server logs.
- Ask ChatGPT and Perplexity about your company and see what they cite.
This is a summary. The full article, with sources and examples, was first published on npws.net.