Technical
Robots.txt and AI crawlers: what to check
The AI crawler user agents SMB sites should review before buying an AI-readiness package.
Many businesses blocked AI crawlers during the early scraping panic. That may be a valid policy choice, but it should be intentional.
User agents to review
- GPTBot
- ClaudeBot
- Google-Extended
- PerplexityBot
- CCBot
- Applebot-Extended
Agentable reports whether these crawlers are blocked and explains the tradeoff before recommending changes.