Free tool
AI Bot Access Checker
Enter a page. The checker tells you whether ChatGPT, Claude, Perplexity, Google and Bing crawlers can reach it, and when one cannot, whether robots.txt or your server's firewall stops it.
Built by Mantas Tamošaitis, SEO specialist in Lithuania · Source on GitHub
What each result means
Every bot gets two checks: the robots.txt rule that applies to it, and a real request to the page with its User-Agent, compared with the same request from a desktop browser.
- Allowed
- robots.txt lets the bot in and the server answers it the same way it answers a browser.
- Blocked (robots.txt)
- A Disallow rule applies to this bot. The tool shows the rule and the user-agent group it came from.
- Blocked (server/firewall)
- robots.txt allows the bot, but the server refuses its User-Agent: 403, 429, a dropped connection, or a challenge or captcha page, while a normal browser gets the page.
- Inconclusive
- The firewall refuses a fake Googlebot as well, so it checks crawler IP addresses. From outside it cannot be told whether the real crawler gets in.
- Check
- The bot gets HTTP 200, but a response under a fifth of the size a browser gets: often a stripped or block page.
- Page error
- A normal browser gets an error too, so the page itself fails rather than the bot.
Use it from code or an AI agent
The same check runs as a REST API and a remote MCP server. No key needed. Source and the agent skill are on GitHub under the MIT licence.
REST API
curl "https://tools.seopaslaugos.io/v1/check?url=example.com"
curl "https://tools.seopaslaugos.io/v1/check?url=example.com&bots=GPTBot,ClaudeBot&format=text"OpenAPI spec: tools.seopaslaugos.io/openapi.json
MCP server (Claude, Cursor, ChatGPT and other clients)
Streamable HTTP endpoint https://tools.seopaslaugos.io/mcp, one tool: check_ai_bot_access.
{
"mcpServers": {
"ai-bot-access-checker": {
"type": "http",
"url": "https://tools.seopaslaugos.io/mcp"
}
}
}Claude Code and Cursor read this block from .mcp.json / .cursor/mcp.json in your project.
Questions
AI SEO services →01Why would a site block AI bots when its robots.txt allows them?
robots.txt is only read by crawlers that choose to obey it. Firewalls act earlier: hosting security rules (ModSecurity rule sets on shared hosting), rate limiters that answer 429 to bots on uncached pages, and CDN bot settings such as Cloudflare's AI crawler blocking. These often switch on by default, so the site owner never decided to block anyone.
02Does blocking Google-Extended remove a site from AI Overviews?
No. Google-Extended is a robots.txt token that controls whether content is used for Gemini models. AI Overviews and AI Mode are part of Google Search, which uses Googlebot, so a site that allows Googlebot can still be shown there.
03Which bots matter for being cited in ChatGPT, Claude and Perplexity?
The search and user bots: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, PerplexityBot and Perplexity-User for Perplexity. GPTBot, ClaudeBot and CCBot collect training data; blocking them does not stop live citations, but blocking the search and user bots does. ChatGPT search also leans on Bing, so bingbot matters too.
04How accurate is the result?
The robots.txt part follows RFC 9309, the robots.txt standard, and is exact. The server part sends each bot's published User-Agent from Cloudflare, not from the vendors' own IP ranges. It catches blocks by User-Agent, which is how most hosting firewalls work. When a firewall verifies crawler IPs the tool says Inconclusive instead of guessing.
05Is it free, and are there limits?
Free, no sign-up and no key. Each IP address can run 20 checks a minute, and a result for the same URL is reused for 10 minutes so the checked site is not flooded.
Stop paying for reports. Start getting customers.
- ✓Free niche analysis
- ✓Zero obligation
- ✓30 minutes