FREE TOOL · DETERMINISTIC · ONE FETCH

Find out which AI crawlers can actually reach your site.

Check the live crawler directives on a public domain and see where GPTBot, ClaudeBot, PerplexityBot, and other major AI crawlers stand today.

No account. One bounded check — never the full crawl or a site-wide score.

WHAT YOU GET

ILLUSTRATIVE

2 of 9 tracked AI crawlers have partial restrictions

GPTBotPartially blocked

User-agent: * · Disallow: /internal

ClaudeBotAllowed

PerplexityBotAllowed

What this check actually measures

AI assistants can only learn your site from what their crawlers are allowed to fetch. This check reads your live robots.txt and llms.txt, evaluates homepage access for the major AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest — and returns a per-bot verdict: allowed, blocked, or partially blocked. It reports the current state of your directives, nothing more.

The failure modes, and what each one means

Blocked without deciding to be: A blanket Disallow: / from a CDN template or a years-old bot panic blocks AI crawlers you never considered. Fix: name the agents you actually mean; a wildcard group written for one scraper is now a policy about every AI engine.
Partially blocked in ways that skew what AI knows: Blocking /blog while allowing / means engines see your product pages but none of your expertise. Fix: review path-level disallows against what you'd want an engine to cite.
A robots.txt that couldn't be read: A 403 or 5xx on robots.txt is not permission and not a block — it's an unknown, and this tool reports it as one rather than guessing. Fix: robots.txt should return 200, or 404 if you genuinely have no rules.

Why it matters for search — and for AI answers

Crawler access is the zeroth stage of AI visibility: an engine whose crawler you block has less of your site to learn from, so decisions here are decisions about whether you exist in training and retrieval corpora. It's also a genuine policy choice — some sites block deliberately, and that's legitimate. What this check ensures is that the state is the one you chose.

What the full product adds

This is stage one of TrustGrowth's five-stage AI visibility funnel: Reach. The product re-checks access on a schedule with ten live per-agent fetches, then follows the funnel down — whether AI can use what it fetched, whether it cites you, whether it names you unprompted, and whether any of it moved your branded search.

Convinced? It takes one URL.

TRY ANOTHER CHECK

Check my schema → Build JSON-LD → Build sameAs JSON-LD → Preview my listing → Inspect my canonical → Create my llms.txt → Create llms.txt from sitemap → Compose my robots.txt → See the AI read → Check trust signals → Ask three engines → Convert this page → Grade my llms.txt →

Browse all free tools The full version lives in AI Visibility →

Frequently asked questions

Which AI crawlers do you check?

The major ones: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Amazonbot and Meta-ExternalAgent. The list is maintained as crawlers appear and change their user agents.

What does the tool actually fetch?

Your robots.txt and your llms.txt, directly from the domain you name, plus the homepage to resolve the final URL and status. It evaluates each crawler's rules against that. It does not crawl the rest of your site.

My robots.txt returned a 403. Does that mean AI is blocked?

No. A 403, a 429 or a 5xx is an absence of information, not a permission, and it records as partial rather than blocked. Only a readable robots.txt with a disallow for that crawler is reported as blocked. A WAF page that looks like a robots.txt is not an answer either.

Should I be blocking AI crawlers?

That is your call, and the tool takes no position on it. Blocking a crawler is a decision this product reports; it is not a defect it tries to reverse.

Does llms.txt do anything?

It is a proposed convention, not something the major search engines have committed to honouring. The tool reports whether you have one and whether it could be read, which is a fact. Whether any given engine reads it is not something we can verify for you.

If a crawler is allowed, does that mean it has visited me?

No. Allowed means it may. Whether it did, and whether anything it fetched ended up in an answer, are different questions — that is the difference between crawler reach and AI visibility, and TrustGrowth keeps them apart on purpose.