← All free tools
Tool 2 of 3
AI crawler checker
Paste your robots.txt and see which AI and search crawlers it allows or blocks from your site root. A blocked crawler generally can’t use your pages for that product’s search or answers.
To check your own site, open yoursite.com/robots.txt, copy the text and paste it here.
| Crawler | Operator | Status |
|---|
Notes: this checks the site root (/) using the standard longest-match rule, and treats a crawler with no group of its own as following the * group. Crawler names and behavior change; verify against each operator’s documentation. Blocking Google-Extended controls use of content for Google’s AI models, not Search ranking.