How to check AI crawler access to your site: robots.txt, firewall and server logs
Short answer. Access is checked in four layers: robots.txt for each bot separately, CDN and firewall rules, server logs, and matching IPs against the platforms’ official lists. robots.txt alone is not enough. On 9 of the 39 large Ukrainian websites we tested on 12 November 2026, the file allowed ClaudeBot, yet the server answered that name with 403, 429, 503 or no response at all.
Which bots to let in depends on your goal. Search bots and user-request bots decide whether an AI system can open and cite a page. Training bots are about the rights to your text.
Below: a matrix of bots by platform, what 30 days of logs from our own site chyzh.agency showed, and a six-step check.
Three roles of AI bots: training, search and user requests
Platforms split their bots by job, so each role is allowed or blocked with its own line in robots.txt.
- Training. The bot collects pages that may be used to train models: GPTBot at OpenAI, ClaudeBot at Anthropic, meta-externalagent at Meta. Google and Apple only have tokens for this, with no separate crawler: Google-Extended and Applebot-Extended.
- Search. The bot builds the index that AI search draws linked sources from: OAI-SearchBot for ChatGPT, Claude-SearchBot, PerplexityBot. OpenAI says plainly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
- User request. The bot opens a specific page when a person asks about it in a chat: ChatGPT-User, Claude-User, Perplexity-User. OpenAI and Perplexity warn that such requests may not follow robots.txt. Anthropic states that all three of its bots honor robots.txt.
Google causes the most confusion. According to Google’s documentation (updated 30 October 2026), Google-Extended controls Gemini training and grounding in Gemini Apps and Vertex AI. It has no effect on a site’s inclusion in Search. To appear as a supporting link in AI Overviews or AI Mode, a page must be indexed in Search and eligible for a snippet, and Google points to Googlebot rules plus nosnippet, max-snippet and noindex as the controls. Blocking Googlebot for this means dropping out of Search.
Matrix of bots by platform
Checked against the raw HTML of the official pages on 12 November 2026; update dates are listed in the sources at the end.
| Platform | Bot | Role | robots.txt per docs | IP verification |
|---|---|---|---|---|
| OpenAI | GPTBot | training | honors | gptbot.json |
| OpenAI | OAI-SearchBot | ChatGPT search | honors, picks up changes in about 24 h | searchbot.json |
| OpenAI | ChatGPT-User | user request | rules “may not apply” | chatgpt-user.json |
| Anthropic | ClaudeBot | training | honors, supports Crawl-delay | bots.json |
| Anthropic | Claude-SearchBot | Claude search | honors | same list |
| Anthropic | Claude-User | user request | honors | same list |
| Perplexity | PerplexityBot | search, not used for training per docs | honors, up to 24 h | perplexitybot.json |
| Perplexity | Perplexity-User | user request | “generally ignores” | perplexity-user.json |
| Googlebot | Search, including AI Overviews and AI Mode | honors | common-crawlers.json, DNS *.googlebot.com | |
| Google-Extended | Gemini training and grounding | token only, no separate crawler | not needed | |
| Google-Agent | Google-hosted agents acting for a user | generally ignores | user-triggered-agents.json | |
| Apple | Applebot | Siri, Spotlight, Safari; data may be used for training | honors; without its own group follows Googlebot rules | applebot.json, DNS *.applebot.apple.com |
| Apple | Applebot-Extended | opt-out of Apple model training | token only, does not crawl | not needed |
| Meta | meta-externalagent | training and indexing | honors | no list |
| Meta | meta-webindexer | Meta AI search | honors | no list |
| Meta | meta-externalfetcher | user request | may bypass | no list |
| Amazon | Amazonbot · Amzn-SearchBot · Amzn-User | services and training · Alexa search · user request | honor; Amzn-User may not follow every rule | lists on Amazon’s page |
| Mistral | MistralAI-Training · MistralAI-Index · MistralAI-User | training · search in Vibe · user request | a separate token per role | lists for Index and User |
| DuckDuckGo | DuckAssistBot | DuckAssist answers, no training | honors, changes in 72 h | duckassistbot.json |
| Common Crawl | CCBot | open web archive | honors | ccbot.json, DNS *.crawl.commoncrawl.org |

Two details people often get wrong. First, a group for a specific bot and the general * group are not combined. A bot named in its own group no longer reads the * rules. Second, Applebot without its own group follows the rules written for Googlebot.
What 30 days of chyzh.agency logs showed
How we counted. We took the nginx logs of our site for 13 October to 11 November 2026 (UTC). In our setup the logs record the real client addresses. We matched every request carrying a bot name against the platform’s official IP list as of 12 November 2026. Separately, we flagged a scanner: addresses that used three or more different AI bot names during the month or probed for .env files, keys and passwords.
| Bot | Requests with this name | From official IPs | Scanner using the name | Other IPs not on the list |
|---|---|---|---|---|
| GPTBot | 2,335 | 2,248 | 87 | 0 |
| ClaudeBot | 1,833 | 1,752 | 81 | 0 |
| PerplexityBot | 930 | 833 | 97 | 0 |
| OAI-SearchBot | 371 | 272 | 99 | 0 |
| ChatGPT-User | 337 | 195 | 109 | 33 |
| Claude-User | 184 | 26 | 71 | 87 |
| Claude-SearchBot | 76 | 0 | 76 | 0 |
| Perplexity-User | 67 | 0 | 67 | 0 |
| Googlebot, for comparison | 2,510 | 2,100 | 177 | 233 |

A scanner hides behind AI bot names. It used 38 addresses. Of these, 24 cycled through three or more bot names and 34 looked for .env files, cloud keys and SSH keys. 20 addresses belong to Google Cloud and 13 to Cloudflare. Claude-SearchBot and Perplexity-User came only from this scanner in 30 days; the log has no genuine visits from either bot. Overall, of 6,301 requests using the names of OpenAI, Anthropic, Perplexity and DuckDuckGo bots, 964 (15.3%) did not come from those platforms’ addresses.
Verified AI bots made more requests than Googlebot. Bots from OpenAI, Anthropic and Perplexity made 5,326 requests from official addresses; Googlebot made 2,100, 2.5 times fewer. The log does not show how many of these visits ended in a citation.
GPTBot mostly reads sitemaps. Sitemap files took 77% of its verified requests, pages and posts 13%. 901 requests went to /ru/sitemap_index.xml, which does not exist and returned 404. From 16 October to 6 November this happened about 40 times a day. The address is listed in the auxiliary /ru/robots.txt that WordPress with Yoast generates for the language version. It does not appear on the site’s pages or in the main sitemap. We found the error while preparing this analysis and fixed it on 12 November 2026: /ru/robots.txt, /ua/robots.txt and sitemaps under the language folders now redirect (301) to the root files.
User requests show up in the logs. ChatGPT-User came 195 times from 164 different OpenAI addresses, all answered with 200: the home page 86, pages and posts 80, case studies 25. Claude-User came from addresses on Anthropic’s list only 26 times. Another 87 requests with this name came from addresses not on the list, and almost half of those addresses are in Google Cloud. Our data cannot tell whether this is Anthropic using new addresses or a fake.
The logs do not show whether bots read robots.txt. The default nginx template on our server disables logging for /robots.txt with an access_log off line. If you want to see when bots re-read your rules, check that line in your own configuration.
Limits of this data. One agency site over 30 days. We used IP lists as of 12 November 2026, and platforms update them: OpenAI updated its ChatGPT-User list on 6 November, Perplexity on 28 October, Anthropic on 20 October. So some of the “other IPs not on the list” may be genuine. Bytespider and Meta’s bots have no official IP lists, so we did not verify them.
What large Ukrainian websites block
How we counted. On 12 November 2026 we downloaded robots.txt from 55 large public websites: 16 online stores, 10 marketplaces and classifieds sites, 16 media outlets, 8 banks, 5 telecom and delivery companies. We parsed the files with the same rules our checker uses. Then we requested each home page three times with a browser user agent and with the user agents of five AI bots.
- 47 files were readable. 8 sites returned an HTML page instead of robots.txt (an error, a challenge or a placeholder), 4 of them a Cloudflare challenge page.
- None of the 47 files blocks the whole site for GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot or ChatGPT-User. 5 sites block other bots: Amazonbot (3), CCBot, Bytespider and meta-externalagent (one each).
- 13 sites name AI bots explicitly. On 4 of them a bot gets its own group with nothing but
Allow: /, so it no longer sees the 3 to 47 rules in the*group: search, account pages, cart. - 33 of the 55 sites run through Cloudflare, judging by the Server header.
- 16 sites answered 403 or 503 even to the browser user agent, so their firewall reacts to our client and there is nothing to compare. Of the remaining 39, 9 answered an AI bot name differently from the browser in all three runs: 403, 429, 503 or silence. ClaudeBot was refused on all 9, GPTBot and PerplexityBot on 4. On all 9, robots.txt allows that bot, and on two of them it does so with a dedicated group.
Limits of this data. We only sent the bot name from our own address. A firewall may let the real ClaudeBot in from Anthropic’s addresses and drop anyone who merely uses the name, like the scanner in our logs. How a site treats the real bots is visible only in that site’s own logs and CDN dashboard.
A six-step access check
-
Read robots.txt the way a bot does. For each bot you care about, find its group. No group means the
*group applies. If there is a group, the bot reads only that one. Subdomains have their own robots.txt. The status code matters too: on 4xx (except 429) Google assumes there are no restrictions, and on 5xx it stops crawling the site for the first 12 hours. -
Run the domain through a checker. Our AI crawler checker parses robots.txt for 23 bots, groups them by role and shows the rule that matched. It cannot see firewall rules, which is why its verdict reads “allowed by robots.txt”.
-
Check the CDN and firewall. In Cloudflare this is Security → Settings → Bot traffic → AI Crawler and Agent access policies. Search, Agent and Training each get Block, Block on pages with ads or Allow, and Training also gets Disallow via robots.txt. If Bot Preference Sync is on and at least one policy blocks, Cloudflare prepends its own rules to your robots.txt. Bot Fight Mode cannot be bypassed with WAF exceptions. Defaults have changed: in July 2025 Cloudflare announced blocking AI crawlers by default, while the current documentation (updated 2 November 2026) gives new domains Allow. Check your own zone’s settings.
-
Request a page as the bot. Compare the status code for a browser and for the bot:
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://example.com/ curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://example.com/Different codes mean the server reacts to the name. Whether it lets the real bot in from its own addresses, this test cannot show.
-
Look at the logs. Count requests and status codes per bot, for example for ClaudeBot:
grep "ClaudeBot" access.log | awk '{print $9}' | sort | uniq -cAlso check whether your server logs requests to
/robots.txtat all. -
Verify the IPs. Compare addresses from the logs with the lists in the table above. Google, Apple and Common Crawl also support reverse and forward DNS checks; here is the example from Google’s documentation:
host 66.249.66.1 host crawl-66-249-66-1.googlebot.comRequests that fail the check should not be counted as bot visits, and the firewall should not be opened for them.
Which bots to allow and which to block
For a service business or a store that wants enquiries from AI search, we keep search bots and user-request bots open. Training bots get blocked when it matters that texts stay out of models, for example at a paywalled media outlet. According to OpenAI, Anthropic, Google and Apple documentation, this does not affect appearing in their search.
General “no training” signals are also emerging. Since 30 October 2026, Anthropic has been gradually rolling out support for a machine-readable signal based on a draft from the IETF aipref working group, while a ClaudeBot rule in robots.txt takes priority. Google plans to support this signal in early 2027.
Open access does not guarantee citations; it only lets a platform read the page. What to check next is covered in our article what a GEO audit is. If the page is missing even from the regular index, start with why a page is not indexed.
Next step
Run the six steps for your home page and for two or three pages that bring in leads, and write down which bots actually reach them. If logs, firewall and robots.txt disagree, start with the firewall: it answers the bot before the bot gets to robots.txt or the page. We check bot access, logs and CDN settings at the start of GEO promotion.
Author: Yevhen Pavlenko, DevOps. Reviewed by Dmytro Popryadukhin, Head of SEO.
Sources: OpenAI, “Overview of OpenAI Crawlers”, no update date on the page, checked 12 November 2026; OpenAI IP lists gptbot.json (22 September 2026), searchbot.json (2 January 2026), chatgpt-user.json (6 November 2026); Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, updated 4 November 2026, IP list bots.json (20 October 2026); Perplexity, “Perplexity Crawlers”, updated 29 October 2026, IP lists 28 October 2026; Google, “List of Google’s common crawlers”, 30 October 2026; “List of Google user-triggered fetchers”, 19 August 2026; “Verify requests from Google crawlers and fetchers”, 20 March 2026; “AI features and your website”, 10 December 2025; “How Google interprets the robots.txt specification”, 23 October 2026; Apple, “About Applebot”, 4 September 2026; Meta, “Meta Web Crawlers”, 21 May 2026; Amazon, “Amazon Web Search crawlers”, 26 October 2026; Mistral, “Mistral crawlers”, undated, checked 12 November 2026; DuckDuckGo, DuckAssistBot help page, undated, checked 12 November 2026; Common Crawl, “CCBot”, undated, IP list 11 August 2026; Cloudflare, “AI Crawler and Agent access policies” and “robots.txt setting”, 2 November 2026, “Bot Fight Mode”, 3 August 2026, blog post “Content Independence Day”, 1 July 2025; chyzh.agency nginx logs, 13 October to 11 November 2026; robots.txt files and home page responses of 55 Ukrainian websites, 12 November 2026; Google Ukraine search results for a Ukrainian-language query on checking AI crawler access, 12 November 2026.
Related articles
We ran free audit.chyzh.agency checks on our own site: scores from 0 to 100. What...
Even Google AI Mode confuses GEO with local SEO. What GEO website promotion includes, which...