--- title: "Which AI crawlers can read your website · PolarJay" description: "GPTBot, ClaudeBot and the other AI crawlers read your site if your robots.txt lets them in. This is how to check." url: https://polarjay.com/en/guides/ai-crawlers lang: en alternate: https://polarjay.com/guias/rastreadores-de-ia published: 2026-10-23 author: PolarJay updated: 2026-10-23 publisher: POLARGATE S.L. (CIF B24895005) --- Guide # Which AI crawlers can read your website AI assistants read your website with their own crawlers, and you let each one in or keep it out separately in your robots.txt. Published on 23 October 2026 by PolarJay Last updated: 23 October 2026 ## Can AI assistants read my website? Yes, if you let them in and the text is in the HTML. Every AI company has its own crawlers, which request your page the way a browser would and read what the server sends back. Four things can stop them: - **robots.txt.** A `Disallow: /` rule for their name, or for everyone (`User-agent: *`), tells them to stay out. The search crawlers of OpenAI, Anthropic and Perplexity follow it. - **Your firewall or CDN.** Some have a setting to block AI bots. If it is on, it does not matter what your robots.txt says: the crawler gets an error. - **JavaScript.** If your website draws its text in the browser (a React app or similar without prerendering), the crawler may get an almost empty page. Many AI crawlers do not run JavaScript. - **noindex.** A page with noindex stays out of Google's index, and Google only uses indexed pages in AI Overviews. The [free scanner](https://polarjay.com/en/scan) requests your home page with the crawlers in the table below, counts the words each one sees and runs 12 checks: readable text without JavaScript, the h1, title and description, structured data, the canonical URL, robots.txt, llms.txt, the sitemap, FAQPage, the sharing image, the declared language and whether the page can be indexed. ## The crawlers the scanner checks | Crawler | Whose and what for | User agent the scanner sends | | --- | --- | --- | | Googlebot | Google. Crawls for Search, including AI Overviews and AI Mode. | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | | GPTBot | OpenAI. Collects content that may be used to train its models. | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot | | OAI-SearchBot | OpenAI. The one behind ChatGPT search: block it and you are left out of its search answers. | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot | | ClaudeBot | Anthropic. Collects content that may be used to train Claude. | Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | | PerplexityBot | Perplexity. Surfaces and links websites in its answers; not used to train models. | Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | There are more. OpenAI has ChatGPT-User for visits a user asks for; Anthropic has Claude-SearchBot for its search and Claude-User for visits a user asks for; Perplexity has Perplexity-User. They are listed in the documentation from [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers). Anyone can put “GPTBot” in their user agent, so a visit that claims to be it need not come from OpenAI. That is why OpenAI and Perplexity publish the IP ranges of their crawlers: if you filter by IP in your firewall, use those lists to let the real ones in and keep out anyone pretending to be them. ## How do I check whether GPTBot or ClaudeBot can access my site? Read your robots.txt and request your page with their user agent. Do both: robots.txt says what you allow, and the test shows what actually happens. 1. Open yourdomain.com/robots.txt and look for a `User-agent: GPTBot` or `User-agent: ClaudeBot` group with `Disallow: /`. If they have no group of their own, the `User-agent: *` group applies. 2. Request your page with their user agent from a terminal, for example `curl -I -A "GPTBot/1.2" https://yourdomain.com`. A 403 or 429 means something on your server or CDN is blocking them even if robots.txt lets them in. 3. Check your server or CDN logs: visits from GPTBot, OAI-SearchBot or ClaudeBot with a 200 status mean they are getting in. 4. Or run the [scanner](https://polarjay.com/en/scan), which does both tests for you. A robots.txt that lets each AI's search crawler in and blocks training: ``` User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / ``` Blocking GPTBot and ClaudeBot does not take you out of ChatGPT or Claude search: OAI-SearchBot and Claude-SearchBot handle that. Anthropic warns in its [documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) that blocking Claude-SearchBot may make your site show up less in its answers. What you decide about training is a separate choice. Google-Extended is not a crawler: it is a name you use in robots.txt to say whether Google may use your content to train Gemini. Google crawls with its usual user agents, and according to its [list of crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), Google-Extended does not affect your presence in Search. ## What is llms.txt and do I need one? It is a Markdown file at the root of a website (/llms.txt) that tells language models what the site contains and links to its main pages. It is an [open proposal from llmstxt.org](https://llmstxt.org/), not a standard, so it is not enough on its own: what matters first is that crawlers can get in and read your site. The format is simple: a heading with the site name, a one-line summary, some notes and lists of links with a sentence each. A section called “Optional” marks what can be skipped when space is short. Some sites also publish llms-full.txt, with the text of every page. Do you need one? It is cheap to add and does no harm, but do not expect it to change your answers on its own: no AI search engine says in its documentation that it uses it to decide whom to name. It mainly helps agents and tools that read a website to work with it. This website has one, in Spanish at [/llms.txt](https://polarjay.com/llms.txt) and in English at [/llms-en.txt](https://polarjay.com/llms-en.txt), and every page has a Markdown version at the same address ending in .md. The scanner checks whether your site has llms.txt. ## Sources - [OpenAI, overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) - [Anthropic, Anthropic's crawlers and how to block them](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) - [Perplexity, Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) - [Google, introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) - [Google, common crawlers (Google-Extended)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) - [llmstxt.org, the llms.txt proposal](https://llmstxt.org/) ## Keep reading - [Why ChatGPT doesn't mention your company](https://polarjay.com/en/guides/why-chatgpt-does-not-mention-my-company): The usual reasons ChatGPT leaves a company out and what to change. - [What AI visibility is](https://polarjay.com/en/what-is-ai-visibility): AI visibility is whether AI assistants name your company when someone asks about what you sell. Optimizing for it is called GEO. - [How to measure AI visibility](https://polarjay.com/en/guides/measure-ai-visibility): To find out whether ChatGPT and Claude recommend you, ask them your buyer's questions without your name and count whom they name. ## Does ChatGPT name you? Check now One minute, no sign-up. [Scan my website for free](https://polarjay.com/en/scan) [Request the free trial](https://polarjay.com/en/trial)