Can AI assistants read my website?
Yes, if you let them in and the text is in the HTML. Every AI company has its own crawlers, which request your page the way a browser would and read what the server sends back. Four things can stop them:
- robots.txt. A
Disallow: /rule for their name, or for everyone (User-agent: *), tells them to stay out. The search crawlers of OpenAI, Anthropic and Perplexity follow it. - Your firewall or CDN. Some have a setting to block AI bots. If it is on, it does not matter what your robots.txt says: the crawler gets an error.
- JavaScript. If your website draws its text in the browser (a React app or similar without prerendering), the crawler may get an almost empty page. Many AI crawlers do not run JavaScript.
- noindex. A page with noindex stays out of Google's index, and Google only uses indexed pages in AI Overviews.
The free scanner requests your home page with the crawlers in the table below, counts the words each one sees and runs 12 checks: readable text without JavaScript, the h1, title and description, structured data, the canonical URL, robots.txt, llms.txt, the sitemap, FAQPage, the sharing image, the declared language and whether the page can be indexed.
The crawlers the scanner checks
| Crawler | Whose and what for | User agent the scanner sends |
|---|---|---|
| Googlebot | Google. Crawls for Search, including AI Overviews and AI Mode. | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| GPTBot | OpenAI. Collects content that may be used to train its models. | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot |
| OAI-SearchBot | OpenAI. The one behind ChatGPT search: block it and you are left out of its search answers. | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot |
| ClaudeBot | Anthropic. Collects content that may be used to train Claude. | Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
| PerplexityBot | Perplexity. Surfaces and links websites in its answers; not used to train models. | Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
There are more. OpenAI has ChatGPT-User for visits a user asks for; Anthropic has Claude-SearchBot for its search and Claude-User for visits a user asks for; Perplexity has Perplexity-User. They are listed in the documentation from OpenAI, Anthropic and Perplexity.
Anyone can put “GPTBot” in their user agent, so a visit that claims to be it need not come from OpenAI. That is why OpenAI and Perplexity publish the IP ranges of their crawlers: if you filter by IP in your firewall, use those lists to let the real ones in and keep out anyone pretending to be them.
How do I check whether GPTBot or ClaudeBot can access my site?
Read your robots.txt and request your page with their user agent. Do both: robots.txt says what you allow, and the test shows what actually happens.
- Open yourdomain.com/robots.txt and look for a
User-agent: GPTBotorUser-agent: ClaudeBotgroup withDisallow: /. If they have no group of their own, theUser-agent: *group applies. - Request your page with their user agent from a terminal, for example
curl -I -A "GPTBot/1.2" https://yourdomain.com. A 403 or 429 means something on your server or CDN is blocking them even if robots.txt lets them in. - Check your server or CDN logs: visits from GPTBot, OAI-SearchBot or ClaudeBot with a 200 status mean they are getting in.
- Or run the scanner, which does both tests for you.
A robots.txt that lets each AI's search crawler in and blocks training:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Blocking GPTBot and ClaudeBot does not take you out of ChatGPT or Claude search: OAI-SearchBot and Claude-SearchBot handle that. Anthropic warns in its documentation that blocking Claude-SearchBot may make your site show up less in its answers. What you decide about training is a separate choice.
Google-Extended is not a crawler: it is a name you use in robots.txt to say whether Google may use your content to train Gemini. Google crawls with its usual user agents, and according to its list of crawlers, Google-Extended does not affect your presence in Search.
What is llms.txt and do I need one?
It is a Markdown file at the root of a website (/llms.txt) that tells language models what the site contains and links to its main pages. It is an open proposal from llmstxt.org, not a standard, so it is not enough on its own: what matters first is that crawlers can get in and read your site.
The format is simple: a heading with the site name, a one-line summary, some notes and lists of links with a sentence each. A section called “Optional” marks what can be skipped when space is short. Some sites also publish llms-full.txt, with the text of every page.
Do you need one? It is cheap to add and does no harm, but do not expect it to change your answers on its own: no AI search engine says in its documentation that it uses it to decide whom to name. It mainly helps agents and tools that read a website to work with it.
This website has one, in Spanish at /llms.txt and in English at /llms-en.txt, and every page has a Markdown version at the same address ending in .md. The scanner checks whether your site has llms.txt.