Is your website blocking AI search? How to check your robots.txt
A single line in robots.txt can keep your business out of ChatGPT, Claude or Perplexity answers. Here’s how to check yours in two minutes, and which AI crawlers to allow.
By LocalListerUpdated 6 min read
Key takeaways
- Open yourdomain.com/robots.txt. It’s a public file, and checking it takes two minutes.
- Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether you can appear in those assistants’ answers.
- Training crawlers (GPTBot, ClaudeBot, Google-Extended) are separate. Blocking them doesn’t remove you from search answers.
- Firewalls and bot protection can block AI crawlers even when robots.txt allows them.
Your website can be invisible to AI search without you ever deciding that. A single line in a file called robots.txt, added years ago by a developer, a plugin or a well-meaning setting, can keep your business out of ChatGPT, Claude or Perplexity answers. Checking takes about two minutes.
What robots.txt does
robots.txt is a small public text file at the root of your website. It tells crawlers, the automated programs that read websites, which pages they may visit. Reputable crawlers read it before they crawl, and AI companies now run several crawlers each, for different jobs.
Step 1: Open your robots.txt
Type your domain followed by /robots.txt into a browser, for example www.yourbusiness.com/robots.txt. Anyone can view it.
- A “page not found” error means you don’t have one. That’s fine: crawlers that follow the standard then assume your public pages are open.
- If your site is on Wix, Squarespace or Shopify, the file is generated for you. Check that platform’s SEO or crawler settings before editing anything.
Step 2: Look for lines that block
Rules come in groups. A User-agent line names a crawler (an asterisk means every crawler), and the Disallow lines under it list what that crawler may not visit. Disallow: / means the entire site.
# Blocks every crawler from the whole site
User-agent: *
Disallow: /
# Blocks ChatGPT search specifically
User-agent: OAI-SearchBot
Disallow: /If you see Disallow: / under User-agent: *, or under any of the search crawlers below, your site is blocked from those assistants.
The AI crawlers that matter
The most important distinction is between search crawlers, which decide whether you can appear in answers, and training crawlers, which collect content that may be used to train future models. Blocking a training crawler doesn’t remove you from that company’s search answers.
| Company | Crawler | What it’s for | If you block it |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfacing websites in ChatGPT search | Not shown in ChatGPT search answers |
| OpenAI | GPTBot | Collecting data that may train OpenAI’s models | Left out of training; ChatGPT search isn’t affected |
| OpenAI | ChatGPT-User | Visiting a page when a ChatGPT user asks it to | OpenAI says robots.txt rules may not apply |
| Anthropic | Claude-SearchBot | Indexing pages for Claude’s search results | May reduce your visibility in Claude’s search results |
| Anthropic | ClaudeBot | Collecting data for training Anthropic’s models | Future content excluded from training |
| Anthropic | Claude-User | Fetching a page during a user’s conversation | May reduce visibility for searches users direct |
| Perplexity | PerplexityBot | Surfacing and linking websites in Perplexity (not training) | Not shown in Perplexity search results |
| Perplexity | Perplexity-User | Visiting a page to answer a user’s question | Generally ignores robots.txt, per Perplexity |
Googlebot | Google Search, which AI Overviews and AI Mode draw on | Removed from Google Search | |
Google-Extended | Whether Google may use your content to train Gemini | No effect on Google Search inclusion or ranking |
Read the details in the documentation from OpenAI, Anthropic, Perplexity and Google.
A robots.txt that welcomes AI search
If nothing blocks them, AI search crawlers can already visit your site, so you don’t strictly need to add anything. Naming them explicitly makes your intent clear to anyone who edits the file later:
# AI search crawlers: allowed to find and cite this site
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Everyone else: the whole public site, except private pages
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlWhether to allow training crawlers is your call. Blocking them is a reasonable choice for publishers protecting their writing. For most local businesses, being well described in the information AI models learn from does more good than harm.
Other things that can keep AI crawlers out
- Firewalls and bot protection. Services such as Cloudflare, or your web host, can block AI crawlers even when robots.txt allows them. Check those settings, or ask your web provider.
- Password protection or maintenance mode left switched on for part of a site.
- Content that isn’t in the page’s text. Details that appear only in images, PDFs or after scripts run may be missed. Keep your services, service area and contact details in plain text.
Edit carefully
Check it for free
LocalLister’s free AI visibility audit checks your website’s crawler settings as part of the report, alongside how often AI assistants actually name your business.
Frequently asked questions
Will allowing AI crawlers let them copy my content?
Search crawlers read your pages so assistants can show and link to them in answers, much as Google Search does. Training crawlers are separate: you can block GPTBot, ClaudeBot or Google-Extended and still allow the search crawlers.
I don’t have a robots.txt file. Is that a problem?
No. Without one, crawlers that follow the standard assume they may crawl your public pages. You only need a robots.txt file to block something or to point crawlers to your sitemap.
Does robots.txt stop every AI tool?
No. It’s a set of instructions that reputable crawlers follow. Fetchers acting on a user’s direct request may not: Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says its rules may not apply to ChatGPT-User.
Could something other than robots.txt be blocking AI crawlers?
Yes. Firewalls and bot protection from your web host or a service like Cloudflare can block AI crawlers even when robots.txt allows them. Check those settings, and ask your web provider if you aren’t sure.
Sources
Is AI recommending your business?
Run a free check. We ask the questions your customers ask, several times each, and show who gets named.