Robots.txt for AI Crawlers: What a Local Business Should Allow

If your website tells AI crawlers to stay away, ChatGPT, Perplexity and Claude may have nothing from your site to quote when someone asks for "the best physiotherapy clinic in Baner". Getting your robots.txt for AI crawlers right takes about ten minutes, and the main decision is simple: block the bots that collect content for training if you want to, but think carefully before you block the ones that power search answers.

Many local business owners have never opened their robots.txt file. Some have a rule they copied from a forum, a plugin that added AI blocks, or a hosting setting that changed on its own. This guide explains what each major AI crawler does, according to the companies that run them, and gives you a sample file you can adapt.

What robots.txt is and what it can and cannot do

A robots.txt file is a plain text file that tells crawlers which URLs on your site they can access. Google's documentation says it must be named robots.txt and must sit at the root of your site, for example https://www.yourclinic.in/robots.txt. A site can have only one, and the rules inside it are case-sensitive.

Three limits matter before you edit anything:

It is a request, not a lock. Google's introduction to robots.txt says Googlebot and other respectable crawlers obey it, but other crawlers might not.
It does not remove a page from Google. Google says robots.txt is not a mechanism for keeping a web page out of Google. Use noindex or password protection for that.
It only covers your website. Your Google Business Profile, your listings on directories and your reviews live on other platforms. A robots.txt rule on your site does not change what those platforms show.

You can see your current file by typing your domain followed by /robots.txt into a browser. If you get a "not found" page, you have no file, and Google treats that as permission to crawl everything.

The AI crawlers that matter, and what each one does

Most AI companies now run more than one bot, each with a different job. The table below summarises what each company's own documentation says.

User agent	Company	What it is for, per the company	If you block it
Googlebot	Google	Crawls for Google Search; a page must be indexed in Search to appear in AI Overviews and AI Mode	Google cannot crawl your pages for Search, including its AI features
Google-Extended	Google	A control token for Gemini model training and grounding in Gemini Apps and Vertex AI	No effect on Google Search inclusion or ranking, per Google
OAI-SearchBot	OpenAI	Surfaces websites in ChatGPT search results	Your site will not be shown in ChatGPT search answers (it can still appear as a navigational link)
GPTBot	OpenAI	Crawls content that may be used to train OpenAI's foundation models	Signals your content should not be used for training
ChatGPT-User	OpenAI	Visits pages when a ChatGPT user's action requires it	OpenAI says robots.txt rules may not apply, since a user started the request
PerplexityBot	Perplexity	Surfaces and links websites in Perplexity search results; not used for foundation model training	Perplexity recommends allowing it so your site appears in results
Perplexity-User	Perplexity	Fetches a page when a user asks a question	Perplexity says it generally ignores robots.txt
ClaudeBot	Anthropic	Collects content that could be used for training	Signals your future content should be excluded from training
Claude-SearchBot	Anthropic	Indexes content to improve Claude's search results	May reduce your visibility in Claude's search results
Claude-User	Anthropic	Fetches pages when Claude users ask questions	Stops Claude from retrieving your pages for those questions

Sources: Google's common crawlers page, OpenAI's crawler overview, Perplexity's crawler guide and Anthropic's crawler help article.

Two points from this table catch people out. OpenAI says each of its settings is independent, so blocking GPTBot does not block OAI-SearchBot, and the reverse is also true. And Google-Extended is not a separate crawler: Google describes it as a product token with no user agent string of its own, which works through Google's existing crawlers.

Should a local business block AI crawlers?

For most clinics, restaurants, salons, showrooms and coaching centres, the answer is to stay visible in AI search. Your site holds the facts an AI answer needs: your services, prices, doctors, menu, timings and the areas you serve. If a search bot cannot read those pages, it has to rely on whatever other sites say about you, or leave you out.

Training bots are a different question. Blocking GPTBot, ClaudeBot or Google-Extended is a choice about whether your content is used to build future models. It is reasonable to block them if you publish original material you do not want reused, such as a coaching institute's practice tests or a real estate firm's market reports. For a typical services page, there is less at stake either way.

One nuance with Google-Extended: Google says it also covers grounding in Gemini Apps, which means supplying content from Google's Search index to the model when someone asks Gemini a question. So blocking it is not only a training decision. If you want Gemini to be able to draw on your site, think twice before blocking it.

A practical way to decide:

Allow every search crawler (Googlebot, OAI-SearchBot, PerplexityBot, Claude-SearchBot) unless you have a specific reason not to.
Decide on training crawlers (GPTBot, ClaudeBot, Google-Extended) based on how original and valuable your content is to you.
Leave user-triggered fetchers alone. ChatGPT-User and Perplexity-User generally do not follow robots.txt anyway, according to their own documentation, so blocking them in the file achieves little.
Never block Googlebot to stay out of AI Overviews. Google's AI features page names robots.txt rules for Googlebot as the crawl control for its AI features. Blocking Googlebot affects your normal search listings too.

A sample robots.txt for AI crawlers

Here is a starting file for a business that wants to appear in AI search answers but keep its content out of model training. Replace the sitemap address with your own.

# Search crawlers: allowed
User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training: opted out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/

Sitemap: https://www.yourclinic.in/sitemap.xml

How the groups work, according to Google's robots.txt documentation (other crawlers may read files slightly differently):

Each group starts with a User-agent line, and crawlers read groups from top to bottom.
A crawler follows only the first, most specific group that matches it. That is why Googlebot here ignores the Disallow: /admin/ rule under *. If you want a rule to apply to a named bot too, repeat it in that bot's group.
Allow can override a Disallow for a page or folder inside a blocked section.

If you would rather let training crawlers in as well, delete the three Disallow: / groups. If your site has a booking engine or customer login area, add a Disallow for that path in each group.

Check whether your site is blocking AI search by accident

Plenty of sites block AI crawlers without the owner knowing. Run through these checks this week.

1. Read the live file

Open yourdomain.in/robots.txt in a private browser window. Look for lines like User-agent: OAI-SearchBot or User-agent: PerplexityBot followed by Disallow: /. Also check for a User-agent: * group with Disallow: /, which blocks every crawler that does not have its own group.

2. Check your CDN or hosting settings

If your site runs through Cloudflare, its dashboard has AI bot policies under Security Settings that can block AI crawlers. Cloudflare's documentation on managed robots.txt explains that when you turn on blocking, Cloudflare can add its own rules to the start of your robots.txt. So the file visitors and bots see may differ from the one your developer uploaded. Check the live file, not the copy on your server.

3. Check plugins and website builders

Some SEO and security plugins on WordPress, and some website builders, offer AI bot toggles. If someone in your team or agency switched one on, it may have added rules for search bots as well as training bots. Compare the live file with the table above.

4. Test in Search Console

Google's robots.txt report in Search Console shows the robots.txt files Google found for your site, when it last crawled them, and any errors or warnings. After a fix, you can request a recrawl from the report, though Google says this does not guarantee an immediate recrawl. Google ignores lines it cannot parse and uses the rest of the file, so an error there can quietly change which rules apply.

5. Give it time

OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust. Google says crawling can take anywhere from several days to several months, so do not judge the change by one day's results.

What robots.txt will not do for AI visibility

Allowing search crawlers makes your site readable. It does not make AI tools recommend you. Google's guidance on AI features and your website says there are no additional technical requirements for AI Overviews or AI Mode beyond being indexed and eligible to show with a snippet, and no special schema markup you need to add.

So once the file is right, the rest of the work is the usual local SEO work:

Make the facts easy to find on your site. A clear page per service and per branch, with address, phone, timings and areas served in plain text, not only inside images.
Keep your details consistent everywhere. AI answers can draw on several sources, so a wrong phone number on one directory can travel. Keeping your name, address and hours consistent across Google, Bing, Apple Maps and directories removes that risk.
Watch what AI assistants actually say. Ask ChatGPT, Gemini and Perplexity the questions your customers ask, such as "best dental clinic in Indiranagar for kids", and note whether you appear. You can also track your mentions in ChatGPT, Gemini and AI Overviews over time instead of checking by hand.

If you want to limit what Google shows from a particular page in Search, including AI features, Google points to page-level controls such as nosnippet, data-nosnippet, max-snippet and noindex. These are more precise than robots.txt for that job.

Frequently asked questions

Does blocking GPTBot remove my business from ChatGPT?

Not from ChatGPT search. OpenAI says GPTBot is for training and OAI-SearchBot is for search, and each setting is independent. Blocking GPTBot signals that your content should not be used for training. To stay in ChatGPT search answers, keep OAI-SearchBot allowed.

Will blocking Google-Extended hurt my Google ranking?

Google says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. It does control use of your content for Gemini training and for grounding in Gemini Apps and Vertex AI.

Is there a robots.txt rule to stay out of AI Overviews only?

Google's AI features page names robots.txt rules for Googlebot as the crawl control and does not list a separate token for AI Overviews. Blocking Googlebot also affects your regular search results. To limit what appears from a page, Google points to snippet controls such as nosnippet.

I run 30 branches on one website. Do I need a file per branch?

No. Google allows only one robots.txt file per site host, at the root. If branch pages sit on separate subdomains, such as pune.yourbrand.in, each subdomain needs its own file.

Your robots.txt is a small file that decides whether AI search tools can read your website at all. Open it today, make sure the search crawlers are allowed, make a deliberate choice about the training crawlers, and check your CDN and plugins for blocks you did not add. Then put your effort into the details AI answers depend on. If you want to see where your business stands in Maps and AI answers today, talk to the RankMonk team.