AI bots and your robots.txt: a plain guide for trades websites
Quick answer: AI firms tend to run one set of bots for search and another for training. To show up in AI answers, let the search bots in: OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot. Shutting out training bots like GPTBot, ClaudeBot and Google-Extended is a different decision and won’t knock you out of AI search results. If your main aim is to get found, letting both types in is the easiest route.
Not long ago, robots.txt was a file you sorted out when the website went live and never looked at again. These days a whole crowd of AI crawlers turn up on small business websites, and they don’t all want the same thing. Turn away the wrong one and your plumbing or roofing firm could vanish from ChatGPT’s search answers without anyone noticing. Wave everyone through and you might be giving your hard-written pages to companies you’d prefer kept their hands off.
Below we explain, in everyday terms, what the main bots do and how we normally set up robots.txt for the trades and local service businesses we work with.
Key takeaways
- Search bots read your pages so AI tools can quote you and link back. Training bots gather text to build the AI models themselves.
- Saying no to a training bot such as GPTBot won’t stop that same company’s AI search from showing your business.
- Google-Extended governs Gemini training and some grounding uses. Your normal Google rankings and AI Overviews are untouched by it.
- Look at your firewall and security settings as well as robots.txt, because they can block AI search bots behind your back.
What does each AI crawler actually do?
| User agent | Owner | What it is for | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Discovers pages to show and link to inside ChatGPT search | Your business won’t appear in ChatGPT search answers |
| GPTBot | OpenAI | Gathers content that may help train OpenAI models | Your pages stay out of training; search visibility is not affected |
| ChatGPT-User | OpenAI | Loads a page when someone asks ChatGPT to open it | Since a person starts the request, robots.txt rules might not be honoured |
| Google-Extended | Not a crawler in its own right: a token that controls Gemini training and some grounding | No change to Google Search, your rankings or AI Overviews | |
| Claude-SearchBot | Anthropic | Builds a search index that Claude draws on when answering | You are less likely to feature in Claude’s answers |
| ClaudeBot | Anthropic | Gathers content for training models | Your pages stay out of training |
| PerplexityBot | Perplexity | Indexes pages for answers given by Perplexity | You are less likely to feature in Perplexity |
| Applebot-Extended | Apple | Sets whether Apple can use your content to train its AI | Apple search features that rely on Applebot are not affected |
| CCBot | Common Crawl | Builds a public archive of the web that lots of AI models learn from | Your pages are left out of future Common Crawl datasets |
These names and roles do shift over time, so we always read each company’s own crawler documentation before changing anything on a client site.
The robots.txt setup we suggest for most local businesses
If you’re an electrician, a cleaning company or a garage that wants AI tools to recommend you, our usual advice is to let the search bots in and then decide deliberately what to do about training. The example below lets AI search through while opting out of training:
# AI search bots: let them in so we can be cited
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# AI training bots: opt out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# All other crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.co.uk/sitemap.xmlLetting the training bots in as well is a perfectly sensible choice. A business that crops up a lot in training data might be more familiar to AI models, which could help it get mentioned. Nobody has solid evidence one way or the other yet, so treat this as a call for you as the owner, not a fixed SEO rule.
Why your firewall matters as much as robots.txt
When we audit a site, the problem we find most often has nothing to do with robots.txt. More and more content delivery networks and security plugins now stop AI bots by default. So your robots.txt can be inviting a bot in while your firewall quietly shuts the door.
- Open the bot management settings in Cloudflare or whichever CDN you use.
- Go through the rules in any WordPress security plugin on the site.
- Check your server logs to confirm which AI bots are arriving and getting a normal page back.
OpenAI says its systems can take roughly 24 hours to pick up a change to robots.txt, so give it a day before you test.
Is an llms.txt file worth adding?
llms.txt is a suggested file that hands AI tools a short plain-text overview of your website, with links to the pages that matter most. Adding one costs very little, and certain AI tools and developer platforms do read it. That said, as we write this, none of the big AI search providers has said it uses llms.txt for ranking or for choosing citations. See it as a cheap bonus, never a substitute for strong service pages and well-judged crawler rules.
Frequently asked questions
If I block GPTBot, will ChatGPT stop showing my business?
No. GPTBot is a training bot. ChatGPT search depends on OAI-SearchBot, so blocking GPTBot on its own won’t take you out of ChatGPT search.
Could blocking Google-Extended hurt where I rank on Google?
No. Google states that Google-Extended has no effect on whether or where you appear in Google Search. AI Overviews are not controlled by it either, as they use Google’s standard crawler.
How can I see which AI bots visit my website?
Search your CDN analytics or server logs for the bot names in our table. Plenty of hosting dashboards, and Cloudflare, break bot traffic down by name.
Do I need an llms.txt file?
It won’t do any harm and only takes a few minutes. Just don’t count on it to change your AI visibility by itself.
Sources and further reading
- OpenAI documentation: the crawlers it runs
- Google: common crawlers, including Google-Extended
- Google Search Central: robots.txt explained
- llmstxt.org: the proposed llms.txt standard
Want help with this? We can check which AI bots are able to reach your website as part of our AI search and GEO service. Get in touch.
More articles
Moving to a new website? An SEO checklist to keep your Google rankings
Changing platform, domain or design? Our step by step SEO checklist helps trades and local firms protect rankings before, during and after launch.
September 2025: the reason Search Console impressions fell overnight
In September 2025 many sites saw Search Console impressions drop while average position improved. We explain why and how to read your figures now.
August 2025: Google’s spam update cracks down on policy breaches
Google began its August 2025 spam update on 26 August. Here is what it targeted and how a local trade business can stay well clear of spam problems.