Technical SEO

AI bots and your robots.txt: a plain guide for trades websites

Aaron, Lead Local Marketing
Published 21 October 2026 · 5 min read

Network cables plugged into a server rack — Photo: Taylor Vick on Unsplash

Quick answer: AI firms tend to run one set of bots for search and another for training. To show up in AI answers, let the search bots in: OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot. Shutting out training bots like GPTBot, ClaudeBot and Google-Extended is a different decision and won’t knock you out of AI search results. If your main aim is to get found, letting both types in is the easiest route.

Not long ago, robots.txt was a file you sorted out when the website went live and never looked at again. These days a whole crowd of AI crawlers turn up on small business websites, and they don’t all want the same thing. Turn away the wrong one and your plumbing or roofing firm could vanish from ChatGPT’s search answers without anyone noticing. Wave everyone through and you might be giving your hard-written pages to companies you’d prefer kept their hands off.

Below we explain, in everyday terms, what the main bots do and how we normally set up robots.txt for the trades and local service businesses we work with.

Key takeaways

Search bots read your pages so AI tools can quote you and link back. Training bots gather text to build the AI models themselves.
Saying no to a training bot such as GPTBot won’t stop that same company’s AI search from showing your business.
Google-Extended governs Gemini training and some grounding uses. Your normal Google rankings and AI Overviews are untouched by it.
Look at your firewall and security settings as well as robots.txt, because they can block AI search bots behind your back.

What does each AI crawler actually do?

User agent	Owner	What it is for	If you block it
OAI-SearchBot	OpenAI	Discovers pages to show and link to inside ChatGPT search	Your business won’t appear in ChatGPT search answers
GPTBot	OpenAI	Gathers content that may help train OpenAI models	Your pages stay out of training; search visibility is not affected
ChatGPT-User	OpenAI	Loads a page when someone asks ChatGPT to open it	Since a person starts the request, robots.txt rules might not be honoured
Google-Extended	Google	Not a crawler in its own right: a token that controls Gemini training and some grounding	No change to Google Search, your rankings or AI Overviews
Claude-SearchBot	Anthropic	Builds a search index that Claude draws on when answering	You are less likely to feature in Claude’s answers
ClaudeBot	Anthropic	Gathers content for training models	Your pages stay out of training
PerplexityBot	Perplexity	Indexes pages for answers given by Perplexity	You are less likely to feature in Perplexity
Applebot-Extended	Apple	Sets whether Apple can use your content to train its AI	Apple search features that rely on Applebot are not affected
CCBot	Common Crawl	Builds a public archive of the web that lots of AI models learn from	Your pages are left out of future Common Crawl datasets

These names and roles do shift over time, so we always read each company’s own crawler documentation before changing anything on a client site.

The robots.txt setup we suggest for most local businesses

If you’re an electrician, a cleaning company or a garage that wants AI tools to recommend you, our usual advice is to let the search bots in and then decide deliberately what to do about training. The example below lets AI search through while opting out of training:

# AI search bots: let them in so we can be cited
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# AI training bots: opt out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# All other crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.co.uk/sitemap.xml

Letting the training bots in as well is a perfectly sensible choice. A business that crops up a lot in training data might be more familiar to AI models, which could help it get mentioned. Nobody has solid evidence one way or the other yet, so treat this as a call for you as the owner, not a fixed SEO rule.

Why your firewall matters as much as robots.txt

When we audit a site, the problem we find most often has nothing to do with robots.txt. More and more content delivery networks and security plugins now stop AI bots by default. So your robots.txt can be inviting a bot in while your firewall quietly shuts the door.

Open the bot management settings in Cloudflare or whichever CDN you use.
Go through the rules in any WordPress security plugin on the site.
Check your server logs to confirm which AI bots are arriving and getting a normal page back.

OpenAI says its systems can take roughly 24 hours to pick up a change to robots.txt, so give it a day before you test.

Is an llms.txt file worth adding?

llms.txt is a suggested file that hands AI tools a short plain-text overview of your website, with links to the pages that matter most. Adding one costs very little, and certain AI tools and developer platforms do read it. That said, as we write this, none of the big AI search providers has said it uses llms.txt for ranking or for choosing citations. See it as a cheap bonus, never a substitute for strong service pages and well-judged crawler rules.

Frequently asked questions

If I block GPTBot, will ChatGPT stop showing my business?

No. GPTBot is a training bot. ChatGPT search depends on OAI-SearchBot, so blocking GPTBot on its own won’t take you out of ChatGPT search.

Could blocking Google-Extended hurt where I rank on Google?

No. Google states that Google-Extended has no effect on whether or where you appear in Google Search. AI Overviews are not controlled by it either, as they use Google’s standard crawler.

How can I see which AI bots visit my website?

Search your CDN analytics or server logs for the bot names in our table. Plenty of hosting dashboards, and Cloudflare, break bot traffic down by name.

Do I need an llms.txt file?

It won’t do any harm and only takes a few minutes. Just don’t count on it to change your AI visibility by itself.

Sources and further reading

Want help with this? We can check which AI bots are able to reach your website as part of our AI search and GEO service. Get in touch.

Oct 2026

Moving to a new website? An SEO checklist to keep your Google rankings

Changing platform, domain or design? Our step by step SEO checklist helps trades and local firms protect rankings before, during and after launch.

Technical SEO Sep 2025

September 2025: the reason Search Console impressions fell overnight

In September 2025 many sites saw Search Console impressions drop while average position improved. We explain why and how to read your figures now.

Technical SEO Aug 2025

August 2025: Google’s spam update cracks down on policy breaches

Google began its August 2025 spam update on 26 August. Here is what it targeted and how a local trade business can stay well clear of spam problems.

Technical SEO

Moving to a new website? An SEO checklist to keep your Google rankings

Aaron, Lead Local Marketing
Published 15 October 2026 · 6 min read

Cardboard moving boxes stacked on a table — Photo: Michal Balog on Unsplash

Quick answer: To move to a new website without losing Google traffic, crawl and record the old site first, point every old URL to its nearest new page with a 301 redirect, test everything on a staging copy, then switch over, submit the fresh sitemap and keep a daily eye on Search Console for the first few weeks. Keep those redirects live for a minimum of a year.

Moving website is one of the easiest ways for a local business to lose its enquiries from Google. Handled carefully, you can hold on to your rankings and often improve them. Handled badly, a big slice of search traffic can vanish overnight and take months to come back, which for an electrician or a removals firm means the phone simply stops ringing.

Thankfully, almost every migration that goes wrong fails for the same few reasons. This is the checklist we work through when we move a client’s site.

Key takeaways

Record how the old site performs before you touch anything, so you can see exactly what changed after launch.
Redirect each old URL with traffic or links to its closest new match, one page to one page.
Test redirects, indexing settings and tracking on the staging site before the big day.
Expect rankings to wobble for a few weeks, and act fast on any page that drops and stays down.

Which changes count as a migration?

Anything that changes a lot of your URLs or how Google understands the site. Common examples:

Switching website builder, CMS or shop platform (for example, Magento to Shopify).
A new domain name, including a switch from .com to .co.uk.
Going from HTTP to HTTPS, or folding subdomains into the main site.
A redesign that alters the page structure or the web addresses, such as a roofer adding separate pages for each of the three towns they cover.

Preparing before you launch

Crawl the old site. Run a crawler such as Screaming Frog and export every URL, page title, heading, canonical tag and status code.
Save your performance data. Download 12 months of Search Console clicks per page, GA4 landing page figures and a list of your most linked pages from a backlink tool.
Note your rankings. Take a snapshot of where you sit for your most valuable searches, such as “emergency plumber” plus your town.
Create the redirect map. A spreadsheet listing every old address down one column with its new destination alongside. Deal with pages that bring traffic, links or enquiries first.
Hold on to what works. Bring across strong titles, content and internal links unless you have a solid reason to change them.
Lock the staging site. Put it behind a password so Google cannot index the test copy.

Getting redirects right

Use permanent 301 redirects, one to one, pointing every old address at the best matching new page. Two shortcuts cause most of the damage:

Shortcut	The problem	Better approach
Sending every old page to the homepage	Google treats it much like a deleted page, so the ranking is lost	Send it to the closest matching page
Redirect chains (A to B to C)	They are slower and signals can leak along the way	Point each old URL directly at its final page

If a page really has no equivalent and has no traffic or links, letting it return a 404 or 410 is fine.

Testing on the staging site

Run a crawl of the staging site and put it side by side with the original crawl, looking for missing pages, titles and headings.
Try out a sample of redirects, making sure your 100 busiest pages are covered if you have that many.
Make sure canonical tags point to the new live addresses, not the staging ones.
Check that structured data, hreflang (only if you run international sites) and XML sitemaps are in place.
Measure mobile speed and Core Web Vitals, because a slower site can wipe out the gains of a new design.
Confirm analytics and conversion tracking, such as call and form tracking, fire on the new templates.

What to do on launch day

Take off the password, and strip out any leftover “noindex” tags or robots.txt rules from staging. This is the most common launch mistake we come across.
Turn on the redirects and spot check a handful.
Send your new XML sitemap to both Google Search Console and Bing Webmaster Tools.
Changed domain? Run Search Console’s Change of Address tool.
Update links you control elsewhere, such as your social profiles and your Google Business Profile website link.

What to watch after going live

Look at Search Console every day for the first fortnight, then once a week for three months. Keep an eye out for:

Crawl errors and sudden jumps in 404 pages.
Pages showing the status “Crawled, currently not indexed”, and pages listed under “Duplicate without user-selected canonical”.
Falls in clicks to particular pages compared with the figures you saved.

A bit of movement in the early weeks is normal while Google works through the changes. A page that drops and has not recovered after roughly a month usually points to a redirect that is missing, content that went astray or an internal link that no longer works.

Keep redirects running for at least a year, and preferably for good. Google’s own advice is a minimum of one year.

Frequently asked questions

How long until rankings recover after a site move?

With good planning, rankings normally settle within a few weeks. A domain change or a big restructure can take two to four months, and a poorly planned move can take far longer.

Should I rewrite my content at the same time?

Ideally keep changes to a minimum. If the content, the URLs and the design all change together, it is hard to work out what caused a drop. Move the site first, then improve it.

Do 301 redirects lose ranking strength?

Google has said 301 redirects pass ranking signals. The bigger danger is redirecting to a page that is not a close match, which can still cost you rankings.

What is the mistake people make most often?

Launching while the “noindex” tag or robots.txt block from staging is still active. That instructs Google to keep the entire new site out of its index. Check for it before launch and again straight afterwards.

Sources and further reading

Want help with this? We handle site moves from start to finish as part of our local SEO service, so talk to us before you fix a launch date. Get in touch.

Oct 2026