AI Crawler Access: Which Bots to Allow in Your Store's robots.txt

7 min read

The Silent Visibility Killer

You can write perfect product descriptions, publish clean schema markup, and earn third-party citations — and still be invisible to ChatGPT, Perplexity, and Claude if their crawlers can't reach your pages in the first place. Access is the foundation everything else sits on. An AI engine that gets blocked at the door has nothing to cite, and no amount of on-page optimization changes that.

This is the least glamorous part of AI search optimization and one of the most common failure points. The problem is rarely a deliberate decision. It's usually a default setting, a CDN rule someone enabled for security, or a copied-and-pasted robots.txt snippet that blocks more than intended. The fix is straightforward once you understand one distinction that most store owners get wrong.

Training Bots and Search Bots Are Not the Same Thing

The single most important concept here: the major AI companies run separate crawlers for training their models versus powering their live search and answer features. Blocking one has no effect on the other.

Here is how the main players split their crawlers:

  • OpenAI runs GPTBot (collects content for model training), OAI-SearchBot (surfaces sites in ChatGPT's search results), and ChatGPT-User (fetches a page in real time when a user's prompt requires it).
  • Anthropic runs ClaudeBot (training), Claude-SearchBot (indexing for search results), and Claude-User (real-time fetches for user requests).
  • Perplexity runs PerplexityBot (indexes pages for citations) and Perplexity-User (fetches a page when a user's question points to it).
  • Google runs the standard Googlebot, plus Google-Extended, which is a training opt-out token — disallowing it removes your content from Gemini model training but has no effect on your Google Search or AI Overviews ranking.
  • Apple runs Applebot and Applebot-Extended (the training opt-out equivalent).

The strategic consequence is what matters. If you block GPTBot because you don't want your catalog feeding a training set, that's a defensible choice — and it costs you nothing in ChatGPT search visibility, because that runs on OAI-SearchBot. But if you blanket-block everything from OpenAI, you also block OAI-SearchBot and ChatGPT-User, and now your store simply cannot appear in ChatGPT's shopping answers. Those are the bots that send buyers back to your site with a citation and a link.

The clean framework for most ecommerce stores: allow the search and retrieval bots always; decide on the training bots based on your own stance about model training. The search bots are the ones that generate referral traffic. There is rarely a good reason to block them.

How Stores Get Accidentally Blocked

Most blocking is unintentional. The usual culprits:

CDN and security defaults. In July 2025, Cloudflare — which sits in front of roughly a fifth of the web — began blocking known AI crawlers by default on new domains, and it has continued to expand default restrictions and its pay-per-crawl program since. If your store is behind Cloudflare or a similar provider and nobody explicitly opted into AI crawler access, you may be blocking the exact bots you want, without a single line in your own robots.txt. A meaningful share of ecommerce sites are locked out this way and don't know it.

Copied robots.txt snippets. Blog posts from 2023 and 2024 encouraged site owners to block AI bots wholesale to protect content from training. Plenty of stores pasted those Disallow blocks in and never revisited them. Those snippets predate the training-versus-search split and now block valuable search bots along with the training ones.

Aggressive bot-management rules. WAF rules, rate limiters, and "block all bots" toggles in security plugins frequently catch compliant AI crawlers in the same net as scrapers.

One caveat worth stating plainly: robots.txt is advisory, not enforcement. Well-behaved crawlers from OpenAI, Anthropic, Perplexity, and Google document their user agents and honor robots.txt directives. Bad actors ignore it entirely. So robots.txt is the right tool for guiding legitimate AI engines, but it is not a security control — if your goal is to actually stop abusive scraping, that belongs at the CDN or WAF layer, not in a text file.

Configuring It on Shopify

A frequent misconception is that Shopify blocks AI bots by default. It does not. Shopify's default robots.txt disallows the paths you'd expect — admin, cart, checkout, internal search results — and leaves product and collection pages open to compliant crawlers, including the major AI bots.

That means most Shopify stores don't need to change anything to be crawlable. The risk on Shopify comes from two directions: a custom robots.txt.liquid edit that overreaches, or a CDN layer in front of the store. If you do want explicit control, Shopify lets you add a robots.txt.liquid template to your theme and customize the rules. Treat that as an advanced, unsupported customization — a bad edit here can silently deindex your catalog, so test it carefully.

For the same reasons, if you're weighing infrastructure choices, it's worth understanding how your rendering setup affects crawlability. Our guide on Shopify Liquid versus headless for AI visibility covers the tradeoffs. Getting crawl access right is core technical foundation work.

A Sensible Baseline Configuration

For a store that wants maximum AI search visibility and is comfortable with model training, allow everything. For a store that wants search visibility but prefers to keep its content out of training sets, the pattern looks like this:

# Allow AI search and retrieval bots
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Opt out of model training (optional stance)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Keep two things in mind. First, you generally still want to disallow the same low-value paths for AI bots that you disallow for everyone — cart, checkout, account pages — so crawlers spend their budget on product and content pages. Second, blocking training bots is a real tradeoff: some evidence suggests being present in training data helps a brand be recognized as an entity later, so weigh brand awareness against content protection rather than treating the block as free.

Verify, Then Monitor

Configuration without verification is guesswork. After any change:

  1. Fetch your own robots.txt at yourstore.com/robots.txt and confirm the rules match your intent — including whatever a CDN or platform injects.
  2. Check your CDN or WAF settings separately. This is where the real blocking usually happens, and it won't show up in your robots.txt at all.
  3. Watch your server logs for the AI user agents above. Seeing OAI-SearchBot, PerplexityBot, and Claude-SearchBot actually crawling is the only proof access works.
  4. Test with real queries. Ask ChatGPT, Perplexity, and Gemini questions that should surface your products, and note whether you appear and get cited.

Because these crawlers, defaults, and even the identities of the bots keep shifting, this isn't a one-time task. New user agents appear, CDN defaults change, and a security update can re-block you overnight. Fold crawler-access checks into your ongoing AI visibility tracking rather than treating robots.txt as set-and-forget.

The Bottom Line

Crawler access is binary in a way most optimization work isn't: either the AI engines can reach your store or they can't. Before you invest in schema, product data, or content, confirm the door is open. Allow the search and retrieval bots, make a deliberate call on the training bots, check your CDN, and verify with logs and real queries. It's the cheapest high-leverage move in AI search optimization — and the one most stores skip.

Want to see how AI engines perceive your brand?

Get Your Free AI Visibility Audit