Web Bot Auth: Why Unsigned AI Agents Are Getting Throttled on Your Store

The User-Agent String Is Losing Its Job

For years, the entire access-control conversation for AI crawlers ran on a single assumption: that a bot tells you who it is, honestly, in its user-agent header. GPTBot, PerplexityBot, ClaudeBot — you read the string, you match it against a robots.txt rule or a WAF condition, and you allow or block accordingly.

That assumption was always weak. A user-agent string is self-declared text anyone can type. Scrapers have impersonated Googlebot for as long as it has existed, and the only defense was reverse DNS and published IP ranges — brittle, and useless for agents on rented cloud browsers with no stable address.

The replacement is cryptographic. Web Bot Auth, an emerging IETF draft standard, lets an agent prove its identity on every single request by signing it with a private key whose public counterpart is published at a known URL. The receiving platform verifies the signature before deciding how much access to grant. And in 2026 this stopped being a standards-track curiosity and started shaping how much of your store AI agents can actually read.

What Shopify Changed, and When

On May 7, 2026, Shopify's developer changelog announced stricter rate limits for bots and agents hitting the Storefront API and Shopify-hosted online store pages, effective May 30, 2026. The core mechanic is simple and worth stating precisely:

• Bots and agents that do not sign their requests are subject to the strictest rate limits Shopify applies.
• Operators who want higher limits are told to sign their requests with Web Bot Auth.
• Operators needing more than the standard signed allowance can request an increase through a Shopify form.
• Merchants who need to crawl their own store can retrieve ready-to-use Web Bot Auth signatures from the Shopify admin.

Read that list again as a store owner, because the framing matters. This is not a blocking policy and it is not something you toggle. Shopify applied it at the platform level, across stores, and the party who has to act is the bot operator — not you. Your visibility outcome is decided by whether the agents that matter to your store have done their homework.

How the Mechanism Actually Works

You do not need to implement this yourself unless you operate a crawler, but you should understand it well enough to debug it. Web Bot Auth builds on HTTP Message Signatures (RFC 9421):

1. The operator generates an Ed25519 key pair.
2. The public key is published as a JWKS document at /.well-known/http-message-signatures-directory on the operator's own domain, over HTTPS.
3. Every request carries three headers: Signature-Input (which components are signed, plus a key ID, timestamps, and a nonce, tagged web-bot-auth), Signature (the signature itself), and Signature-Agent (pointing at the key directory).
4. The receiving edge or platform fetches the directory, verifies the signature, and now knows — not guesses — which operator sent the request.

The tagged, timestamped, nonce-bearing signature is what makes this unspoofable in a way user-agent matching never was. An attacker can copy your headers, but cannot produce a valid signature without your private key.

Adoption has real infrastructure weight behind it. Cloudflare, Amazon, Akamai, and OpenAI are among the backers, an IETF working group was chartered for the standard in 2026, and Cloudflare has folded message signatures into its Verified Bots Program — operators applying with well-formed signatures get prioritized for approval, replacing the old IP-range validation dance. On the agent side, OpenAI signs requests from its agent cloud browser, and platforms like Browserbase and Manus sign today. Coverage is still uneven: general search indexing crawlers have been slower to sign than the newer user-directed browsing agents.

The Part Most Stores Get Wrong: Your Own Tools

Here is the practical failure mode nobody warns you about. When a platform throttles unsigned automated traffic, it does not only throttle AI companies. It throttles your automation too.

Think about everything hitting your storefront on a schedule:

• Rank and AI-visibility trackers that fetch your product pages to check content and schema
• Feed generators and channel sync tools that read collection pages
• Uptime, price, and stock monitors
• Site auditors, broken-link checkers, and schema validators
• Custom scripts someone on your team wrote two years ago

None of these are signed. Under strict rate limits they get slow, incomplete, or silently partial results — and a monitoring tool returning partial data is worse than one returning an error, because it reports success. If your schema validator now samples 40 products instead of 400, you will not notice the 360 it skipped.

This is exactly why Shopify put merchant signatures in the admin. If you crawl your own store, get those signatures and use them. If a third-party tool crawls your store on your behalf, ask the vendor whether they sign — it is a fair and increasingly necessary procurement question.

What This Means for AI Visibility Strategy

Three consequences follow, and they change how you think about crawler access.

Access control is moving to identity, not intent. robots.txt still declares what you want; it has never enforced anything. The layers that actually enforce — your CDN, your platform — are shifting to verified identity. That is a net win for stores that want AI traffic, because for the first time you can allow a specific agent with confidence instead of allowing a spoofable string. It also means your policy needs to live where enforcement happens. We covered that shift in detail in Cloudflare's AI crawler defaults, and the robots.txt fundamentals still apply as the declaration layer.

Verified identity is a better allowlist primitive. If your store sits behind Cloudflare, verified bot traffic is labeled at the edge and addressable in WAF custom rules and rate-limiting policies via the cf.verified_bot_category field. That lets you write rules like "allow verified search and agent categories, throttle everything unverified" — a policy that was simply not expressible when the only input was a text header.

Throttling is a visibility problem that looks like nothing. A blocked crawler produces a clear signal: a 403, a missing page, an obvious gap. A throttled crawler produces a thin crawl. The agent gets your product page but not the variant list, the reviews, or the shipping details, and then answers a shopper's question from an incomplete picture. Nothing in your logs screams. Your product just shows up in AI answers with less detail and less confidence than a competitor whose pages got read fully.

The Audit

Concrete steps, in order of payoff:

1. Inventory every automated system that touches your storefront, including internal scripts. For each one, answer: does it sign requests, and has its success rate or completeness changed since late May 2026?
2. Pull your merchant Web Bot Auth signatures from the Shopify admin and wire them into anything you control that crawls your own store.
3. Ask your tooling vendors whether they sign with Web Bot Auth. Treat a blank stare as a data-quality risk in whatever that tool reports.
4. Move your AI access policy to the enforcement layer — CDN bot rules keyed on verified categories, not just robots.txt directives.
5. Compare crawl depth, not just crawl access. Check that AI agents are retrieving your full product data, not the first fragment before the limiter kicks in.

Signed identity is the direction the whole web is heading, and for ecommerce it is mostly good news: an internet where you can reliably tell a real shopping agent from a scraper is an internet where you can be generous with the former. But the transition period punishes stores that assume nothing changed. If you want a systematic read on whether AI engines are getting complete access to your catalog, that is the core of a technical foundation review, and part of what an AI visibility audit is designed to surface.