Google explains why crawling spikes after a CDN move and how to handle it
A Google Search Central blog post (November 11) explains how Googlebot crawling can spike right after a site moves to a CDN, putting load on the origin server, and what to do about it. Faster responses raise crawl volume, while an empty cache means first requests all reach the origin. Blanket bot protection that blocks Googlebot can cause pages to drop out of search results.
Key points
- Fast, stable responses gradually raise the crawl capacity limit (host load), while slower responses lower it
- A new CDN starts with a cold cache, so the first request for every URL goes through to the origin
- Blanket bot protection that returns CAPTCHAs or 403s stops Googlebot, which backs off, and pages and product listings may start dropping out of Search
Background
A CDN serves content from edge servers around the world to speed up page loads and shield the origin. Google notes that human visitors cluster on a small set of popular pages, while crawlers work through the breadth of a site's URLs, so many uncached URLs can expose the origin to heavy load.
Google's suggestions are to allow verified bots in CDN and firewall settings if you need to block bots, to pre-warm caches by fetching key pages beforehand, and to return a temporary 429 or 503 for a short window (a few hours, or one to two days at most) if the origin needs a breather.
What it means for shops
This matters to small shops and their web agencies who are moving a site or online store to a CDN or tightening bot protection in a security service. A misconfiguration could remove your pages from Google results and product listings and hurt customer acquisition.
What shops should do
- When adding or changing a CDN or security service, ask your developer or server admin whether Googlebot is allowed
🔒 3 more action items for subscribers
Summaries are written by Seiichi Studio based on the original releases. Always check the source for the exact details.