Googlebot vs Google-Extended: don’t block the wrong one
Googlebot and Google-Extended are not two names for the same thing. Googlebot is the crawler most businesses mean when they talk about Google Search crawling. Google-Extended is a separate product token for defined Google AI-use preferences.
That distinction matters when a WordPress service business wants its service pages found in Search but is uneasy about a particular AI use. One robots.txt edit made in a rush can address the wrong concern entirely. Bit of a waste of an afternoon, that.
For businesses in Dubai, the UAE, the UK or Liverpool, the practical question is simple: what product or use are you trying to control? Define that before editing a bot rule.
Googlebot handles Search crawling. Google-Extended does not.
Google’s crawler documentation describes Google-Extended as a standalone product token that lets publishers manage whether their content may be used to improve Gemini Apps and Vertex AI generative APIs. Google also states that Google-Extended does not affect a page’s inclusion in Google Search or act as a Search ranking signal.
So allowing Google-Extended is not the switch that makes a WordPress page visible in Google Search. Blocking Google-Extended does not, by itself, remove a page from Google Search either. Search crawling, indexing, ranking and AI-use preferences are related only in the sense that they concern the same public website. Their controls are not interchangeable.
Googlebot versus Google-Extended in WordPress: Googlebot is the documented crawler associated with Google Search access, while Google-Extended is a product token used for a narrower documented AI-use preference. A robots rule for one should not be assumed to govern the other, or every Google AI feature now or later.
Map the instruction to the business decision
Robots rules are blunt instructions. They work best when the owner has stated the intended outcome plainly, rather than pasting a line from an AI-SEO thread and hoping for the best.
| Business preference | Control to review | What not to assume |
|---|---|---|
| Keep public service pages available to Google Search | Googlebot access, page indexability, server response and any noindex instruction | That allowing Google-Extended creates Search visibility |
| Set Google’s documented preference for defined generative AI improvement uses | Google-Extended rule in robots.txt | That it controls every Google AI feature or guarantees no AI appearance |
| Keep an area private, such as a staging copy or client file area | Access control, authentication and production deployment settings | That robots.txt is a security control |
| Check whether AI systems can read important service-page content | Rendered content, crawl access, JavaScript dependency and page structure | That a bot permission alone makes content usable or citable |
The third row catches plenty of WordPress sites. An old staging subdomain is sometimes left publicly reachable, with a robots rule doing the heavy lifting in somebody’s mind. Robots.txt is a crawl preference, not a lock on the door. If the material is genuinely private, it needs proper access restrictions.
A WordPress example with two separate aims
Take an illustrative plumbing and maintenance business with WordPress service pages for emergency repairs, landlord work and commercial contracts. It wants Google Search to continue crawling those pages because search enquiries matter. At the same time, it wants to express a restriction for the particular Google AI use covered by Google-Extended.
Its robots.txt might need rules that keep the public pages available to Googlebot while setting a separate Google-Extended preference. The exact syntax and scope should be checked against current documentation and the site’s wider robots rules. A broad disallow copied above an existing group can quietly affect more than intended, particularly where an SEO plugin, host tool or deployment script also writes robots output.
Check the public file at the live domain rather than trusting a WordPress plugin screen. Confirm the HTTPS version, the preferred hostname and any redirected version. Then inspect whether a CDN, security plugin or server-level configuration is serving a different robots.txt response. It is boring work, but less boring than explaining why a lead-generating page vanished from crawl access after a tidy-up.
Inspect the rules your website actually serves
Before changing anything, record the current live robots.txt file and identify who owns it. On WordPress, that may be WordPress itself, an SEO plugin, a physical robots.txt file in the web root, a host configuration, or a deployment process. Editing one layer while another serves the final response achieves very little.
- Write down the product or use you want to control in one sentence.
- List the relevant documented user-agent or product token separately.
- Fetch the public robots.txt file from the live production domain.
- Check for broad wildcard rules, duplicated groups and conflicting deployment files.
- Verify Search-specific settings separately, including noindex directives, canonicals and whether Googlebot can fetch the important pages.
- Keep a dated copy of the previous rule before publishing a change.
If core content relies heavily on client-side scripts, robots access is only part of the job. A crawler may be allowed through the gate but still receive a thin or delayed page. Our guide to a JavaScript readiness audit for AI search bots covers the practical checks around rendered content and machine-readable page delivery.
Do not promise more certainty than the control provides
There is a temptation to treat one line in robots.txt as a full AI policy. It isn’t. Google’s own documentation defines Google-Extended’s scope, and that scope should be read carefully at the time you make a decision. Do not assume every AI-related Google product uses the same permission, and do not claim that allowing or blocking a token guarantees citations, summaries or future AI visibility.
Where the decision involves search performance, content reuse, technical delivery and internal policy, it deserves a broader review. The difference between an AI readiness audit and an SEO audit is useful background when deciding whether this is a narrow robots-file adjustment or a wider website governance job. For a fuller review of content structure, crawlability and machine interpretation, consider an AI readiness audit for your website.
Questions about Googlebot and Google-Extended
Is Google-Extended a separate HTTP crawler?
No. Google documents Google-Extended as a standalone product token rather than a separate crawler. It can appear in robots.txt user-agent rules, but that does not mean it behaves as a separate HTTP crawler in the same sense as Googlebot. Follow Google’s current documentation for the precise product scope before setting a rule.
Does blocking Google-Extended remove a site from Google Search?
No. Google states that Google-Extended does not affect inclusion in Google Search and is not a Search ranking signal. Search visibility should be assessed through the controls relevant to Googlebot, indexing directives, page availability and the content Google can access.
Should a WordPress site allow every bot by default?
That is a business decision, not a sensible default for every site. Start with the public content you want discovered, the products you are comfortable permitting, and any private areas that need genuine access control. Then apply documented rules carefully and test the live response.