Quick answer: In 2026, a business can allow AI search discovery while choosing separately whether its public content is available for model training. OAI-SearchBot is associated with search in ChatGPT, while GPTBot is associated with potential model-training collection. Google’s Google-Extended is a separate product control for certain Gemini uses and does not govern ordinary Google Search inclusion. On 30 October 2026, Google announced plans to support a shared AI-training preference in robots.txt in 2027. That future control is a proposal, not something to assume is operational today.
For a Delhi NCR service business, the practical goal is to keep genuine service pages crawlable, protect private URLs with actual access controls, and make deliberate choices about search discovery and AI training. This guide covers the technical policy decisions. For broader visibility strategy, see our technical SEO services overview.
Why AI crawler rules matter for websites in November 2026
A traditional search-engine audit asks whether Googlebot can reach a canonical page and whether that page is eligible for indexing. AI services add a second question: what purpose is the automated visitor serving? Some crawlers discover content for answer-oriented search, others collect material that may be used to train models, and others retrieve a URL because an individual user requested it.
These purposes are not interchangeable. A business might want its Delhi website development guide discoverable in ChatGPT search while choosing not to make the same content available to a training crawler. Conversely, a publisher may decide to allow both. Neither policy automatically improves rankings, leads, recommendations or AI citations.
AI crawler comparison: which user-agent controls what?
| Token | Documented role | Practical decision |
|---|---|---|
| Googlebot | Google Search crawling, including the Search foundation used for AI Overviews and AI Mode | Allow important public pages if you want Search eligibility. |
| Google-Extended | Google product control for certain Gemini training and grounding uses; not a separate crawling user agent | Choose according to your Gemini-use preference; it is not a Google Search ranking control. |
| OAI-SearchBot | Discovery for ChatGPT search features | Allow when ChatGPT search discovery is a goal. |
| GPTBot | Collection of content that may be used for model training | Allow or disallow independently of OAI-SearchBot. |
| ChatGPT-User | User-triggered retrieval, not routine search indexing | Robots rules may not apply to user-requested fetches; do not treat it as a privacy boundary. |
| Bingbot | Microsoft Bing web search crawling | Keep relevant public pages accessible for Bing discovery. |
These roles are documented by OpenAI’s crawler reference, Google’s crawler documentation and Bing Webmaster Tools. Recheck the providers’ current policies before changing a production file.
What changed on 30 October 2026: Google’s shared AI-training proposal
Google announced that it plans to support a shared AI-training preference signal developed through the IETF’s AIPREF work. The idea is to let publishers communicate a broad training preference across participating providers, rather than maintaining only vendor-specific instructions. Google said that, where the shared opt-out is present and Google-Extended is not explicitly specified, it plans to interpret that signal as a disallow for Google-Extended.
Important timing: Google’s announcement says it aims to support the new mechanism with other providers in 2027. As of this article’s 1 November 2026 publication, do not paste an unverified future token into production and assume that it works. Existing documented provider-specific rules remain the practical starting point. Read Google’s 30 October 2026 announcement for the exact status and future updates.
The announcement concerns training-use preferences, not a universal way to appear in AI answers. It also does not replace a site’s security, licensing or privacy obligations.
Does blocking GPTBot remove a site from ChatGPT search?
No, not by itself. OpenAI documents separate settings for GPTBot and OAI-SearchBot. A publisher can allow OAI-SearchBot while disallowing GPTBot. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although they can still appear as navigational links. It also notes that search changes after a robots.txt update may take around 24 hours to be reflected.
This separation is especially useful for a professional-services site publishing original research, product specifications, pricing explanations or case studies. The choice is a business policy decision, not an SEO trick. If you want AI search discovery, check both the robots.txt rules and whether your firewall or CDN is blocking the provider’s verified crawler IP ranges.
Does Google-Extended control Google AI Overviews or AI Mode?
No. Google says Google-Extended does not determine inclusion in ordinary Google Search and is not a Search ranking signal. AI Overviews and AI Mode are Google Search features; their supporting links depend on the normal Search crawling and indexing foundation. Blocking Googlebot can therefore undermine Search eligibility, while changing Google-Extended is a separate decision about certain Gemini uses.
Google’s AI features guidance says there are no extra technical requirements to appear in AI Overviews or AI Mode beyond Search eligibility. An eligible page must be indexed and permitted to show a snippet; even then, inclusion is not guaranteed. For local companies, maintain useful content, truthful business details and accessible page text rather than relying on a special ‘AI ranking’ robots directive.
Example robots.txt: allow search discovery, opt out of selected training crawlers
The following is an illustrative policy for a site that wants Google and Bing Search access and ChatGPT search discovery, but chooses to disallow GPTBot and Google-Extended. It is not the only valid choice, and a Google-Extended opt-out can also affect eligible Gemini grounding use. Preserve the site’s existing disallow rules and sitemap declarations when adapting it.
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com/sitemap.xmlReplace the example sitemap with the actual canonical sitemap URL. If your business prefers to allow Gemini training and grounding uses, do not copy the Google-Extended disallow. If you also want to allow GPTBot, remove its disallow. An explicit policy is easier to audit than copying a viral ‘AI robots.txt template’ without understanding its consequences.
What robots.txt cannot do: privacy, noindex and guaranteed visibility
robots.txt is a crawl instruction, not an authentication system. It does not protect a CRM dashboard, private quotation, unpublished proposal or customer record. Sensitive resources need real authorization, appropriate response codes and server-side access control. A disallowed URL may still be discovered through external links, and a crawler blocked from fetching the page cannot read an HTML noindex directive on that page.
For public pages that must be excluded from indexing, use the relevant supported noindex method while allowing the crawler to retrieve the directive. For confidential pages, require authentication rather than exposing them and hoping that bots obey a text file. Review our website maintenance and security services if the site’s crawling rules are mixed up with access-control requirements.
WordPress implementation: where to edit and what to test
- Open the public site’s root robots.txt URL and save the current contents before changing anything.
- Determine whether robots.txt is a physical server file, a WordPress-generated virtual file or managed by a CDN or SEO plugin. Avoid editing a second copy that never reaches visitors.
- Record the business decision for each crawler: search discovery, training collection and user-triggered retrieval.
- Keep important service pages, the homepage and published blog articles crawlable by search bots where appropriate.
- Preserve existing asset access and sitemap declarations. Do not accidentally block JavaScript or CSS required for rendering.
- Publish the change and test the live HTTPS response, not merely the plugin preview.
- Check the server and CDN logs for unexpected 403, 429 or challenge responses to legitimate crawlers.
WordPress installations can be affected by caching plugins, security tools and CDN bot rules. A robots.txt rule that looks correct in the editor is not proof that the live site is serving it correctly.
Headless WordPress and Next.js: the public domain is the source of truth
For a React or Next.js frontend using WordPress as a headless CMS, the public React domain’s robots.txt is the relevant file for public article URLs. A robots.txt file on the WordPress backend does not automatically control URLs served from a different host. Check both hosts separately, because their intended indexing policies may differ.
For example, a private WordPress backend may reasonably be excluded from indexing while the public Next.js site remains crawlable. The frontend also needs correct canonical URLs, rendered text, working article routes, appropriate metadata and a discoverable sitemap. If posts publish in WordPress but the React site remains stale, changing robots.txt will not repair that sync problem. See our published guide to structured data on headless sites for a related example of why backend metadata must appear on the public frontend.
How to audit AI crawler access without making assumptions
| Check | Evidence to collect | What it proves |
|---|---|---|
| Live robots.txt | HTTP status, content, canonical host and effective user-agent groups | Which instructions the public host currently serves. |
| CDN and firewall | Challenge, block and rate-limit logs by verified IP | Whether legitimate crawler requests are being rejected at the edge. |
| Page fetch | Public 200 response, rendered text, canonical, robots meta | Whether a specific URL can be fetched and interpreted. |
| Search Console | URL Inspection and indexing report | Google’s reported indexing status, when available. |
| Server logs | Timestamp, user-agent, request path, response and IP verification | Whether a documented bot actually requested the site. |
| AI answer tests | Recorded query, date and visible citations | A sample observation, not a ranking guarantee. |
A user-agent string alone can be spoofed. When reviewing suspicious traffic, compare source IPs against the provider’s published IP ranges and the relevant reverse-DNS or verification guidance. Do not label every request containing ‘bot’ as a verified search crawler.
Seven-day action plan for a Delhi NCR service website
- Day 1: Inventory the live WordPress, React and CDN hosts and their robots.txt responses.
- Day 2: Agree on the company’s search-discovery and training-use preferences.
- Day 3: Review Googlebot, Bingbot, OAI-SearchBot, GPTBot and Google-Extended rules against official documentation.
- Day 4: Update the authoritative public robots.txt file and preserve necessary existing directives.
- Day 5: Validate public service pages, article URLs, sitemaps, canonicals and noindex signals.
- Day 6: Review firewall logs for legitimate crawler access and false-positive blocks.
- Day 7: Document the policy, monitoring owner and a quarterly review date, including Google’s proposed 2027 change.
For a business serving Delhi, Noida, Gurugram or Ghaziabad, this audit should support useful public service information, not create fake location pages or claim special access to AI recommendations. Our Delhi NCR local SEO overview explains the separate work required for local discovery.
Frequently asked questions about AI crawlers and robots.txt
Should I allow OAI-SearchBot for ChatGPT visibility?
If ChatGPT search discovery is part of your business goals, OpenAI recommends allowing OAI-SearchBot and requests from its published IP ranges. This supports eligibility for discovery; it does not guarantee a mention, citation or customer enquiry.
Can I block GPTBot but still appear in ChatGPT search?
Yes. OpenAI documents independent controls for GPTBot and OAI-SearchBot. Disallowing GPTBot is a training-use choice; OAI-SearchBot is the relevant automatic search-discovery crawler.
Does Google-Extended block my website from Google AI Overviews?
No. Google-Extended is a separate product control and is not the mechanism for Google Search indexing or AI Overviews eligibility. Googlebot access and normal Search requirements still matter.
Is Google’s new shared AI-training robots.txt control active in November 2026?
Google announced its plan on 30 October 2026 and said it aims to support the mechanism in 2027. Do not assume an implementation is live until the provider confirms it and publishes instructions.
Can robots.txt stop ChatGPT-User from fetching a private page?
Do not rely on it. OpenAI says robots.txt rules may not apply to user-triggered ChatGPT-User requests. Protect private information with authentication and server-side authorization.
Do I need a separate robots.txt for WordPress and Next.js?
Each host has its own root robots.txt. If the public blog is on a Next.js domain and the WordPress backend is on a subdomain, audit the public and backend hosts separately. Their intended indexing policies may differ.
Will allowing AI crawlers improve rankings or generate more leads?
No specific ranking or lead result is guaranteed. Allowing a crawler removes one possible access barrier. Useful content, correct indexing, relevance, trustworthy information and the search provider’s systems still determine visibility.
How often should businesses review AI crawler policies?
Review them after major website migrations, CDN changes and provider policy announcements, and schedule a periodic audit. The October 2026 shared-control proposal is a good reason to revisit the policy when implementation details are published in 2027.
Official references and next steps
Source-check this guide against OpenAI’s crawler documentation, Google’s AI Search guidance, Google’s 30 October 2026 announcement and Bing’s indexing guidance. The providers may update their policies; verify before production changes.
Need a technical SEO and AI-crawler access audit for a WordPress or Next.js business site? Keshav Enterprises can review the live robots.txt response, crawl barriers, public React rendering, canonical URLs and sitemap behavior. Contact the team with your website and the specific pages you want to keep discoverable.
