Google Simplifies Control Over AI Data Usage

Google has revised its Google-Extended user-agent documentation to provide clearer guidance for website owners. This update explains how publishers can manage their content usage by Google's Gemini and Vertex AI models.

What is Google-Extended?

Google-Extended is a user-agent token that allows publishers to control whether Google uses their website data for training future Gemini models and for "grounding" AI responses. Grounding refers to using web data to improve the accuracy and relevance of AI-generated content.

Key Changes and Clarifications

The updated documentation clarifies several key points:

  • Simplified Language: The documentation now uses simpler language, making it easier for publishers to understand the purpose and function of Google-Extended.
  • No Impact on Search Ranking: Google explicitly confirms that using or blocking Google-Extended has no impact on a website's search ranking or inclusion in Google Search. This clarifies previous documentation that only mentioned search *inclusion*.
  • Control Over AI Training and Grounding: The documentation clearly explains that Google-Extended allows publishers to control whether their content is used for both training future Gemini models and for grounding AI responses in Gemini Apps and Vertex AI.

How Google-Extended Differs from Robots.txt

Google-Extended is separate from robots.txt. While robots.txt controls which pages search engines can crawl, Google-Extended specifically manages how crawled content is used for AI training and grounding. Existing guidance on Google Search Central reinforces this distinction, stating that Google-Extended is not a method for managing how content appears in Google Search. For search appearance control, methods like robots.txt should be used.

Google-Extended is not a method for managing how your content appears in Google Search. Instead, use other methods to manage your content in Search, such as robots.txt or other robot controls.

Key Takeaways for Publishers

This update provides greater clarity and control for publishers regarding their content's usage in Google's AI initiatives. By using Google-Extended, publishers can choose whether to contribute to the development of Gemini and Vertex AI without affecting their search visibility.

For more information on unauthorized access and robots.txt, see: Google Confirms Robots.txt Can’t Prevent Unauthorized Access