How to Control What Appears in Google SERP?

Google offers website owners various methods to control the content displayed in Google search results.

While most of the time, website owners want Google to index their web pages, there are also situations where it’s necessary to prevent Google from accessing and indexing certain content.

Why control the display of content on Google?

Here are some common reasons for blocking Google’s access to content:

• Limiting Data: To protect sensitive data and only display it to users who visit the website.
• Avoiding Low-Quality Content: To prevent low-quality or spam content from affecting the website’s ranking. For example, paginated content.
• Optimizing Crawling: For large websites, it’s beneficial to focus the Googlebot’s crawling resources on important content. For instance, in a fashion website with extensive attribute filtering, this content can be optimized.

How to Prevent Google from Accessing Content?

If you want the content on your website to not appear in Google search results, you can take the following measures:

• Remove Content: To completely remove content from Google search, the most direct method is to delete it entirely from your website.
• Password Protection: For sensitive or private content, using password protection is an effective method. This not only prevents unauthorized access but also prevents Google from indexing.
• Use Noindex Tags: By adding a noindex directive to the page’s meta tags, you can tell Google not to index these pages. For example:
  • For most search engines: <meta name="robots" content="noindex">
  • Specifically for Google search engine: <meta name="googlebot" content="noindex">
• Robots.txt File: For media files like images and videos, you can prevent Googlebot from crawling through the robots.txt file. Examples include:
  • To block Googlebot from crawling all content on your website: User-agent: Googlebot Disallow: /
  • To block Googlebot from crawling specific directories on your website: User-agent: Googlebot Disallow: /private/
  • To block Googlebot from crawling specific pages on your website: User-agent: Googlebot Disallow: /page.html

How to Handle Sensitive Content Already Indexed by Google?

If sensitive or private content on your website is already displayed in Google search results, you need to take measures to remove it. Two methods are provided:

• Use GSC’s “Remove” Tool: This tool can quickly temporarily remove your web page from Google search results, usually taking effect within a day. Log in to your Google Search Console account, click on “Index” under “Crawling,” select the page you want to remove, and click the “Next” button.
• Permanently Delete Content from Your Website: If you no longer need the content, it is recommended to permanently delete it from your website. After deleting the content, Googlebot will automatically remove the page from the index during the next crawl. You can also prevent Googlebot from accessing and indexing the page by encrypting access or using noindex meta tags.

The choice of method depends on your specific needs. The “Remove” tool only temporarily removes the web page: it will be hidden for about 6 months, after which it may reappear in search results.

FAQ

• Q: Why can password-protected pages prevent Google from crawling?
  • A: Password-protected pages require authentication and session management, which crawlers cannot simulate the interaction process of real users.
• Q: Can using robots.txt rules prevent Google from indexing?
  • A: Yes, but it’s not completely reliable. The robots.txt file is just a suggestion, and Googlebot may ignore it. Even if Googlebot follows the robots.txt file, your website content may still be indexed by Google, for example, through links from other websites or sitemaps.