Meta robots are HTML elements that tell search engines how to crawl and index your web pages. They can help you avoid indexation issues, such as duplicate content, low-quality pages, or pages that you don't want to rank for. In this article, you'll learn how to use meta robots to control your site's visibility and optimize your SEO.
Meta robots are snippets of code that you can add to the <head> section of your web pages. They consist of a <meta name="robots"> tag and a content attribute that specifies one or more directives. Directives are instructions that tell search engines what to do with your pages, such as follow or nofollow, index or noindex, archive or noarchive, and so on. For example, <meta name="robots" content="noindex, nofollow"> tells search engines not to index or follow the page.
Meta robots is the on-page instruction for search engines crawlers and other bots that would hit your page. It is a meta tag in the <head> section of the HTML in most cases. It can be left off the page and that would be considered "index , Follow". It is normally added to a page to include "no index" or remove from indexation if a bit hits the page. It can also be used to "no follow" which is where the bot would see this as the end of its crawl and not follow other internal links. This doesn't replace the robots.txt and is used in connection with those allow/disallow rules when crawlers or bots hit the domain. The robots meta tag affects only the page it is on. There are other more sophisticated methods to setting the tag.
Meta robots can help you prevent indexation issues that can hurt your SEO performance and user experience. For instance, you can use meta robots to avoid duplicate content by telling search engines which version of the same page is the preferred one and which ones to ignore. Additionally, you can use meta robots to exclude low-quality pages from being indexed and wasting your crawl budget. Furthermore, meta robots can be used to protect sensitive pages such as internal reports, invoices, or user profiles from being exposed to the public.
Meta robots tags listed in the <head> tag of the html in pages are one of the most efficient ways of signaling to crawlers that pages should or should not get indexed or have the links on the page followed or not (amongst several other use cases.) The reason this is the most efficient is because Googlebot (or other crawlers) will crawl the page, see the code in the <head> and then follow the instructions as a directive for each individual page. Google also no longer will follow the meta noindex tag amongst others in robots.txt (not that it was a good place to put them before anyway).
Using meta robots requires adding the <meta name="robots"> tag and the content attribute with the directives you want to apply to your web pages. You can't use multiple contradictory directives, such as index and noindex together. If you want to allow search engines to index and follow your page, use <meta name="robots" content="index, follow"> or leave the tag empty. To prevent search engines from indexing and following your page, use <meta name="robots" content="noindex, nofollow">. Similarly, if you want to prevent search engines from indexing your page but allow them to follow the links on it, use <meta name="robots" content="noindex, follow">. To allow search engines to index your page but prevent them from following the links on it, use <meta name="robots" content="index, nofollow">. Additionally, using <meta name="robots" content="noarchive"> prevents search engines from showing a cached version of your page, while using <meta name="robots" content="nosnippet"> prevents search engines from using a snippet of your page as a description in the search results.
To ensure your meta robots are functioning correctly, you can inspect the source code of your web pages for the <meta name="robots"> tag and its content attribute. Additionally, you can utilize a browser extension or plugin to show meta robots directives on any page you visit. Moreover, crawler tools or site audit tools can scan your website to detect any meta robots issues or conflicts. Finally, Google Search Console and Bing Webmaster Tools can help you view how search engines index your web pages and identify any crawl errors or index coverage issues.
Google Search Console is prob the best way to check if page-level meta robots tags are being interpreted correctly by Google since it’s straight from the horses mouth so-to-speak (and Bing webmaster tools for Bing). By using the inspect tool in Google Search Console / Bing WMT, within 30 seconds it will recrawl the page and show if the meta robots directives are showing up the way you expect. For a more robust check for multiple pages, the tool Screamingfrog is better way to identify what meta robots tags overall are on pages. Finally a basic manual check of the source code by using the inspect tool in chrome or a third party extension or tool like mozbar / semrush / ahrefs are routine way to check what’s there too.
Optimizing your meta robots can be beneficial, and to do so, you should employ some best practices. This includes using meta robots consistently and accurately across your website, avoiding conflicting or contradictory directives, and combining them with other SEO techniques such as canonical tags, hreflang tags, sitemaps, robots.txt, and redirects. Additionally, you should test and monitor your meta robots regularly and update them as needed to reflect any changes or improvements on your website or in the search engine algorithms.