How do you leverage sitemaps and robots.txt files for crawling and indexing?

An accurate sitemap & robots.txt file is essential. I'm running a research project 18 months plus, testing the crawling/rendering while also reading server logs to identify crawler agents and the type of files they crawl. When a new page is either discovered by google "out in the wild" or on the submission of a sitemap, there are three (3) crawler agents that usually are sent and each has a specific mission. The first is a non-chrome desktop crawler which crawls the sitemap and then immediately seeks out the robots.txt. The next group of crawler agents that arrive are chrome-builds - one as a desktop and another as mobile. Each of these will perform a discovery crawl of the html plus all the scripts and other content that makes up the page.

These both save time and let search engines know what it is people should be seeing on your site. They help your crawl budget and ensure that search engines can more frequently crawl your site.

If you are using an SEO plugin, you likely already have a sitemap created. Such plugins as YOAS, All in One SEO, or RankMath are good examples. Check the settings and decide what kind of content you want to be in the sitemap (pages/posts etc.) and what should be kept out of it (e.g., specific pages or images). Submit your sitemap to Google Search Console/Bing Webmaster Tools.

I typically use a tool called Screaming Frog to create XML sitemaps. Here's how: 1.) Open Screaming Frog and enter the website URL you want to create a sitemap for. 2.) Once the crawl is complete, navigate to the "Sitemaps" tab. 3.) Click on the "Create XML Sitemap" button. 4.) Configure the sitemap settings, such as the sitemap file name, URL inclusion/exclusion rules, and frequency/last modified date. 5.) Click on "Create Sitemap" to generate the XML sitemap. 6.) Once the sitemap is created, you can save it to your computer or upload it to your website's root directory.

In some ways, in 2023, basic XML sitemaps are not as necessary as in the past (just a list of URLs). For most sites, Google can pretty much find all your content - nav, links, backlinks, HTML sitemaps, etc. So XML can be redundant. That said, if you have large enterprise sites that auto-generate tons of pages where some might be orphans, then XML sitemaps can help. I would still create them for all sites, just not always make it seem High-Priority and vital. Other more advanced features of XML sitemaps like frequency and including images and videos can be a great enhancement over basic ones. If you go this route, then XML sitemaps have a distinct function and can help a lot.

XML is great for ensuring that Google indexes your SEO content. The more detailed, the better. For example, to rank in "how to x" Google searches, include the relevant chapters in your video XML. Once created, click "Add a New Sitemap" in Google Search Console. Make sure you find a way to automate sitemap generation, like YaySEO.

Many, many people think blocking content in Robots.txt means Google won't index it. Robots.txt is a CRAWLING directive, not an Indexing one. So I'll see clients block a section of their site, and even put noindex tags on the individual URLs. But months later, a simple site:search shows that content is still in Google's index. Because you told Google NOT to crawl it, so it can't "see" the Noindex tag, and thus keeps the content in its index.

Just this morning I used the Google Search Console robots.txt tester to successfully test a robots.txt file. I then made changes to the file, pasted into the tester and confirmed there were no errors and that Googlebot was allowed. However, upon trying to submit the changes, the tester keeps reporting 404 file not found. The robots.txt file can be successfully browsed and other testers report success with no errors. I'm assuming there's some delay after making the changes but seems a bit wierd?

In my forensic SEO work, often I will check sitemaps where the non-secure designation is used (https:// on a site that is https://) or where a site programmed WITH a trailing slash at the end of the URLs has a sitemap of URLS without the trailing slash. Both of these translate to a sitemap filled with 301 status pages. This is one the most common error I've found and correcting it is the easiest way to avoid crawl budget waste.

For those managing large websites, here's a strategy that can significantly enhance your SEO efforts: Utilize multiple sitemap links in your robots.txt file. This approach is a best practice and helps organize your content and ensuring that search engines crawl and index your site more effectively. While it might be tempting to host a sitemap on a different domain for various reasons, linking from robots.txt will not work. If you find yourself in a situation where you must host your sitemap on another domain, verify ownership of both the domain where your sitemap is hosted and the domain of the URLs listed in the sitemap in Google Search Console (GSC). Then, manually submit your sitemap directly through GSC.

I think sitemaps are invaluable but need not always be XML. An HTML sitemap can be used in most cases. Testing these things is key, as always when dealing with Google's bots.

The easiest way to measure impact of your sitemaps is by reviewing the traffic to your pages. You can do this either in the Google Search Console or with your website hosting company - Wordpress is great for tracking detailed usage statistics and Similarweb will let you know whether your users are coming from Google.