How DOM.RIA.COM Doubled Their Googlebot Visits Using JetOctopus

DOM.RIA.COM is a Ukrainian real estate marketplace. Thousands of ads for the sale and rent of real estate are added to the site every day. As the site structure is large and complex, to ensure the freshness and relevance of content appearing in Google search, there was a need to maximize the crawl efficiency of the site by search robots.

The term “search robot” (sometimes also referred to simply as a robot or “spider”) refers to any program that automatically detects and crawls sites by following links from page to page. Google’s main crawler is called?Googlebot.

What was the task?

Real estate is an industry that depends on geography and location. Accordingly, pages have a vertical hierarchy (For example, city ? district ? micro-district ? metro station and so on). And so, the promoted directories are not always within three clicks from the main one, which, in turn, makes it difficult for search robots to find and scan such directories.

Having studied semantics, the SEO team of dom.ria.com decided to expand the structure with such directories:

• real estate on city streets
• real estate on city lanes
• real estate by house number in the city

But it was not without problems. When the directories were first created, the pages were poorly indexed, and Googlebot rarely visited them.

To make it easier for crawlers to access these pages of the site, dom's team created an updated sitemap and placed it in the footer. It looks like a neat list of links.

dom.ria.com Sitemap

It may seem that the problem has been solved as all links are within 2-3 clicks from the main page. Not yet.

Since DOM.RIA is a sizable website, therefore, the sitemap is rather big — it contains several tens of thousands of pagination pages. But the bot, rather stubbornly, didn’t visit the pages on the?sitemap pagination.

Here JetOctopus.com became very useful. Dom.ria.com has been cooperating with JetOctopus for 3 years already and JO log analyzer was the best helper in solving this task to understand Googlebot's behaviour after implementing the experiments.

About JetOctopus.com:

• Tools for On-page Tech SEO analysis (interlinking, duplicates, slow pages, thin pages, Page Speed Insights, JS crawl etc.)
• Logs Analysis reflects how Googlebot behaves at your website, which part of a website is visited and not. And why.
• GSC module reflects SERP efficiency in dynamics in many vectors. (Cannibalisation, keywords analysis etc.)
• Responsive and caring support team.

So back to the case in 15 days, the bot visited:

• 525,000 street pages in total
• Only 10,000 metro pages
• Total of 196,000 district pages

Logs of bot visits to street directories (visualised with JetOctopus)

And only 16 pages from the sitemap were visited by the bot in 15 days.

Logs of visiting the sitemap by a bot (visualised with JetOctopus)

This was simply not good enough — these numbers were very little, given the geography of Ukraine.

Rethinking internal linking

With logs analysis using?JetOctopus, dom's team of SEOs quickly knew that the search bot tends to visit the pages of cities, regions, and the like. For example, over the same period of time, Googlebot visited 1,048,000 real estate catalogs.

Logs of bot visits to real estate catalogs (visualised with JetOctopus)

Based on this insight, they decided to create mini-sitemaps on each directory for the city, including URLs for streets, districts, metro, etc., which were poorly scanned.

They assigned them meta tags, and also added links to pages that Google often visits.

An example of how the pages were linked

At the same time, they removed the original sitemap.

How have bot visits changed?

15 days after the changes, over the same period (15 days), the team saw the following results:

• A total of 1,398,000 street pages (+ 266%)
• Only 23,000 metro pages (+ 230%)
• Total of 450,000 district pages (+ 230%)

Logs of the bot visiting street pages after the experiment (visualised with JetOctopus)

This is a big win for DOM.RIA.COM after implementing a simple but smart experiment with interlinking and sitemap.

I'm Julia, CEO of JetOctopus. Let's have a call and I'll show you some examples of how to boost your site indexability, optimise internal linking, find keywords opportunities and much more. Pick up the best day and time.