Havoc with Googlebot, Google Analytics, and Phantom Traffic

Bot traffic in Google Analytics or Adobe Analytics

Do your Google Analytics or Adobe Analytics show you the levels of Googlebot traffic on your site? No. Why? It's because those analytics platforms are required to filter out bot traffic, specifically the bots in the IAB list of bots and spiders. That means you can't see that bot traffic in your own Google Analytics or Adobe Analytics. In the 4 charts below, from FouAnalytics on-site measurement, you can see a large amount of yellow, Googlebot pageviews. For smaller sites, this is a very large portion of their traffic and this represents a significant incremental load on these site's servers, even if you can't see it in your current analytics platform.

This is why we recommend adding FouAnalytics to the site, next to GA or Adobe Analytics, not to replace those. You can use FouAnalytics to troubleshoot discrepancies and better understand whether you need to take action or not.

"That means you can't see that bot traffic in your own Google Analytics or Adobe Analytics."

Phantom traffic in Google Analytics

It is well known that Google Analytics has basic vulnerabilities that allow attackers to write false data directly into GA. Most site owners have experienced this as "referrer spam" over the years where bad guys write fake pageviews into your Google Analytics and set the referrer/source to be whatever domain they want (see screenshot below). Most people, when seeing these strange domains, visit the domains to see what they are. This is free advertising for the bad guys selling traffic.

And yes, GA4 is still vulnerable to false data written into it. Bad guys can write anything to any field in Google Analytics. They can fake the source and they can write messages into the language field, as in the following example "Vote for Trump"

Keep in mind this is all "phantom traffic" in the sense that it never occurred. There was no page served from your servers, etc. This was entirely false data written into Google Analytics directly. If you have FouAnalytics next to Google Analytics on the same site, you will be able to see this. For example, if Google Analytics reports traffic, but FouAnalytics does not show any traffic (e.g. from a particular referrer or source) then the traffic did not actually occur.

How is this useful?

SSPs (supply side platforms) use FouAnalytics to vet sites before letting them onto the platform. Historically, I have observed bad guys present entirely falsified Google Analytics numbers to SSPs to convince them they had a lot of traffic and therefore to let them into the SSP. But by having the site put FouAnalytics on the site next to Google Analytics, the SSP can verify not only the quantity of traffic (actual pageviews, not phantom data written into GA), but also the quality of the traffic. For example, some sites were using real bot traffic to load pages and generate pageviews.

Site owners that have FouAnalytics on the site can also see the bot traffic that is not shown in Google Analytics and then use the supporting data to decide whether they need to take action or not. In the following example, note the large surge in dark red (bots) in the FouAnalytics time series chart.

The question is NOT whether this is bot or not. The question is whether any action needs to be taken. The supporting data grids in FouAnalytics (screen shot below) shows that these are obvious bots from U.S. datacenters of Amazon Web Services. The IP addresses are also highly repeated, and from those same data centers.

But the referrer data shows mostly blank (the bots hit the page directly) and the urls confirm there are not utm_source= to suggest the pageviews came from the bots clicking on ads. With this supporting data you don't have to just trust that Google Analytics or Adobe Analytics labeled the sources/referrers correctly. You can independently verify and "see Fou yourself" that it was marked correctly.

Because these pageviews did not come from clicks on ads, but rather the bots directly visiting the site, the site owner did not need to take action -- e.g. block some bad sites and apps in their digital marketing campaigns. If, however, the clicks are coming from a paid media source, like the ones below, with a utm_source= in the url, FouAnalytics automatically parses these out for you. And looking at the color coding under each row, you can see which sources need troubleshooting right away -- i.e. the ones with a lot of dark red.

FouAnalytics helps you answer whether the traffic is bot or not. This is not shown to you in Google Analytics or Adobe Analytics because they filter out bots and don't display them. But more importantly, FouAnalytics helps you answer the business question, whether you need to do something about the bots or not. In the first case above, crawlers were hitting the site directly. These are a nuisance, but did not require remediation. But the bots in the latter example, did click through on ads from various paid channels. So troubleshooting was necessary.

Don't let bots and phantom traffic wreak havoc on your site analytics and your business insights. You are welcome to add FouAnalytics to your sites, next to Google Analytics or Adobe Analytics. FouAnalytics Self-Serve is free for websites under 100,000 pageviews per month.

https://fouanalytics.com/?utm_source=GA_article