FouAnalytics exposes overactive Facebook and Google bots on your site

Google Analytics and Adobe Analytics don't show you bot traffic, because they are required by industry standards to filter out bots. That is problematic because it causes large amounts of discrepancies that can't be resolved. For example, Facebook might report 1,000 clicks, but Google Analytics only shows 100 clicks arriving on your site from Facebook. How do you explain the 90% gap? Similarly, Google might report 10,000 clicks, but your site analytics only shows 1,000 visitors arriving from your Google campaigns. How do you reconcile the 90% discrepancy?

Overactive Facebook bots

In the following example, the orange means declared bots (bots that say their name honestly).

In the supporting data, you can easily see why this was marked as orange and not dark red. Not only did the bot declare itself in the HTTP_USER_AGENT:facebookexternalhit, we can also corroborate that it came from Facebook data centers and Facebook IP addresses. The supporting data in FouAnalytics allows you to "see Fou yourself" and not just trust that the HTTP_USER_AGENT is declared properly. Keep in mind false-flag attacks where bad bots pretend to be this Facebook bot or Google bot, etc. So we always check the other supporting data to confirm.

Facebookexternalhit is a legitimate crawler which loads the webpage external to Facebook. It is typically seen when new campaigns are set up. This bot checks to make sure the landing pages to which the ads click are working. But in the time series chart above, you can see that these declared bots are coming to the site continuously, not just during campaign set up. This is way way too much activity from this particular bot. You may not need to do anything about this, BUT just be aware that your current Google Analytics or Adobe Analytics doesn't show you this traffic because they are required to filter it out from view. These are all clicks from Facebook and Facebook may record these as clicks. But your site analytics don't show these visits. That's why there may be large discrepancies between the clicks Facebook reports and what you see on your site analytics. If you had FouAnalytics next to your site analytics, you can see the above and understand why there is a reporting discrepancy.

Overactive Google bots

The next example shows incredibly overactive Google bots. That's the large yellow section in the time series chart below.

Again we can see the name "Googlebot" declared in the HTTP_USER_AGENT. And we can check that these bots came from Google datacenters and the IP addresses are in the ones published by Google itself, starting with 66.249.73.***.

Googlebot typically crawls a site to index the content. Even on major news sites that publish content constantly, the Google search crawler should only represent a fraction of a percent of the total pageviews. In the time series stacked percentage chart from FouAnalytics above, the yellow portion is almost 60% of the traffic, over a multi-week period. This is way way too overactive. We don't know why Googlebot was coming to the site so much, but it appears to have just died down yesterday. Good news is that this surge in yellow appears to be going away.

Again, your current Google Analytics and Adobe Analytics won't show you the Googlebot activity because they are required to filter it out. But having FouAnalytics next to your current site analytics will allow you to see the bot activity on your site and decide whether you need to do something about it. In this case, since it was the official Googlebot, no action was needed. But see the following example of where bad bots are costing you money (eating up your paid ads and clicking on your ads).

Good bots don't fake engagement on your site, bad bots do

In the vast majority of the cases, when we see Facebook bots and Google bots click through to the landing page, these bots don't fake engagement (clicks, scrolls, mouse moves, etc.) on your site. Note the blank click charts below.

But bad bots DO simulate engagement on the landing pages to trick performance marketers and the performance algorithms into thinking these were valuable visitors. Remember the hilarious fake clicks seen below? And remember the clicks arriving from PMax (Google's "performance max")? If you don't have FouAnalytics on your site, you won't see these bots and you won't see the simulated clicks on your site, which trick Google Analytics' new "engaged user" metric. These bots will all look like "engaged users." Performance marketers, be aware. These are not humans who will convert, even though your "performance" reports look great.

Let me know if you'd rather "see Fou yourself" the bots on your site that Google Analytics and Adobe Analytics don't show you, and also what these bots are doing on your site?

FouAnalytics Self-Serve is live now. Sites with less than 100k pageviews per month can use it for free. Sign up here: https://fouanalytics.com/?utm_source=overactive_article