Googlebot traffic doesn't show up in Google Analytics but does in FouAnalytics.

In FouAnalytics, yellow is used to designate search crawlers (bots), so they are separated from orange (declared bots) and red (bad bots). In the FouAnalytics chart below, you can see the large surge in yellow (Googlebot) starting around October 5. Note also the top third of the time series chart, where the shape of the red, blue, and yellow lines are marked by the colored arrows. You can see the blue line goes up and down; this is human activity on the site which goes up during waking hours and goes down at night when humans are sleeping. The red line is practically straight across, representing the same amount of bot traffic hour after hour. And the yellow line surges after October 5 and is slowly declining, but remains a large percentage of the traffic (the large yellow area in the stacked percentage part of the chart, the middle). And the green volume bars also shows the sustained Googlebot traffic.

How do we know it is Googlebot? Because in this case you can see Googlebot declares itself in the HTTP_USER_AGENT. How do we know it is not a fake Googlebot -- i.e. "false flag" by mis-declaring the name in the user agent? Look at the supporting data grids from FouAnalytics below. These were all from the U.S. They were from Google's data center. The IP addresses were all in the known ranges, starting with 66.249.73 .xxx, and the number of cores is mostly 128. How many mobile phones have 128 core processors?

Furthermore, these bots did not fake any activity on the site, like bad bots sometimes do. When isolating just these yellow search crawler bots -- label:equals(search) -- in FouAnalytics, you can look at the clicks tab and notice there's 0% clicks, 0% mouse move, and 0% touch events, even though these were declared to be mobile devices.

Given all of the above, we know that these are Googlebot. But these bots don't appear in Google Analytics ("GA"), because GA is required to filter out G-IVT ("general invalid traffic") to comply with IAB standards. So the large amount of Googlebot traffic (larger than normal) doesn't appear in Google Analytics but is recorded and seen in FouAnalytics. FouAnalytics does not discard any traffic; we record everything so you can see it and decide if something needs to be troubleshot. In this case, by looking at the urls loaded by Googlebot, it was clear that it was doing its job, indexing pages. So there is no further action needed.

One more pro tip for practitioners. If you are screen shotting from FouAnalytics, the following may be helpful. Since this Googlebot traffic is overwhelming, you can choose to not display it in the time series chart. Look for the gear icon on the right side, click it to reveal the list of checkboxes for the different types of traffic. You can UNcheck the search one in this case and see the time series chart without the overwhelmingly large amount of yellow. The bottom version of the example below shows that the rest of the traffic appears to be consistent and repeatable.

What other kinds of bots don't show up in Google Analytics, Adobe Analytics, Piwik Pro, or verification vendors' reports, that you can "see Fou yourself" in FouAnalytics?

Subscribe for more case examples and screen shots from FouAnalytics -- https://www.dhirubhai.net/in/augustinefou/recent-activity/newsletter/