COMMON SENSE bot detection, but only if you have the right (Fou)Analytics

"I'm getting too old for this sh**t." Some of you will remember this quote from a movie; some won't. But I've been doing this -- ad fraud research -- for almost 14 years now. And it's time for you all to take the ball and run with it. Hint: FouAnalytics Self-Serve is available now. Looking at analytics and bot activity is almost second-nature to me now. And I think it will be for you too, once you start digging in and looking at FouAnalytics yourself.

Bot traffic is at least half of the internet

This has been true for at least the last 10 years. So bot traffic is probably more than "half" now. fraud0.com and Piwik PRO recently presented the following slide, showing 45% invalid sessions on their own site. While overall averages are nice, it's not actionable. So what if half the traffic on the internet is bots? The business question that folks want answers to is whether they need to do something or not. For example, in digital advertising, I can definitively tell you that traffic on fake sites is 100% bot traffic. Humans have never heard of sdfvnosdivn48askld .com so no humans visit it. 100% of the traffic on these sites are bots, and they are used to generate pageviews in order to generate fake ad impressions to sell through programmatic exchanges. So what? read on...

Google Analytics and Adobe Analytics don't show bot traffic, why?

GA and Adobe Analytics are required by internet standards to filter bot traffic. This means you won't see the obvious bots, or other bots that these analytics platforms didn't detect. This is highly problematic when your bot problem is like the following. Literally 99% of the traffic (bots, dark red) won't be shown in Google Analytics and Adobe Analytics. This means you won't know if you have a problem, especially when these bots are coming from your paid media, digital marketing campaigns.

So what do you do?

You will continue using Google Analytics or Adobe Analytics on your site. But you can add FouAnalytics next to GA and Adobe. This way, you get additional data that you never had before. With this additional data you can do what I have done for the last 14 years -- common sense fraud and bot detection. Let me show you some examples.

Googlebot - is marked yellow in FouAnalytics because it's a search crawler. It declares itself in the HTTP_USER_AGENT. But as you know from reading my articles over the last 13 years, you shouldn't trust anything that is declared. All HTTP headers are declared, and therefore easily falsifiable. So we look for additional data. For example the GEO_AS which is the name of the data center, in this case AS15169 Google. Googlebot should come from Google datacenters so this checks out. The IP addresses are also highly, and evenly, repeated, starting with 66.249.73.* These are known Google datacenter IP addresses because Google publishers their IP address ranges. So this checks out. And finally, hardwareConcurrency means the number of cores -- the majority (83%) are between 107 and 128 cores. How many cores does your iPhone have, or your laptop? Right, not 128 cores. So common sense will tell you these are servers. And that checks out, since this is Googlebot from Google data centers.

Declared bots - are marked orange in FouAnalytics because they declare themselves to be bots. Dark red is reserved for bad bots that disguise themselves. In the following example of declared bots, you see "HeadlessChrome" in the HTTP_USER_AGENT. These are automated browsers typically used for scraping content from sites. You will also notice "Synthetic monitoring" and "SiteCheck" (bots that monitor sites to make sure they are up) and "AdsBot-Google. This is a different bot from Google meant to check ad creatives and click through on the ad to make sure the landing page is active. A friendly reminder, you get none of this data from legacy bot detection vendors, nor from your current analytics platform.

Bad bots - are marked red in FouAnalytics because they disguise themselves and don't declare themselves to be bots. Some of these bots are obvious, easily identified as bots. The following example shows that bots don't just come from China, India, or Russia any more. They come from Amazon data centers in the U.S. You may have noticed a lot of visitors from Ashburn, Virginia, for example, in your Google Analytics. Those are bots from Amazon data centers in Ashburn, Virginia (3rd row of the GEO_CITY data grid below).

Other bots are less obvious, because there are many tricks that bots use to disguise themselves. The following example are bots (fake users) clicking through to the site from PMax ("performance max") campaigns. Not only are 84% of the visitors dark red (bots) but these bots ALSO faked clicks on the landing page. The click maps on the right side shows the click locations. Common sense will tell you these are not normal click patterns, how humans normally navigate around sites. The larger circles in red means clicks in the same location. Common sense will also tell you most humans don't click the very left edge of the screen, as seen in the 4 vertical click maps below.

In addition to measuring bot traffic on sites, FouAnalytics also measures programmatic ads - display, video, CTV. The following example is from display ads -- specifically 320x50 ad units (green rectangle). Notice the clicks are spread out across the face of the entire 320x50 ad unit.

Common sense will tell you that humans have to touch the screen on their smartphones in order to click something. In the touch charts below, from FouAnalytics in-ad measurement, we can indeed see the corresponding touch events that go with the click events seen above. This is how we can confirm it is a real click on the ad.

Once you have the right analytics in place -- e.g. FouAnalytics -- you can "see Fou yourself" that common sense can help you identify what makes sense (real clicks on ads) versus what doesn't make sense (clicks with no touch event). You will also soon realize many many other things you can verify for yourself, with simple common sense -- e.g. 1) on websites, humans can't click something without first moving the mouse to it, 2) clicks that don't have "mousedown" and "mouseup" events are faked by algorithms, 3) humans don't move their mouse in perfectly straight lines, 4) humans can't hold their phones perfectly still when using them, etc. I am sure you're able to tell what the users were trying to click on in the following 300x250 ad units?

Right, the close button [x]. The users were trying to close the ad not click the ad. The concentration of green dots at 300x0 tell you this. You can "see Fou yourself" and use common sense to detect bots and fraud, as well as confirm humans, and real clicks.

Happy hunting y'all.

For more screen shots and examples from FouAnalytics, here are 750+ more articles: https://www.dhirubhai.net/in/augustinefou/recent-activity/newsletter/