Analytics that report bots on your site - FouAnalytics

Most sites use Google Analytics. Some sites use Adobe Analytics. Neither of these packages show you the bots hitting your site, let alone differentiate which are good bots that you want to visit your site (like search crawlers) versus which are bad bots, like the ones that scrape and steal your content.

Good bots, bad bots, and humans labeled by FouAnalytics on-site

In the series of 10 charts below, you can see that bot traffic comes in all "shapes and sizes." For those who have not seen FouAnalytics dashboards before, blue means humans, red means bots. Orange means good bots that honestly declare themselves to be bots. Yellow means search crawlers like googlebot, bingbot, etc.

FouAnalytics is free to use for websites that serve less than 1 million pageviews per month. Site owners can install FouAnalytics on their websites, wordpress blogs, shopify commerce pages, etc. by copying and pasting 2 lines of code. You can also place the FouAnalytics on-site tag into Google Tag Manager (GTM). Just note that GTM is blocked by most ad blockers, which means any tags inside GTM will also get blocked entirely. FouAnalytics is not blocked by ad blockers and most privacy enhancing tracker blockers do not block FouAnalytics because FouAnalytics is privacy preserving. FouAnalytics never uses cookies; it never collects PII (Personally Identifiable Information); it only collects anonymous javascript parameters from the browser; and FouAnalytics conforms with GDPR.

Where did the bots come from? Do I need to take action?

Knowing THAT bots are coming to your site is useful. But knowing WHERE the bots came from will give you clues about whether you need to take action against them or not. For example, if you are spending money on digital ads, of all kinds like search ads, display ads, social media ads, etc. you might want to check if the clicks you are getting from those paid media sources are bots or humans. If you have FouAnalytics installed on your website and/or landing pages and if you have proper tracking codes like utm_source, utm_medium, and utm_campaign in your click through urls, the FouAnalytics platform automatically parses those for you and summarizes them in data grids like the ones in the screen shot below.

You will be able to tell that 28.8% of the pageviews have UTM_SOURCE (yellow highlight above) which implies around 29% of your site traffic came from some form of paid media. The 20,263 means the number of data points that were loaded into the dashboard in order to populate the data grids. The data in the grid itself is the breakdown of the values observed. For example, out of 5,827 pageviews that have UTM_SOURCE, 2,187 have a value of "facebook" (that comes from urls that contain UTM_SOURCE=facebook). 2,187 is 37.5% of the 5,827. The second value in the example above is UTM_SOURCE=gclid (google click ID) -- 1,912 pageviews, which is 32.8% of the 5,827. The top 10 values are displayed, sorted by highest percentage first.

By looking at the color-coding right under each row, you can see that the row with "criteo" has a lot more dark red than the rows with facebook and gclid. That tells you that criteo is one of the sources of the bot traffic on your site. Since you are paying for media campaigns with Criteo, that is where you can focus your attention and troubleshoot further. Both Facebook and gclid (google search) in this example appear to have a lot of dark blue and very low dark red; so those need no further action. You can look at the UTM_CAMPAIGN and UTM_MEDIUM data grids to do the same exercise.

Supporting data to help you can understand whether bots cost you money

If you have a website that is public on the Internet, anyone or anything can visit it. Not only will humans visit your site, bots will too. Sometimes you need good bots to visit the site, like search crawlers which index your content so it can be found on search engines. Sometimes you notice something like the following: a massive surge in dark red. Bots hitting your page even though you don't want them to. The question is whether these bots are harmful or not and whether you need to take action or not.

I have a whole article written about troubleshooting this example: https://www.dhirubhai.net/pulse/fraud-dr-augustine-fou-yy3sc

For brevity here, I will just show you the supporting data which shows why this was labeled a bot and also why the site owner understood they didn't need to take action. Note in the GEO_AS_0 data grid, you see the name of the data centers/ISPs from which the bots originated -- Amazon, Microsoft, Cloudflare. All of these data centers were in the United States. The bot is an automated browser that repeatedly loaded pages, and you can see the window size (320x500) and screen resolution (video=320x550x24) were repeated thousands of times. And the IP addresses that start with 52.9.x.x were also heavily repeated. These are characteristics of bots (automated browsers) and not of human visitors. This way, you can easily understand why FouAnalytics labeled these as bots. The fact that all of the bots loaded pages directly -- i.e. urls that had no UTM_SOURCE in them -- tells you these were bots that did NOT come from your paid media campaigns. So you were not losing money to fake bot clicks and traffic. The supporting data below helped the site owner to understand that they didn't need to take further action because the bot was not costing them money (perhaps a little bit of server costs and bandwidth).

I'll leave you with a fun example to ponder. Which of the following is a bot attack? Chart A or Chart B? (think of an answer first, the answer is below)

Chart A shows what appears to be a surge in orange (declared bots). But notice the green volume bars right below that show almost no volume (few to no pageviews). So just a few declared bots will appear to be large swaths of color in the stacked percentage (middle) section of the time series chart. However, Chart B shows a massive surge in red (bad bots) and the green bars right below that show surges in volume SO large that they completely drown out the tiny green bars in the days before the bot attack (on the FouAnalytics site).

Before the end of Q1 2024, I will be releasing a free, entirely self-service version of FouAnalytics Lite for site owners. Happy bot hunting y'all.

Subscribe to FouAnalytics Practitioners' Newsletter for regular updates like the above

Further reading: The FouAnalytics Origin Story