Marketers, It's Time to Wean Yourself From Bad Programmatic

When I first started in online advertising more than 25 years ago, things were simple. There were only a handful of sites that had large enough human audiences to be meaningful for advertisers to buy ads from. For example, portals like Yahoo aggregated all sorts of content, from news to weather, sports scores to stock prices, and thereby aggregated a large human audience. Then as publishing tools like Wordpress and Blogger made it easier for more people to put content online, the number of small websites with niche content blossomed. Some of these sites wanted to make money by tapping into advertising dollars too. But often they were too small by themselves to sell ads to large advertisers. By the 2010s, there were enough of these so called "long tail" sites and enough dollars shifting into digital advertising that ad exchanges were created to facilitate the buying and selling of ads on thousands of sites at once. While the innovation and business opportunities for newly formed advertising technology ("adtech") companies were enormous, it was also the time when digital advertising took a turn for the worse.

The Ad Fraud Crisis Started 10 Years Ago

By dissociating the advertisers (buyers of ads) from the publishers (sellers of ads), the rise of ad exchanges also led to the dramatic rise in ad fraud. Previously advertisers bought from publishers directly -- publishers' sales reps that they could meet and negotiate with across a conference room table. Once advertisers started buying from exchanges, however, they started losing sight of the thousands of sites their ads went to. The early exchanges were eager to grow as fast as possible to please their investors and deliver on the "hockey stick" growth charts they showed in powerpoint decks. So they happily let everyone in, including bad guys. Bad guys didn't just create a few fake websites; they were prolific. They automated the process and used Wordpress templates to create hundreds of thousands of fake websites and added them to as many ad exchanges as possible. This drove the enormous growth of the first wave of ad tech companies. Both they and the fraudsters were feasting on the ad dollars flowing their way.

How much were they feasting? The charts above from an AdExchanger article from 2016 showed that AppNexus cleaned up 92% of their impression volume by kicking out large numbers of sites that were so obviously fraud, AppNexus would have gone out of business if they didn't clean it up. Think about it, who reduces their own sellable inventory by 92% voluntarily, unless the alternative was even worse? By 2015, we were already at >90% ad fraud. You may argue that since then fraud detection has gotten better. (My data shows otherwise). But fraudsters' tech has also gotten better -- more prolific and better at evading detection. So ad fraud has gone up since then, and the number of dollars flowing to fraudsters and criminals is larger today than it was in 2015.

The Long Tail is a Tall Tale That Adtech Companies Sold

Over the last 10 years, since the rise of ad exchanges in 2012, advertisers bought into the myth of the "long tail." The theory was that even though each of these sites were small, if you put together enough of them, advertisers could still get "scale" (enormous quantities of ad impressions to buy). This belief provided the perfect cover for ad fraud. Real long tail sites had specialized niche content that a few humans really liked and engaged with. Fake sites, however, were never made for real human audiences. They were "made for advertising" -- i.e. their sole purpose was to generate as much ad revenue as possible. The content might be entirely plagiarized or non-existent. And the traffic to these sites was entirely bot traffic (no humans even knew they existed). This became a very easy formula for making lots of money from unsuspecting advertisers pouring money into digital, chasing more "reach" and lower prices. Media agencies loved it too and encouraged their clients to increase spend in programmatic channels as much as possible (because the agency made margin on selling it, sometimes undisclosed margins).

After a decade of buying into the myth of the long tail, let's look at some actual stats. In the table above, data from deepsee.io shows the percentage of sites that carry ads, grouped by the size of the site. In the top bucket of the 1,000 largest sites, 330 of them carry ads. Some sites like wikipedia.org are in this group of large sites, but they don't have programmatic ad tech code on the site to run ads. As you move down in the rank buckets, you see that by the 8th bucket, which contains smaller sites outside the top 1 million, the percentage of these sites that carry ads drops to 11% -- about 1 in 10. That means that 9 out of 10 of these "long tail" sites don't carry ads. So yes, there are legitimate long tail sites with real content that humans love. But those are small in scale and 9 in 10 of them don't run ads. Your ads are not going into these high quality, long tail sites.

A second data set (table to the right) shows how dramatically the number of bids fall off when going from the top 10,000 domains to the next, and the next. By the 7th group of 10,000 sites, the number of bid requests is less than 1/100th of the top bucket. Directionally, this shows that the vast majority of humans visit a small number of sites regularly. Previously I had just assumed that if we took the Alexa top 1 million sites, we could account for the majority of humans' visits to websites. The data here suggests that we only need 1/10th of that -- advertising on the top 100,000 domains is sufficient to get your ads in front of the majority of humans. Still don't believe me? Try this. As quickly as you can, name ten websites you use every day; and also name the top 10 mobile apps you use every day. When I ask this question, most people can't even name ten sites or ten apps that they use regularly every day. That should tell you that humans visit a very small number of large sites regularly. The two entirely different data sets above, and common sense, shows you that the long tail is NOT what you've been told, or what you've been sold, for the last decade. Large numbers of your ads did not go to high quality long tail sites, even though there are high quality long tail sites. The REAL high quality long tail sites don't have enough "scale" and 9 in 10 don't even carry any ads.

Made for Ad Fraud Sites Masquerading as Long Tail Sites

So where did your ads and ad dollars go? Your ads went to MFA ("made for ad fraud") sites, masquerading as long tail sites. As you read above, not only are these sites easy to create, by the thousands, the traffic to these sites is also easy to obtain. Fraudsters just buy the traffic, and bots deliver the traffic. You can't force a bunch of humans to go to your site. But if you "rent time" on a botnet, the bots will faithfully deliver as many pageviews as the fraudster paid for. Remember "bought traffic = bot traffic." Since the fraudsters that created these sites are cheating anyway with bot traffic, why not pile on other shady methods to multiply the ad revenue. Why not stick 20 ads on top of each other in the same ad slot ("ad stacking"). Why not do naked ad calls -- load the ad without the webpage to save time and bandwidth. Why not load 20 ads on the page, and refresh the ad slot every 2 seconds to maximize the number of ad impressions created out of thin air? You may have seen sites like the one below, with many ads on the page, that reload every time you click to the next slide. Fraud detection tech does not detect any of these forms of fraud, because they are not even looking for it. The tech was built to detect IVT (invalid traffic) or bots. So when they report 0.6% IVT, you should interpret that as they failed to detect any fraud in the other 99%; not that there was no fraud in the other 99%. These companies were part of the problem; they enabled fraud to flourish because they misled advertisers into thinking there was low fraud.

So What?

Marketers, it's 2022. After a decade of pouring money into digital, and specifically into programmatic ads, it's time to wean yourself from "bad programmatic." Around 2012 (ten years ago) the blue line in the chart above started to diverge from the green and yellow lines. In other words, the amount of digital ad spend (blue line) started diverging more and more from the amount of humans' usage of the internet, social, and mobile. If human activity cannot explain the large numbers of ad impressions, what can? Right, non-human activity, also known as bot activity. The bot activity is not on the mainstream sites that humans visit; bots only go to the sites that pay them for traffic. They go to the MFA sites that buy traffic to increase their own ad revenue, and the ad exchanges facilitate the flow of ad dollars to them.

Hopefully the above has convinced you that the long tail is a tall tale that ad tech companies sold you to separate you from your money as quickly as possible. ;-) You may also realize that there are simply too many fraudulent or "MFA" (made for ad fraud) sites to add to block lists. Some exchanges limit block lists to 10,000 rows; that's not nearly enough to block all the bad sites and mobile apps out there. So it's time to move to a whitelist approach, also known as allow-lists or inclusion lists.

After a decade of measuring sites and ad impressions for ad fraud, with my tech platform FouAnalytics, I have the historical data and methodology to help clients set up whitelists of domains and apps that humans use. Not only are we avoiding high fraud and bot sites and apps, we are also optimizing ads towards high human sites and apps. Remember, the most important variable in your programmatic campaigns is showing your ads to humans in the first place. Without this, nothing else matters.

If you have any questions about the above or want to learn more about the FouAnalytics whitelists, please reach out.