When There Is Too Much Power

Clearly when you see this red thing all over your screen in a maximized browser, the last thing you wish is to press advanced so that you can go further to an impeding cyber-doom, so from where you stand you would probably be very grateful to the search engine provider for being as thoughtful as to saving your skin from these malicious web sites. But there is another side to this story, a dark one that has to do with monopolistic behavior and the arrogance of big corporations towards small businesses, which cannot be left uncommented.

Seeing this screen fills you with a sense of elation, a bullet dogged, and this is exactly what the search engine provider behind it seeks to achieve, it has nothing to do with your security but rather only with the image of their brand that forms in your brain: it's a marketing maneuver, like so many others. Why I say this? Because sites don't infect computers. The malware may be stored on the site, but the actual vector that downloads and facilitates its installation on a computer is the browser. And so happens that the search engine provider that you would so worship after such experience is also the maker and owner of one of the, if not THE, most popular web browser and second in the top of most vulnerable applications in 2014, second only to Internet Explorer, Chrome.

Yet Chrome continues to have vulnerabilities and the responsibility for keeping the web clean of malware is being delegated to web site owners. Why? Because they can do that. They are big enough to make site owners bend over and backwards and jump whenever they make a change in the search engine algorithm because let's face it: nobody can afford not to be on Google. So when you see this page instead of your company's web site, the feeling is rather different: every second that that page stays there means clients lost, slipping back in the search engine, a dent in your reputation and hole in your pocket. This is what happened to a friend of mine but with an added bonus which would be hilarious if it weren't so sad and frustrating.

By comparison, Firefox is making efforts to secure the client during the browsing process. They, for instance, just introduced a functionality that alert users when a web page on an insecure connection is requesting a password. But a more secure browser is nowhere as glamorous as the red page, it can even have the opposite impact: annoy the user with pesky security warnings or plugins that are blocked for reasons of insecurity and so on. Doing the right thing sometimes comes with a cost, a cost that a giant corporation like Google does not need to incur, cause why bother making the browser secure and annoy the user in the process when you can create the impression of security, keep the user vulnerable but happy and gain invaluable market advantage by doing so. From a corporate perspective it's a no-brainer: money comes first, all the rest is just collateral. But the ugliness of this story does not end here.

One fine September morning 2015, I get a call from this friend of mine who operates a bear-metal server with several websites that something happened with some of his sites (multiple), that whenever he visits them he sees a scary red page. So I browsed there as well and sure enough, scary red page, and no additional info. I have to admit, this was the first time I was faced with this particular kind of situation so I went on and Googled the solution. Found it in the "Search Console" which explained to me that the site in case is spreading malware and that I need to fix it, submit a report about how the problem was fixed and then, they would consider taking the quarantine off the sites. So I think... it'll take a day for them to analyze it, this is a disaster... So what can I do to fix this as fast as I can, cause this friend of mine makes a living on these sites. He had good backup so the easiest answer was to wipe the server and reinstall from scratch, it would take like four hours, not very fast, but a reliable estimate nonetheless.

Now the only problem was, how to pick the backup, cause what if the malware was backed up along with the code and database? There was no way to know when the penetration happened. I grabbed the code for analyses, wiped the server, reinstalled a combination of 4 months old code with fresh databases. Meanwhile, the comparison of code with the development code was not yielding any results. Everything was in perfect order there were no changes to the code or directory structure, no new files, not a single line was hinting towards altered pages or anything of the like. The puzzle was deepening. The affected sites were also different in technology: some were Java based some PHP based, so what could have happened? How was the server penetrated so that it affected both these different technologies? The natural answer was that this was a system level penetration.

With the re-installation finally finished I was preparing to submit the report when bad news came. My friend looked at the page, bypassed the security warning and when the page loaded something was trying to download itself onto his computer. I did the same and after several attempts, I was able to reproduce the behavior. I just could not believe my eyes. Just worked half a day on fixing the problem and the problem was still there. But the frustrating outcome proved to be the solution to the problem. File comparison eliminated code altering malware, database injections seemed unlikely because the applications were running on different databases, even different database engines. There was only one common denominator to all the sites that were banned from the search engine: they all served advertisement through the same private ad-server. So I realized that the sites themselves were not infected, they were only serving ads, which were providing malware together with the ads. But the malware download uncovered yet another interesting evidence: the place of origin of the file and it was not on my friend's server, it was on doubleclick.net (see image below), a domain owned by Google. The intermittent behavior of the malware download was also explainable with this new insight. The sites were rotating ads, and when no ads were suitable to be displayed my friend configured Google Ads to be displayed via Ad Sense, channeled through his ad-server, instead of leaving the spots empty.

So it turned out that his server was in fact not hacked at all, not system level, not application level, not even at ad-server level. The sites were banned by Google, from Google search results for serving malware through Google Ads from Google servers. I reported the situation and within two hours the sites were cleared of the quarantine and the malware downloads disappeared. I did not however receive a confirmation for my analyses, nor did my friend receive any compensation for his inconvenience. He was left with the cost of lost business and a sower taste in his mouth with regards to how small he is and how big companies can thrash with small ones without any consequence.

Of course a compensation would have been nice, even a "thank you for discovering our problem for us", would have been nice. Many small companies would pay for such discoveries, after all, it is a valuable service. But not Google. They don't need to pay, because in a monopoly position progress and better services don't matter. All the concern needs to be focused on maintaining image and the status quo. To them, a thank you note would have been an admission of guilt and that's bad business for the image that was so effortlessly built with hoax security measures.