Would You Bet Your Business on Google?

Panda is a set of updates to Google's search algorithm that was released in 2011. The name honours one of the inventors, Navneet Panda. Although updates to these algorithms through evolving Search Engine Optimization (SEO) techniques remove inorganic, illegitimate and unsolicited content to improve user experience, one undisclosed update has been associated with a stunning impact on a legitimate business and its internet market.

Before there was Reddit and Quora, there was MetaFilter. As Neal Ungerleider (of Fast Company) and David Auerbach (of Slate) remark, this online community, which combines a question-and-answer site with a forum to discuss and exchange links, debuted in 1999 and developed into one of the Internet's most popular, intelligent and civil discussion communities having a core group of sharp, technically savvy users, while a tiny group of moderators walked a careful line between policing abuse and allowing free expression.

Despite maturing to a website that makes the Web better, MetaFilter never gathered a sizable user base outside North America, Britain, Australia and quite recently, India, perhaps because the rest of the English-speaking world at the time of MetaFilter's youth did not embrace technology. Although this factor restricted the number of users and consequently, that of moderators, it allowed MetaFilter to maintain a remarkably high quality of content that is ostensibly superior to that of Reddit and Quora.

MetaFilter earned the bulk of its revenue through Google's AdSense program, working with Google to advertise on MetaFilter's web pages and, in exchange, to receive a sizable proportion of visitors via search results, much like YouTube's model. In contrast to this revenue stream, members were offered an option to pay a small one-time fee for hiding advertisements on their session.

On the 17th of November, 2012, traffic to MetaFilter decreased overnight by a staggering 40% of usual. According to its founder, Matt Haughey, momentary reductions were not uncommon in the past and these usually occurred when Google periodically tweaked their search result ranking algorithm. A connection between instances of abnormal traffic into MetaFilter and launches of software updates at Google seemed apparent. Naturally, this event was alleged to correspond with a software update as well. MetaFilter had gotten caught in the crossfire between Google Search and spam.

This time, MetaFilter lost its visitors in perpetuity (see figure below, illustrating daily traffic into MetaFilter's domain. Source: Matt Haughey). Traffic had reduced to the audience who frequented the site. In his analysis, Auerbach explores the Clickthrough Rate curve, an illustration of the frequency of clicks on search results against the result's rank. He shares an estimate from the advertisement network Chitika that search engine users do not access results after the third more than 10% of the time. With fewer visitors come fewer clicks on MetaFilter's advertisements, which would hit where it hurt the most. MetaFilter instantly became an old monk bedridden with a chronic illness.

Haughey disclosed his troubles to the public the following day in a detailed blog post. Over the hours, he attempted in vain to resurrect traffic inflow by reducing advertisements, heeding suggestions that cited Google's penalty on ad-heavy websites. MetaFilter's AdSense representative had no remarks other than offers for more advertisements. Within weeks, revenue through AdSense had nearly halved and MetaFilter practically disengaged from Google.

Haughey announced that MetaFilter was laying off three of their staff members amid a massive decline in revenue. Out of five full-time employees and three contractors, one full-time employee and two contractors were now out of positions as moderators for the busy online community.

In addition to the loss of revenue, MetaFilter encountered another problem from Google that threatened to obliterate its content. Haughey began receiving forwarded e-mails (see figure below, showing Haughey's mailbox filled with link removal requests. Source: Matt Haughey) through members from Google Search bots that directed them to launder content served on the community to qualify for renewed quality standards. This flood of e-mails meant that the search engine appeared to annihilate the community.

Google never publicly commented on these allegations soon after the incident, but Matt Cutts, a senior member of the webspam team, declared much later that issues causing such incidents were being addressed. In May 2014, Cutts tweeted a status referring to comprehensive discussions with Haughey. Over twenty months following the original event, in July 2014, Cutts confirmed that a previously undisclosed software update, like Panda, had hit MetaFilter.

When it comes to ranking search results, Google appears to be a 'black box.' The only descriptive source of information is likely the patents filed on the algorithms, which are obviously too technical for operators to interpret. When updates to this algorithm are launched, repercussions rattle the search result board, juggling links to websites like MetaFilter. For the operators in such a situation, Google's involvement can be alleged, but not warranted. Consequently, they may have nowhere to turn to for assistance, mandating the need to shield a website from 'update volatility.' Haughey's reaction seems entirely appropriate since there was nothing wrong with MetaFilter per se while traffic dropped.

It is impossible to profitably manage a website that is completely disconnected from search engines. In the case of eHow, which raised US$ 36 million in venture capital in 1998 but failed miserably by 2004 before being sold for a paltry US$ 100,000, the new owners unlocked a revenue of US$ 4 million simply by unblocking search engine access and performing basic optimization. However, eHow subsequently encountered similar troubles from Panda as MetaFilter did during Google's campaign against content farms. The profit-motivated media industry needs to be extremely cautious under the whims of one major search engine that promotes fair practice.

In those twenty months and beyond, MetaFilter's revenues regressed to those levels seen five years before the incident. Haughey concluded earlier that the only way to save MetaFilter from collapse was through a donation campaign, which was inherently restricted in scope by the number of dedicated users. It became established that excessive reliance on AdSense for generating revenue turned detrimental and that the business had been steered sans extensive market research, expansion of the audience or optimized website design for mobile access.

The problems of MetaFilter are likely to echo in some form in other companies in the industry. MetaFilter's tale highlights that advertisements (from a single agency) are not robust to drive revenues, besides recommending that websites aim for diversified traffic sources and provide alternate user interfaces. Reddit derives additional revenue through subscriptions; Quora's web interface receives visitors from Google, Facebook, Twitter, Microsoft and LinkedIn, while its mobile interface serves more than 40% of direct traffic.

Cutts is quoted as telling operators to have "eggs in lots of different baskets." It is considered wise business practice for operators to liberate their websites from dependence on search engine traffic.