The Seasonality Of Paid Search Effectiveness From A Long Running Field Test

Paid search, also known as Search Engine Marketing (SEM), allows advertisers to target users of a search engine with relevant ads. It is broadly adopted by advertisers due to its superior capability to drive users, traffic, and conversion compared to other marketing channels. It also provides direct consumer behavior metrics such as ad impressions, clicks, and website visits for advertisers. However, the true effectiveness of paid search has been hard to measure, as the sales led directly by paid search ads might lack causal effects. What are the incremental sales or user acquisitions truly driven by paid search campaigns? The answer typically involves detecting small signals out of large noisy consumer behavioral data in a controlled experiment.

At eBay, we have pioneered using A/B tests to measure advertising effectiveness. The Harvard Business Review reported our 2013 field test where no material sales difference observed with SEM turned off in half of U.S. In this latest study, we further enhanced our experiment with a hybrid Geo+User design. The result showed significantly improved test power with only 10% U.S. turned off (lower opportunity costs). With this enhancement, we have conducted the first ever long-running field test over one and half year to measure the incremental impact by Google paid search campaigns on one of the largest e-commerce platforms: the U.S. eBay marketplace. Among our findings:

• Paid search drives statistically significant sales lift to the U. S. eBay marketplace. The amount of incremental sales is a portion of the sales from the "last click attribution" via paid search. Advertisers are encouraged to actively conduct their own A/B tests to measure the true paid search incrementality and avoid overspending based on the often-inflated attribution value.
• Paid search is an important source for acquiring new users. The incremental new user acquisition lift is even higher than the immediate sales lift from paid search campaigns.
• The long-running test reveals a strong seasonality trend in paid search effectiveness. Paid search campaigns made the biggest incremental lift% on sales during holiday seasons. This is causation, not just correlation, as proved from this controlled filed experiment with repeated and consistent test results in two years' holidays.
• It appears that ad spends need to reach a certain threshold to enable the overall effectiveness of paid search. However, further increasing spends beyond that threshold increases cannibalization of other marketing channels or organic sources. Advertisers are encouraged to run experiments regularly to guide optimal marketing spends. 
• Natural search performance experienced nearly double-digit gains when paid search was turned off completely. Nevertheless, natural search cannot fully substitute for paid search traffic. Natural search and paid search shall work in a complementary way to enhance brand presence and promote customer conversion.

Published at the 2016 ACM Conference on Economics and Computation (EC'16), the full paper can be accessed at: https://dl.acm.org/authorize?N12776.