How to learn from data: Empirics First

“Practical men who believe themselves to be quite exempt from any intellectual influence, are usually the slaves of some defunct economist."

An open mind is a key asset for a great decision maker, leader or scientist. But what topics have you changed your mind about in the past years? As discussed in my earlier post, I was originally against (serving my family) genetically modified food (GMO) but later found the evidence compelling enough to change my mind.

Examples abound of folks unwilling to change their assumptions, from once dominant companies who failed to adapt to new realities to Putin’s miscalculations in Ukraine. But what explains this human reluctance to update beliefs based on new incoming information? Psychology offers fun reads on integrating disconfirming information, such as Cook and Lewandowsky’s free?The Debunking Handbook. For managers and social scientists, I believe holding on to cherished theories is also influenced by the philosophy of ‘Theory First’ versus ‘Empirics First’ as described in a new Journal of Marketing paper. ?

Theory first?(TF) researchers have a specific theory they aim to support or falsify and design experiments or tests to do so. For instance, attribution theory implies that some consumers buying your product on a discount attribute their purchase to the discount and thus think less of your product (lowering brand equity and baseline sales). Loss aversion holds that losses loom larger than gains, and thus implies that the sales loss from returning to the regular price outweighs the sales gain of the discount. Strong theory is the dominant paradigm in social sciences, where reviewers often decry the ‘absence of a strong, unifying theory’, top journals specifically call for ‘conceptual work’; i.e. theory without a shred of evidence, and doctoral students are taught to build their model and write up their research starting from a specific theory they set out to test in data. While such strong theory-based science has plenty of benefits (see e.g. Bass 1995), it also suffers from lots of pitfalls, such as selection bias and confirmation bias. Basically, researchers have the tendency to select situations where their theory is likely to hold, and ignore evidence against the theory (e.g. consumer segments for which gains outweigh losses). Proponents of strong theory need to depend either on the highest ethical standards in each researcher or on the highest alertness and replication enthusiasm of the scientific community.

Empirics First?(EF) researchers have prior notions, sure, but they appear more open to opposing points of view. This is reflected in their research and papers, which often offer alternative hypotheses or no formal hypotheses at all (e.g. Ehrenberg 1995, Trusov et al. 2009). I especially like studies on the?conditions?under which e.g. price promotions decrease brand equity (DelVecchio et al. 2006) and give rise to loss aversion (Alkis 2014, Bronnenberg and Watthieu 1996). Based on such research, we have updated our beliefs: price promotions only lower brand equity if the brand is unfamiliar and the discount substantial, and price loss aversion is reversed when consumers care more about quality or are promotion-focused. In my own research, competitive reaction to grocery product price promotions have a minimal impact in the US but a large one in Turkey, and paid online media is more effective than owned media for familiar product brands, but not for unfamiliar brands and services (Pauwels et al. 2016). My UCLA advisor Prof. Mike Hanssens’ books on empirical generalizations (2009, 2015) are full of examples of strong evidence studies and the insights they generated. Indeed, Golder et al. (2022) find that EF is viewed as better able to generate real-world insights relevant to stakeholders, especially on new phenomena.

Ironically, such studies have a hard time getting published in top journals and getting impact in popular press and further research. Is the study set up with competing theories too complicated for reviewers? Or do empirics first researchers have no good alternative to the linear, strong-theory write-up taught in their doctoral seminars?

?Golder et al. (2022) offer a wonderful answer and target the Jekyll-and-Hyde contradiction in that many scientists prefer EF as authors, but demand TF as reviewers. Empirics first research (1) originates from a real-world marketing phenomenon, problem, or observation, (2) involves obtaining and analyzing data, and (3) produces valid marketing-relevant insights without necessarily developing or testing theory. They propose an EF paper should (1) Motivate the phenomenon, (2) describe the various analyses, and (3) end with insights gained. Relevance increases as stakeholder control over the scholar’s independent variables increases and as the stakeholder finds the scholar’s dependent variables more valuable. EF research is especially appropriate when stakeholder intuition leads to multiple plausible, yet conflicting, outcomes, when observations taken from the world or opinions expressed in business reports do not align with theoretical predictions, and when rich and newly emergent data allow the scholar to probe unexamined relationships. Theory rarely informs effect sizes, which are key to marketing stakeholders such as companies and government. For example, the sales-takeoff study documented an effect size of a 4.2% increase in the probability of a new product’s takeoff for a 1% decrease in price.

Machine learning (ML) may not always fit EF research, because it often lacks transparency and interpretability, especially at the causal level. ML cannot replace the EF scholar’s hunch or experience of where to look in the data, what relationships are meaningful to study, which additional data to consider, and whether the results are coherent. Moreover, abstraction is crucial to yield generalizable marketing knowledge. This means going from the empirical specifics to more generalizable ideas, concepts, or relationships. The scholar mindset should be to entertain the highest level of abstraction enabled by the empirical results.

As to reporting and evaluation, Golder et al. (2020) recommend EF research to be reported as EF research, not as TF research. Document your fishing expeditions and hunches as such to explain where you looked, which steps you took and which results emerged.

Ultimately, stakeholders and editors are best positioned to answer a basic question: does society, business and the journal benefit from communicating these findings? Golder et al. (2022) offer below table for evaluating Empirics First research:?