Experimentation in Data Science

Bargava Subramanian and I are joining our friends at Scribble Data and hasgeek.com for a conversation about the central role of experimentation in data science (and business!). Join us this Wednesday (15 July 2020) evening (Asia) / morning (US) and contribute to the conversation!

Experimentation in Data Science - Making Data Science Work: Session 5

We had a warm-up/prep call last week and I'm really excited about some of the questions, perspectives, and lines of thinking that came up. Experimentation is severely underweighted in data science and so many product and data science teams are missing out by not making it a core part of their work. (Connect with us at Aampe if you want help bringing systematic and successful experimentation to your customer communication!)

Since it's such a vast topic and deserves so much attention and study (beyond our short session on Wednesday), I thought I'd share a few resources in advance of the discussion:

Technical - If you're interested in causal and statistical inference, measurement in complex scenarios (let's be honest, what scenarios aren't complex?), code, engineering, data pipelines and APIs, and other technical aspects of experimentation, check out some of these books and papers:

Inferno: A Guide to Field Experiments in Online Display Advertising - this is a recent, excellent paper reviewing many of the key challenges of running field experiments in online advertising, within the fun scheme of the seven circles of hell in Dante's Inferno.

Designing and Deploying Online Field Experiments - the original paper behind open-source PlanOut library (from data scientists at Facebook). The paper offers excellent discussion of why it's so important to keep experimentation design separate from application code.

Large scale experimentation - this long blog post from the team at StitchFix provokes thought on efficient navigation of the explore-exploit tradeoff and features a really snazzy and clarifying interactive cartoon for showing the difference between some of the sampling strategies discussed in the post.

p-Hacking and False Discovery in A/B Testing - a lot of poor experiment and statistical design hide behind common statistical methods - don't get misled!

Experimental and Quasi-Experimental Designs for Generalized Causal Inference - a classic textbook and one of my favorite books on experimental design and experimentation in general.

Stories and Concepts - If your job has you more on the business side of things, working with stakeholders and customers, or in product management, helping evaluate and prioritize trade-offs in developer time and attention, and you want to read some powerful examples of experimentation as a practice in business and product development, check out some of these posts and articles:

Why These Tech Companies Keep Running Thousands Of Failed Experiments - this article is chock-full of fascinating examples and links to further reading begging to be be explored.

Building a Culture of Experimentation - a recent piece from HBR about the organizational side of making experimentation possible and effective at your company.

Test Everything: Notes on the A/B Revolution - an older piece from 2012 that makes some good points - though it makes me wince to see A/B testing as a placeholder for experimentation, because it really isn't.

The Science Behind Those Obama Campaign E-Mails - it seems like political campaign teams have done a better job of realizing the value of experimentation than many marketers, sadly this piece is behind a paywall. For a more recent example: Here's How Facebook Actually Won Trump the Presidency - "Coby's team took full advantage of the ability to perform massive tests with its ads. On any given day, Coby says, the campaign was running 40,000 to 50,000 variants of its ads, testing how they performed in different formats, with subtitles and without, and static versus video, among other small differences. On the day of the third presidential debate in October, the team ran 175,000 variations. Coby calls this approach "A/B testing on steroids."

Why Big Testing Will Be Bigger Than Big Data - I never liked the term Big Data. The idea that Big Testing should be way bigger than Big Data is superb.

Column: Why Businesses Don’t Experiment - Don't let your company fall prey to the mistakes Dan Ariely describes in this short HBR post.

Alright, I hope these are provocative and useful. Hope to see or hear from you during our session!