The 'Recommendation Engine' Future

In this wide-ranging interview with MIT IDE Content Director, Paula Klein, Schrage discussed the theme of the book as well as his passion for the wider historical context of advice: “From divination, to astrology; the Oracle of Delphi, to Machiavelli, to Samuel Smiles and Dale Carnegie! I want to convey that recommendation engines are the next logical and inspirational step in the history of how humans try to create options for their future.”

MIT IDE: Does the upcoming book represent a new research direction for you? 

Michael D. Schrage: It actually represents an unexpected convergence. I believe recommendation engines are the biggest and boldest way digital experiments, network effects, and personal identities will all converge. Recommendation engines have already radically transformed our relationship with choice: Just consider Tik Tok, Amazon, Netflix, YouTube, Spotify, or Instagram. Few things are more important than our ability to make good and informed choices, and too few people understand or appreciate how pervasive and persuasive recommendation engines have become.

How could I not be interested in a ubiquitous technology that literally changes what we want?

IDE: What is the essential theme?

MDS: Recommendation Engines is the book’s title, but this book is really about the past, present, and future of advice. If we really are the choices we make, the advice we take — and the advice we ignore — becomes astonishingly important in defining who we are and what we become.

Increasingly, the things we buy, the people we meet, the places we (used to) go, our jobs, entertainment, investments, books, music, and work are shaped by recommendation engines: data-driven advice from ever-smarter machines that learn from you and about you. In nearly every dimension of life — relationships, health care, pop culture — machines demonstrably can offer better advice than our best friends. I’ll go even further and say that machines can give you advice and counsel to make your best friends even better. I find this new capability fascinating.

IDE: What are the key takeaways for readers?

MDS: The book offers a swift and sure way to get up-to-speed on a technique, platform, and technology that is going to matter more each day for consumers, workers, and coders. It’s important to understand the underlying dynamics that influence and inform our world. The more we learn about the tools and technologies that shape our choices, the better off we will be. That said, this isn’t just a techie textbook; it’s a book that literally spans the human history of advice — from divination, to astrology; the Oracle of Delphi, to Machiavelli, to Samuel Smiles and Dale Carnegie! I want to convey that recommendation engines are the next logical and inspirational step in the history of how humans try to create options for their future.

I hope readers will be surprised by the simplicity of incredibly complex and complicated algorithms and the enormous volumes of data processed to personalize people’s choices.

Most people don’t understand or appreciate how ensembles of relatively simple algorithms can generate remarkably relevant and appealing recommendations in almost any domain.

IDE: What is your view of the ubiquity of recommendation engines and hyper-personalization? Is there a down side?

MDS: There are several questions we can ask in this regard: For instance, are ubiquity and pervasiveness bugs or features? Do individuals want multiple choices or one choice?

Will people feel creeped-out by data-driven digital intimacy that feels invasive, or will they be thrilled by a technology that makes them feel good about what’s available for them?

It became clear to me that organizations that sought to use recommendation engines primarily as sales tools had completely different UXs than those wanting to build trusting and trusted lifetime value customer relationships.

IDE: Do you anticipate a consumer saturation point when we’ll wake up [hopefully, post-pandemic] and find we really don’t need 20,000 Prime movies and videos, or twenty million songs on Spotify? Could there be a backlash?

MDS: Let’s not confuse quality and quantity of choice or search engines with recommendation engines. I may want to find the right musical choice not just for me, but for my family or my Friday Office Zoom call. Recommendation engines are the microscopes, telescopes, oscilloscopes, and MRIs of choice. They give us ways to slice, dice, and evaluate the features and elements that matter most to us in particular contexts. It’s true that people may be afraid of technology that knows more about what they like and want than they do, but that’s the future!

IDE: What was your biggest surprise writing this book?

MDS: The history of efforts to merge, marry, and fuse data into practical and actionable advice, is nothing short of awesome. Frankly, I was blown away by the very human yearning throughout history and across cultures to get great advice…whether from the gods, the odds, the stars, or iPhones.

I also didn’t think that machine learning would so swiftly and overwhelmingly transform recommendation engines worldwide.

Something I thought would likely take a decade, happened in barely two or three years. I was surprised by how quickly recommenders got super-smart about their users.

IDE: What’s the most important change you see on the horizon for recommendation engines in the next two years?

MDS: My answer today is not the answer I would have given when this book began. The most important worldwide change I expect is regulation. By the time governments from Asia, Europe, and the Americas are through with them, recommendation engines will be more regulated than internal combustion engines.