A quarter century of predictive analytics

On a hot sweltering day in Lincoln, Nebraska in July of 1990 I delivered my first predictive model.

Working on a contract basis for Metromail (acquired by Experian), I analyzed the performance of about half dozen recent mailings for a non-profit to predict the characteristics of best donors.

I used demographics like age, income, home value, history of donations and created a linear regression model. The top decile (we group data into equal ten percent, each is called a decile) was yielding a theoretical 7% response rate, and the bottom about 0.5%. Everything looked good, and I presented the results to the director of the non-profit team, with “a little help” from my manager who smoothed out my explanations. I then created a “score card” with the coefficients and passed it to the production team to score 115 million households to prepare for the next mailing.

It feels like yesterday. Yet somehow, to paraphrase Sir Richard, seems like I’ve gone “from youngest modeler to the (almost) oldest”.

I consider myself very fortunate that since then I have been developing predictive analytic models for sales and marketing throughout my career. Especially considering there were not many of us back then. As a budding economist, modeling was something very much I wanted to do since high school, learned as much as I could in college, and had the chance to implement from startups to Fortune 500.

Over the years I have read many technical articles, papers, and so on, but one that crystallized for me in 1991 was a visionary – and to me, influential – article by Robert Blattberg and John Deighton called “Exploiting the Age of Addressability”. A Must Read! I knew reading that paper, that we were at the dawn of a new era. How far we’ve come!

It has been very gratifying to have chosen the practice of what we now call data-driven marketing analytics, come from being a back office operation to a board level critical initiative of every enterprise. And I ask myself, what were the biggest changes from back then? After all, the underlying statistical and mathematical principles haven’t changed in a century.

Here are the important trends that I believe have shaped this discipline in the past two decades and a half, and will continue to do so in the future:

1. Data is becoming messier

Back then, data came one way: structured. It was created by professionals and maintained in systems by specialists. Relatively speaking, there was not a lot of data anyway. I recall those data center trips being very cool (literally… had to wear a pullover before entering as temperatures were much lower).

I don’t have to tell you that data being created now is coming from so many directions – thus Big Data. It’s the result of having anyone (and any number of machines) create data that is stored digitally – compared to tapes as there really were no hard drives or servers back then. And with today’s platforms, languages and devices encouraging this data capture, and with more things being connected (see point below), with help from trained humans still, this will continue as intelligent systems become adept at reading and acting on all this complex data automatically.

2. Latency to results is getting close to zero

When I started my career, it took a lot of time to build and deploy a model. From data extraction to creating a modeling dataset to creating scoring program, we had to move from a production database to an analytics sandbox, back to another platform where data was stored. And then the mailings had to be sent out. It took six months for a model to come to life!

As companies realize the value of acting on this information, and as interactions become even more digital oriented, all these processes are becoming instantaneous. The most innovative companies cannot – and do not – wait to store, extract, analyze, interpret and then execute on this data and insights. Retargeting while less than perfect is a poster child of acting instantly on information, but this will roll out to many aspects of digital exposure and beyond.

3. Everything is interconnected

Clearly it was the opposite in the beginning days of digitizing information. Very few if any systems talked to each other without a great deal of effort, much less exchange information across several systems. While the analytics data was on tapes, order data was collected in phone systems which were not digital, and there were still people walking the floors carrying information on paper. Yes kids, this was true back in the day!

Once we saw the benefits of systems talking to one another, protocols were created, then shared databases, then APIs, and so on enabled information exchange with little effort. Today with the Internet of Things (IoT ) movement, we are finally here. Cars are sharing information with the manufacturer, toothbrushes are sharing information with your dentist, and your watch… well, wearables are here. This is going to continue in ways not yet imagined today (although my favorite is from this great visionary’s mind here on LinkedIn).

4. Privacy lines are blurring

In order to perform analytics that required combining data across divisions or domains, we had to get “permission” and wait. From database/list owners, from customers (properly notified and able to opt out), we couldn’t just put datasets together – especially if it involved purchase transactions. Of course compiling a list afforded the benefit of multi-sourcing, but still it was not a blanket, automatic, instantaneous key to merge data limited by your imagination.

Once for a large auto manufacturer, I remember having to go to great lengths to get approval to merge the customer database (with customer ID as primary key) with the vehicle database (with VIN) from the financing division, so we could model and predict the relationship between purchase timing, vehicle type and financing type for direct marketing campaigns for the customer loyalty program. Then, relatively speaking, merging Teradata and Oracle databases was a piece of cake.

Today, it is increasingly difficult to tell where ownership of data stops and where it begins, as customers and businesses alike traverse the digital spectrum of their interactions across domains, devices and platforms. Nearly every online community, streaming service, mobile app or most sites come built in with Terms of Service that you click to accept (or it’s there in one corner waiting if you care, or buried in the settings somewhere), or you are not using that community, app or service, sorry. While that is great for the customer experience, and trust is very important, more or less this access is a given.

For example, recently I was installing what I thought was a rather standalone app on my smartphone (think something like weather or trip planning), and during the installation, it was asking for access to my contacts, phone calls, calendar – made me wonder, “really?”. This trend will continue. You will be a participant, willing or not, unless maybe you move to the Himalayas, become a sadhu and get completely unplugged. Interesting scenario, but not likely.

5. It’s all about the experience

This one is my favorite. Going back two plus decades, the technical proficiency – academic or professional – was the ticket to the modeling team. Although we certainly heeded to the business conscience (more than say IT roles), we were quantitative first and last.

In short, it was about the analyst experience. Needed to get the data right, models right, scoring right and so on. Battles had to be fought (with and) getting the right data to support the analytics. The best outcome was a model that did the best in test vs. control, and the mailing performance was scrutinized, and models tweaked, to get the best possible outcome.

Many of the above tasks and workflow still exist, but now the value of every predictive model starts and ends with judging its impact on the user experience, and rightly so. Did the models make a difference to a salesperson’s paycheck? Did it improve the customer experience in an interaction? Did it move the customer along on the marketing journey? Was their online experience optimized, thus resulting eventually in greater sales and higher customer value?

So where will this predictive analytics with humble beginnings go in the next twenty five years? Everywhere. It will be embedded in every decision made by humans or machines. What will happen to the huge teams of statisticians and quantitative analysts? I can only speculate, but there could be fewer, more elite, very sophisticated group that is monitoring the embedded decisioning made by machines and software. Companies that don’t make this transition will perish, replaced by those that adapt and thrive in this new paradigm.

Predictive Analytics will be so pervasive and ubiquitous, and so deeply enmeshed in the fabric of our lives that we will look back from 2040 and say, “you mean, there was a time when devices and software didn’t act like this?”

Parth Srinivasa is an analytics expert in B2B Sales and Marketing and president of Valgen. When not modeling, he is thinking of modeling, or wondering if something he just came across could be predictable at the 95% level of confidence.

_______________

Valgen delivers data quality and analytics services for sales and marketing in commercial fleet, high tech, finance and insurance. We serve enterprise and mid-market B2B customers worldwide. Valgen is a Salesforce consulting partner and Salesforce ISV partner. Visit us at valgen.com.