Get Ready, A New Engineering Paradigm is Coming!

It’s not about the sauce on a Big Mac.

Over the past twenty years, two paradigm shifts radically improved the way software gets made.

In 2001 the agile approach, favoring individuals and interactions over processes and systems, was borne. The rest is history.

Circa 2005 the notion of test driven development (TDD) went mainstream. The oddly simple “upside down” notion that you write the tests first, then you write the code that passes those tests. Shoot then aim. The result? Clarity and quality skyrocketed.

Both were adopted by a few damned smart people, and both quickly went viral.

There is a new, pre-viral, paradigm shift that is coming soon to a team near you…..

Data Driven Development.

“What? That’s been around forever.”

WRONG

data driven development puts broad data capture as a central design goal

Yes, we have always had data at the center of application design, but the focus was on data efficiency - single sources of truth, tables to ensure the data can be stored with virtually zero repetition, transactions stuffed into temporary logs to limit persistence, and so on. But massive compute power, incredible bandwidth and almost limitless storage, mean we need to change our data attitudes.

Put simply, data driven development puts broad data capture as a central design goal. The mindset being to continually look for, and capture, all data that might be germane. It’s the antithesis of data efficiency. It’s data indulgence. And its not for advertising.

Why?

MACHINE LEARNING.

If machine learning was a dog, then data would be the oxygen, the food, the water, and that fake chewy bacon stuff. Put more succinctly, the richer the data, the stronger the machine learning, and the higher the impact.

Here are a few examples of ways to get indulgent with data:

• If you are writing a manufacturing system, consider capturing weather data along with all the shop floor telemetry.
• If you have a retail POS system, think about capturing DJIA data and the prevailing sentiment on the USA Today front page. You might improve your demand forecasting.
• If your app interacts with homo sapiens, consider adding a survey feature that allows you (or your customer) to quickly run ad-hoc surveys to gather, well, data.
• Capture more and more and more functional usage within the app.
• If your app has a search feature, really think about what search data you should capture. Channel Google and capture lots of context such as time of day, length of session at moment of search, along with all the data indicating results relevance etc
• If you have multiple products, combine these data (even if they were acquired and were engineered very differently) and see what you (or your "machines") learn.
• ….. you get the idea.

Challenge yourself to really evaluate what data just might be impactful.

And yes, with great data comes great responsibility. You need to be highly morale and transparent in your data practices. Good data doesn’t need to be hidden. It’s not the Big Mac sauce recipe.

There you have it. Shed the outdated data efficiency mindset, and embrace the fact that more data is undisputedly better. Its time for data indulgence.

Please “like” if you like (its my way of getting more data)

And please share if you “like” :)

Bob