Product Orientation to Enable the Strategic Potential of Data Engineering & Machine Learning

Data and the massive at scale analysis of that data by machine generated models can either be reactionary or visionary. It can definitely answer well developed questions in existing value chains. Which in turn drives significant incremental gains at the margins. Or it can be bold and open up entirely new avenues of possibilities; novel revolutionary revenue streams in both adjacencies and infill opportunities.

Nothing about these statements are controversial. Most industries have been aware of the potential of #Data&ML as force multipliers for over a decade. Most market makers are already reaping the epic benefits of pushing down data driven decision loops from the C-Suite to the increasingly automated operations layer of their business.

The controversial part is the following...as data increases in both volume & value...the strategic pivot defining long term success of an organization becomes the #DataEngineering and #MachineLearning level. The flow of prompt, clean, #MachineDerivedInsights to the various decision making edges of a company becomes not just an indicator of the operational health of an organizational entity...but also of its mid to long term strategic viability. Decision making needs to be made more collaborative and at faster tempos across distributed value chains and geographic markets. And that alignment can only flow from Data. And the initiative is no longer held in the operational engagement layers playing the role of customers of data...but rather the Data & ML Engineering as originators and internal marketers of information products that might not be visible or technically accessible to all stakeholders of a value chain.

Who wins and who loses in the marketplace? More and more that question can be inferred by contrasting the Data & ML engineering of a company vis a vis its industry peers. Who prevails in a #DataDriven world augmented with #PervasiveAutomation? Who can stand up #DataIngestion layers faster? Who has the #OrganizationalCulture to nurture the talent needed to leverage machines to #AnalyzeDataAtScale? Who has the #OrganizationalStructure to discover the spark of innovation out of the noise, harness it and make strategic leadership aware of the potential? In a world where Data decides the winner, these questions all have the same answer. Organizations built around Data. And by extension this is also true of other organizational entities beyond the economic sphere; which is the scope of this work.

How do you create a strategically minded Data & ML engineering nexus that can understand their internal consumers of data, innovate information products, make the case for these products to operational business units, maintain the quality of service of these subscription products and feed the timely distributed decision making necessary to compete in a 24/7 global world. Product Orientation. What does that mean? It means surrender the consumer orientation. Don't build for the present alone, build a vision of tomorrow and the day after. Market what you've built, search and scour the company looking for consumers of the data you have, trade specs and ingestion forms for conversations and relationships.

How to structure the team to deliver in such an environment. How to structure the team to stitch the operational environment together behind the scenes in a web of secure internal data? What skillsets are needed? What are the roles and responsibilities?

First, un-structure the team, embrace the chaos, live #Agile. To handle the workload and not miss fleeting opportunities, reduction of power distance between team members is crucial. Don't search for stability, rather build dynamic teams that need to be in motion. Identify on a continuous basis stagnant processes and prune them. Trade cultures of non-scalable superheroes and silent onlookers and risk-adverse bystanders for one of facilitators enabling hordes of engaged project based doers. Because in this era of rapidly evolving technology, where technical seniority means less and less each day, you should be hiring HowToDo's not WhatToDo's.

Embrace Data #BDD and #TestAutomation. Merge #QA and #ProductTeams to create nimble Product Pulls. The technical skillsets required to power the next level of #DE&ML are going to need data specific #ProductOwnership that can understand the changing value of the data. This is going to require a crop of skills that go beyond standard #Facebook product ownership or platform productization gurus. It will require an ability to understand #BusinessProcess at an integrated level. Not to drive the conversation, but to even have the conversation with the key stakeholders. It will require #QA that can create test cases before code has been written, develop and package data specific go-along test suites that can span the gulf between lower environment testing and smart business-oriented production monitoring.

And to find the right skillsets let go of arbitrary boundaries; if your #DataScientist are being bottlenecked by your #DataEngineers and vice versa, then you need to start exploring the Journey we've taken with our clients. We have all witnessed the massive transformations that have unfolded during the pandemic. But this is just the beginning, we haven't reached anywhere near a stable plateau in the field of Data Engineering & Machine Learning, we've only scratched the surface.