Dynamics of Modern Data Solutions in the Enterprise

In my last article, I talked about a few key trends taking shape. Here, I'd like to delve specifically into modern data solution

For the past few years, I’ve had the opportunity to explore, develop, execute and maintain various solutions that depend effective insight generation from data– from digital analytics, segmentation and personalisation, marketing campaigns, web and multi-channel experience syndication, all the way up to helping to shape strategic solutions and influence roadmaps for strategic enterprise data platforms.

On this journey, I’ve had the opportunity to deal with various dynamics under which modern enterprises operate, w.r.t. enterprise data solutions.

I am sharing some of my key takeaways from my experience so far.

This will have 2 parts. Part 1 expand on these dynamics, while Part 2 will focus on key considerations when pursuing an “end state” enterprise data platform.

Data and the Enterprise – the fear of change

Data has always been ( or at least it should be ) at the heart of every decision in an enterprise. From hiring new candidates, to developing new products, to even something as seemingly trivial as deciding what caterer to use for the local canteen. However , this isn't always the case.

While there is much to be said w.r.t. “following your gut” for developing new features of an app, or conducting a marketing campaign, it is increasingly becoming inexcusable to not supplement those decisions with data driven insights.

Historically, data systems have been fragmented, data pipelines expensive to build and maintain, and insights simply not available in a reasonable timeframe. Such a situation has lent credence to “intuition based decision making”.

While this would be tolerable when the data itself is not available, with advents of modern marketing and ad tech as well as data technologies, it is becoming increasingly less so; And yet, many organisations continue tend to “ go with their gut” rather than invest in their data capabilities.

The problem is essentially one of culture and resistance to change in face of the unknown

A reliance on “what has always worked” can be quite seductive, while admitting that we don’t know what we don’t know can seem quite intimidating, especially without knowing if it will yield fruit; Thus many departments, organisations, enterprises, continue to stave off investment in their data foundations, the most common excuse being of not having the budget to go there right now.

However, unless we chip away at the issue, this simply results in the problem gets bigger and bigger until it threatens the very viability of the business.

To clarify, I do value intuition in  decision making – some of the most revolutionary success of our times could be attributed to visionary thinking that bucked the “ data based trends “ of the day to take great risk, persevere and ultimately succeed

What I am advocating for, is to invest in foundation capabilities, leverage the tools and insights available today to supplement our decision making process, helping it to constantly evolving towards greater and greater speed to market and accuracy.

This will require a significant culture change and will potentially be controversial, with some camps ( across technology and business alike ) seeing it as a threat to their position and power. However , if adopted correctly, it can be quite the opposite, freeing up people to focus on higher value thinking and reducing the amount of grunt work that needs to be done.

This will invariable require unlearning, re-tooling and acquiring new skills across the spectrum and can be a very arduous change to manage enterprise wide.

The new frontier can be quite a scary place.We just have to have the will to develop the right culture to take those baby-steps, bit by bit , and ensure we enable and empower the people behind it.

Commoditisation of Data – The changing role of IT

The advent of new capability and/or technology often follows a typical trend.

In the early stages, only a few early adopters exist, evolving an maturing the capability to the next level. Over time, it becomes a standard offering, often forming the backbone of various services, products, functions etc. We still typically see, however specialists in the field, with the capability not yet accessible to the wider public

However gradually, the tech / the capability starts to become commoditised – it becomes far more commonplace – it becomes a key lever for achieving greater scale. Ownership begins to diffuse outside of the one dedicated team or set of specialists, with their role itself changing from that of an owner to facilitator of that capability.

We’ve seen this for infrastructure and hardware provisioning ( from dedicated specialists to self-service infra powered by AWS / AZURE / Soft Layer etc ) , Financial Services ( from dedicated SMEs guiding consumers to self-service direct customers ), Retail ( from big ticket monopolistic retailers to platforms providing access to indie and home grown goods ) and way more. The crowd-source economy is a good example for this trend

Access to data driven insights, is no exception. With more and more start-ups in this space and a large variety of solutions offered to business and consumers, the dedicated IT departments that look at big data and analytics within an enterprise are feeling the heat to keep up and provide a service that is scalable and in-line with expectations of today’s consumers.

With petabytes of data flowing in every day, and trends changing from week to week, if not daily, it is no longer acceptable to continue a model that requires months or even weeks of traditional implementation time before value can be realised. However, at the same time, it is imperative that the access and use of data is done within suitable governance framework, meeting privacy & regulatory requirements.

As a result, the next few years will see more of a “self service” model that allow effective, fit for purpose and compliant access and use of data driven insights.

Such an enterprise platform would have to cater for providing non-technical staff governed, yet flexible capability for data exploration, experimentation (including data science based algorithm) , and above all, fast operationalisation.

It simply won’t do to allow discovery capability only to take weeks to productionise and execute a data outcome.

The effect of this, more apparent in some industries than others is to have the role of IT changing, from being 'contractors' building capability for each request from scratch to platform owners offering a virtual menu of capability that business can avail of themselves.

Not only will this free up IT to invest in higher value adding capability such as automation and micro-services , it will also obviate the need of the business to stand up isolated data labs to get what they need.

Accountability of Data – The changing role of Business

It is not only IT facing change.Business to has a knock on impact to their roles. While commodisation covers aspect of widespread and ready access to data related capabilities , its actual governance, curation, execution and usage is still driven very much by the business.

As more and more capabilities shifting from traditional IT shops to business powered SMEs and centers of excellence, so too shift the responsibility of effective data governance and curation.

The data itself doesn’t respect the org structure. Customers don’t ( and shouldn’t ) care what department sell them products , vs which department manages support. Yet within a modern enterprise there is significant fragmentation. With many departments and systems producing overlapping , complementary and supplementary sets of data , what we end up with , at best , are inconsistent outcomes for the end customer.

In such a scenario, the Business need to take ownership of, shaping emerging data models across their organisation, ongoing curation and metadata management of data, and ultimately tapping into cross departmental synergies to realise value for their customers and themselves.

This goes far beyond just sourcing all data into a single data lake.

For example, a simple question of data mastery ( e.g. Which source is to be considered as master source for customer name, address, preferences etc ) – can become quite challenging , depending on the level of fragmentation in the enterprise.

Such decisions will not only have an impact on overall efficacy of data driven solutions, but can also have serious implications from consumer privacy as well as regulatory point of view.

In the past, these challenges would have been hidden away simply due to lack of fit-for-purpose solutions for bringing the data together and making it accessible.Today, with that block being removed, business personnel are put at the forefront of such significant and far-reaching decisions.

Ultimately business is accountable for how the data gets used and the up-skilling required to be able to effectively do that in light of modern data solutions should not be underestimated.

Analytics and Marketing CoEs

Continuing along the same theme of changing nature of roles, the analytics and marketing departments within the business also face the same scenario. 

Analytics and marketing departments are typically the bastions of the essential skills and data / analytics literacy within most organisations.

With an expanding role of the business in managing the effective use of data, such teams can no longer operate in isolation and must step up to the challenge to

1. Build up data literacy within the organisation.
2. Guide IT teams to build solutions to facilitate capture of contextual data across channels – e.g. hooks within application , front end and server side code.
3. Empower front facing business to be able to leverage data driven insights themselves e.g. providing SME access to and working with analytics dashboards and drill down reports
4. Unlock higher value capabilities within the analytics space – e.g. advanced data science and modelling , predictive and prescriptive analytics

In this way, marketing and analytics team need to hand over some of their tradition roles to other teams, generate new value and ultimately operate as centres of excellence within a federated operating model between Technology and Business .

**

Well, those were some of the big ticket dynamics at play – each one of them could be spoken of at length, but I hope this short list is of relevance for you all.

What’s your experience in trying to build out strategic data solutions for the enterprise ?  Was there anything major that was missed ? Let me know in the comments

In the next blog, I talk a bit more specific about some key considerations to make when mapping the way to a strategic end state data solution for the platform.