"Carpe Data" before it is too late

As companies digitalize their businesses, the data they generate is piling up. The idea that once intelligently exploited, this data could help them become significantly more productive is gaining traction. The idea is sound, but the reality is that progress remains unassertive. Companies are still agonizing over five key questions, which they should not allow to impede their progress for fear of being too late...

The explosion in the volume of data generated and stored worldwide is one of the main consequences of the digital transition. This effect, which McKinsey described back in May 2011, is accelerating at a breathtaking rate: together, we have all produced as much data in the last 10 months as in the entire history of humanity up to that point.

Conversely, the cost of data storage continues to fall. The result is that prudent organizations everywhere are amassing gigantic volumes of information by taking the attitude that says: "Who knows? It could come in useful one day." But for the time being, they are exploiting only a tiny proportion of it.

Little by little, we are seeing the emergence of the first examples of convincing data usage that point towards a future economy driven to a large extent by data: custom-written TV series, preventive maintenance in industry, 'analytical' hiring and retention of talented individuals, advertising campaigns optimized in real time, etc.

But with the exception of a few pioneers, the majority of companies are simply observers hesitating as they continue to develop their projects. In reality, they are often confronted by five complex questions, which I will address later in this paper.

However, I would like to state my belief right from the outset: these issues will be resolved not by taking a conceptual approach, but by embracing an experimental approach. Those companies that take the plunge, test, reproduce their successes on a larger scale and learn from their failures could very well establish an unbeatable lead over their more hesitant competitors.

Question one: how should the regulatory framework be interpreted?

When it comes to using data, companies face enormous difficulties in finding their way through the jungle of regulation. In the first instance, they are often multinational, and the legal frameworks they are required to comply with vary considerably between the USA, the EU and the other regions they operate in. Secondly, the directives issued by legislators are constantly changing in response to developments in technology and data usage. Lastly, there is often a multiplicity of stakeholders involved in setting constraints and/or with which companies must negotiate: legal bodies, naturally, but also independent agencies, such as the CNIL data protection agency in France, and even employee representatives (as in Scandinavia) or non-profits. Last spring, the European Commission adopted its GDPR (General Data Protection Regulation) to give companies greater forward visibility. They now have two years in which to comply and have a slightly clearer view of their rights and obligations. But we still seem to be a long way away from a stable regulatory framework, so active supervision will remain essential.

But there is another complication: in addition to legislation, companies must also address the issues posed by the virtual 'operating license' granted or denied to them by their customers. These 'licenses' effectively represent the agreement of customers to companies using their data, subject to two conditions: on the one hand that they take every possible precaution to maintain data security, and on the other that the value created by using the data concerned is shared fairly (for example, more accurate targeting of advertising that is less intrusive for web users, and a higher marketing fee for the publishing company). If a company falls foul of one or more of these conditions, its reputation can be severely dented.

Question two: how can data be 'uncorked'?

I have derived this unusual term from an image conjured up by Pentaho Chief Technology Officer James Dixon, who was the first to talk about the data lake concept. So what does it mean? Let's imagine that, until now, the majority of companies have stored and structured their data like 'filling bottles', and then grouped these bottles together in packs that they have then stored on shelves (their subsidiaries, divisions and branches in different countries). To use data stored in this way, you first have to find the most promising bottles and mix their contents by hand before you can even taste the cocktail.

But given the recent progress made in the cognitive sciences, it is now much more effective to pour all of this data out of its many bottles to create a data lake in which semi-autonomous algorithms 'fish' for useful and usable correlations between data items (the 'insights' we hear so much about). The main advantages of these lakes are that, like a real lake, their growth is unlimited, and they get over the issues of integrating disparate data types.

These lakes can contain the 'first party' data generated and owned by the company, the 'second party' data exchanged with partners and 'third party' data bought in from specialist companies that gather it online.

In many silo-structured companies, the change of IS architecture essential for creating a data lake is accompanied by misgivings about the transition or resistance to change, especially when it involves hosting such a large amount of data in the 'cloud', a virtual storage facility often hosted in a geographically remote location.

Question three: how can organizational silos be dissolved?

Pooling all the data held by a single company, let alone a corporate group, raises more than purely technical questions. The questions surrounding governance are at least as challenging. As senior executives gradually gain a better grasp of the potential value offered by data, the 'political' challenges typical of any company resource - from financial capital to talented employees and strategic information - begin to emerge. Lament this situation as we may, it is inherent to human organizations.

Inside corporates, it has often taken years to set up strategic committees and processes to manage such challenges effectively, collaboratively and with minimum bias, which could otherwise lead to bad decision-making.

In the same way, companies must encourage those that lead their divisions and branches to share their data by developing mechanisms that make it possible to redistribute the value created inside the company, either in monetary form or as symbolic capital. Appointing a Chief Data Officer, as around a quarter of leading companies have now done, could eventually facilitate this transition, but certainly not take it to its ultimate conclusion.

Question four: which skills are needed, and where can they be found?

As soon as companies engage actively with the issue of 'big data', they come up against the thorny issue of talent. The reality is that they need a number of people with extremely specialist skills.

Beginning with data architects. To return to the data lake analogy, these are the people responsible for the 'abstract system': their task is to ensure that all the data flows generated by each organizational entity converge in real time via a standard semantic system that makes them intelligible.

They then need data scientists. These mathematicians are specialists in statistics, who also come complete with robust programming experience. These are the designers of the algorithms and automated learning systems that will convert data to the insights that will give the company a competitive edge.

They will also need translators in the form of employees from individual business lines who have a perfect understanding of the challenges involved, but also have a knowledge of statistics or even the basics of programming in order to direct the efforts of the data scientists. Companies are quick to grasp the crucial importance of these individuals previously eclipsed by the data scientists.

Lastly, companies can choose the formal path of appointing a Chief Data Officer. On paper, his or her missions consist of managing a team made up of all the people listed above, of preparing, disseminating and measuring group data quality and security standards, of building partnerships to enrich the data available, and of leading innovation through data usage by selecting the most promising usage scenarios and learning from them. But alongside this formal role, he or she must also be an evangelist with the ability to dispel internal reluctance and actively advocate change.

So it's clear that what companies need is a complete 'skills chain'. But as everyone knows, any chain is only as strong as its weakest link. It is therefore in the interest of companies that haven't already done so to develop a specific HR plan for identifying, hiring, developing, rotating and retaining these exceptionally valuable talents.

Question five: how can the decision-making process be reinvented?

The fifth element is the one that enables the articulation of the previous four, and the one that I see as the most challenging, because it requires a change of culture within the company, and possibly on the much wider scale of society.

 It means making the transition away from a value system that promotes and rewards 'business flair' - the shrewd and instinctive acumen of the decision maker - towards a culture in which large and small decisions are widely informed by data. This transition can be seen as a kind of dispossession at every level: the banking adviser who may be recommended by the 'system' to push a particular product, the salesperson who will be asked to focus his or her efforts on a particular customer or the chief executive who should perhaps withdraw from a particular project, despite 'feeling good' about it.

It may be the case that a data analysis produces a result diametrically opposed to the initial convictions or intuitions. For example, a service company has discovered that the average productivity of its IT engineers improved when they shared their time between a dozen simultaneous projects, rather than focusing on two or three projects at the same time, as managers previously believed was more effective.

It will take time to adjust to this new business culture. Instead of the 'storytelling' of the brilliant chief executive, alternative narratives are needed, like that of Moneyball, the film that tells the story of how Billy Beane, the manager of the Oakland Athletics baseball team, used statistics to take his team to the top despite a meager budget. Top management must be the focus of evangelizing 'learning trips' to visit unfamiliar environments in which the use of data is already boosting performance dramatically, including hospitals, safety and sport. Lastly, as with any process of transformational change, it's important to learn from the initial demonstrable successes to amplify the dynamic within the company.

***

This takes me back to my initial argument: don't wait for each of these eminently complex questions to be answered fully. It may never happen, or if it does, it may be too late.

All the companies that I have seen achieve initial successes have one thing in common: they do not go over the top in intellectualizing the issues to be resolved. On the contrary, they tend to embark at the earliest-possible stage on experimentation with a small team, a single computer with no network connection to maintain security, and an incomplete database.

On this basis, they identify a small number of 'usage scenarios' in which data analysis gives them a significant and measurable competitive advantage: for example, an online retailer that has boosted sales using targeted buying suggestions, an insurer that has cut its attrition rate by implementing a preventive customer retention plan, and a major advertiser that has successfully optimized its media expenditure.

It is by building on these initial successes that they then create the dynamic internally, involve the highest level of management, and secure the flow of resources required. By embracing this approach, they have already covered a lot of the ground towards resolving the five difficulties described above, while their competitors continue to consider the issues 'behind closed doors'. And in some cases, this head start will prove decisive.