Seven Challenges and Fallacies that Obstruct Big Data Implementation in the Enterprise

Making Strategic Decisions

Our CEO makes decisions every day. Some of these decisions are strategic, when visiting upcoming objectives for the next one or five years, or when considering an expansion strategy to capture opportunities in the market, or even lately to survive the pandemic. His strategic decisions then ignite a domino effect propelling business unit managers, directors and functional managers to make operational and tactical decisions. The quality of all of their decisions is directly linked to their capability of capturing facts, understanding contextual mandates in the industry, and their proficiency in eradicating cognitive biases.

Making the right strategic decisions to acquire, merge, initiate or kill an investment, or to head hunt an outside leader, CEOs need to rely on the intelligence collected by their people, and sometimes by specialized providers. The problem is that intelligence is as good as the foundations it has been built upon, i.e. the solid facts or bits and pieces of data combined together to formulate an opinion. This is where the context and those who write the report might not address the self-reflexivity - a concept that attempts to remove some of the personal perspective biases from the decision. Bottom-line, when the intelligence is not smart, not based on facts, it is better not to misguide the CEO with a sub-optimal dashboard and let him or her make judgmental decisions, as their father and grandfather did before, rather than base the decision on rather fictional report.

The Role of Big Data in Decision Making

With the advances in technology, the concept of big data evolved to solve unchartered challenges: When BI makers do not possess complete, clean and factual data, they attempt to make assumptions or become subjective in building reports that will be used by the CEO to make solid strategic decisions. Unlike BI, big data does not mandate the existence of fully relevant, cleansed, structured data in order to provide intelligence. It uses statistics, mathematics, probability, algorithms, heuristics and a lot of iterations and computational power to make up for the missing pieces, and still converge to one specific answer, hence providing an alternative to knowing a specific fact, by looking at a lot of the surrounding data, which inevitably are affected by the existence of this unknown specific fact, that is being uncovered in an indirect but rather solid manner.

Challenges in implementing Big Data in the Enterprise

There are quite a few challenges that have been limiting the spread and utilization of Big Data for making decisions in various organizations, be it in the government or public sectors. Unless we learn what these challenges are, and attempt to address them, our organizations will be left behind, and the quality of our decisions shall not match our future challenges and objectives. The following are not a fully inclusive list, but rather as findings and observations that are common - readers can authenticate easily by looking around in their own or nearby organizations:

1. Leaders' illiteracy

2. IT is the King

3. The Vendor is our Savior

4. Absence of Big Data policy, governance and standardized organizational processes

5. Roles, responsibilities are not defined

6. Skills and Competencies are not known

7. Architecture, Functions and algorithms do not exist

____________________________________________________

Challenge #1 | Leaders' Illiteracy

Organizational strategy, change and transformation always originate at the top, spreading down and across, but often implemented in a bottom-up approach. When the C-level executives are not aware of the importance of a certain concept for their growth and success, such as Big Data for better decision making, they will not seek to learn, nor ask the business units to build their capabilities accordingly.

Solution: In every industry, there are successful implementations for Big Data that provide a competitive edge for their respective leaders, affecting the quality of their strategic decisions. The leaders need to probe the more mature organizations, seek some coaching and perhaps even attend executive sessions on better decision making using Big Data. When this happens, and they see the value of the better decision and the immense ROI and improvement, resulting in the next disruptive transformation in the organization. All is geared by value; what is the value of betting on the right horse, and how much are CEOs willing to invest for possessing this future insight?

Fallacy #2 | IT is the King

Well, no more! The overarching problem and the misconception that has and will still drain numerous organizations is relying on IT because of three factors: Some challenge comes up that nobody knows anything about; second, the challenge contains some technical term that the novice can associate to to IT such as Big Data and AI, and finally the fact that IT has been working with one or more powerful vendors who are successful. This reminds me when a CEO asked IT to setup or operate the Corporate PMO, since there is a PMIS and dashboard involved, forgetting about Business Analysis, Business Case, Scope of Work, Planning, Estimation, Development, Governance, Risk, Quality, Procurement, Contracts, Execution, Stakeholder Management, Communications, Collaborations and what not, only because there is a software involved for the dashboard, so inevitably it should be IT! Big massacres have been the result. Mind me, I do not hate IT, but no sound minded CEO agrees to hand over budgeting to sales, or performance management to technical services, and what not, except when it comes to project management, hand it over to IT, and now of course Big Data, just the same. IT will not let go of the extra authority, funds and the fact that now people need them more; going back and resolving this mess is not easy and comes with a hefty price.

Solution: Big Data implementation is not straight forward in that one single entity, be it IT or other can handle completely on their own. Better understand the roles and functions, then a proper assignment for responsibilities can follow necessary technical functions, skills and competencies. For example Big Data Analysis requires specific knowledge in algorithms, statistics and some platforms such as R or Python; some good knowledge on traditional BI reporting comes handy. Perhaps strategic planning, business analysts and financial analysts might have some of these traits, especially if they can learn some necessary mentioned skills above. On the other hand, the Big Data architect might well be handed over by IT, but their role is limited to providing interfaces and connectivity to specific channels of information, and bringing them over for the Big Data analysts, so IT here play their technical role.

Fallacy #3 | The Vendor is our Savior

Often, and quite related to the second challenge, when business-related tasks are trusted with IT, they consult their trusted providers, who feel overwhelmed to increase their sale with another line of business, and we find ourselves subscribed to a specific Big Data solution. The problem is that without a proper selection criteria, crafted on the basis of understanding our current and future needs, we might be investing in a sub-optimal solutions. No solution should be selected before understanding the depth of the problem first, and looking for specific industry solution providers. What increases the challenge is that various decisions require different access to data sources, alternate algorithms and specific inference or deep learning engines to benefit from AI. In Big Data and AI, standardization might prove to be counter-productive, as no one-size-fits-all.

Solution: Selecting a technical provider, as with all types of technology selections and within the digital transformation era does not drive our selection, but rather comes as a result of assessing and updating our strategy, operating model, structure, customer-centricity, employee empowerment level and various cultural changes. Enterprises will require various data sources and different technology providers, based on the specific decisions being made, their maturity, ecosystem, context and various other factors. Hence better start with setting our policy, and understanding our readiness, our needs, based on required objectives, the current gap, and the existing challenges or weaknesses.

Challenge #4 | Absence of Big Data policy, governance and standardized organizational processes

Some organizations attempt to follow and subscribe to every trendy buzz word and new technology in a blind manner, as if this enhances their image or value; they wish! They are happy with using terms that they do not necessarily comprehend, such as Big Data and AI, especially in front of the shareholders and the media. A fool with a tool is just a fool, sometimes a bigger fool. Without some policies, processes and guidelines, together with the above challenges, we will find several, often counter-productive practices by various providers in a chaotic manner. Remember that a sound judgmental decision making process is better than a dashboard with cosmetic or sugar-coated KPIs. Another dimension is the possible leakage of proprietary data, as cybersecurity is mostly about common sense and disciplined policies, more than just firewalls and technical gigs.

Solution: Executives have been hiring management consultants to develop or update their operating model, restructure the organization, or conduct an upskilling assessment, not to mention the many SME hires for conducting feasibilities or market research. Just the same, implementing big data in an organization requires the development of a blue print that links it to the value-chain with a clear policy, processes and assignment of accountability to make this happen and realize the needed objectives in a measurable manner, with relevant KPIs or OKRs. Together with spreading awareness, this is a most important activity to start with.

Challenge # 5 | Roles, responsibilities are not defined

Inevitably, with the absence of a clear policy that aligns the use of big data to the value-chain, no governance is present, hence the various roles for the successful implementation of Big Data for proper decision making are not identified. When roles are not identified, it follows that competencies are not known, nor their upskilling needs. Why should we bother, when IT can take good care for this?

Solution: Let us start by dividing the roles into two categories; the first is the decision maker's, who needs the awareness addressed in the first challenge, followed by competencies that enable the same decision makers to understand the framework of big data, and how to ask the right questions that they did not have the privilege to ask before. This is a must for every executive. The other roles deal with answering decision makers' inquiries, and this requires more than one set of talent. Big Data Analysts usually use existing algorithms on big data to provide answers, with some tailoring based on each type of inquiry. To provide them with relevant content, a Big Data Engineer might prove to be handy, but the one who connects and interfaces with existing sources of data, be it structured or raw, inside and outside the organization is the Big Data Architect.

This role provides much needed protection by not ascribing to one solution or source all the time, but act in an agile manner according to the need. Last role is quite hard to build, and could be an external SME, which is the big data scientist, who will be capable of inventing new heuristics or algorithms to resolve specific challenges not handled efficiently by existing algorithms that big data analysts are aware of.

Challenge #6 | Skills and Competencies are not known

These challenges formulate a continuum, as they are tightly connected in a causal relationship. How can we know the needed competencies before identifying the required roles, but also, how can we define roles without their associated competencies.

SOLUTION: The Enterprise Big Data Framework, founded by Jan-Willem Middelburg defined five roles, for each there is a standard, with specific competencies defined with the SFIA Foundation framework, which is a separate topic on its own.

Challenge # 7 | Architecture, Functions and algorithms do not exist

Within enterprises new to big data, there is no existing framework that addresses the needed functions to serve the purpose, within an architecture that serves the value-chain. Though, algorithms are very well known and available, but not necessary elected to suit the relevant industry challenges yet.

Solution: The utilization of the open Big Data Framework will answer each and every one of the above listed fallacies and challenges that hinder proper implementation for Enterprise Big Data. The big data framework consist of six building block which at all connected:

1. Big Data Strategy
2. Big Data Processes
3. Big Data Architecture
4. Big Data Functions
5. Big Data Algorithms
6. Artificial Intelligence

Organizations can become data driven by growing capability in all six building blocks, therefore it is a model that every organization can use as a reference framework, irrespective of vendor solution. This is perhaps the most critical and essential requirement for Sovereign Organizations.

ADVISORS is APMG International Accredited Training Organization (ATO) for Enterprise Big Data, providing executive workshops, consulting on implementing the non-vendor specific framework, accredited training, including the prestigious UK certification.