The Challenges Facing AI Innovation: Navigating the Data Lifecycle

I Recently had a discussion with a long-term client about some of the challenges that have prevented wider use of artificial intelligence. You don’t even have to look hard to find countless AI experts and the latest vendor to include AI in their product. However, scratching the surface, there are still some fundamental challenges that I feel warrant discussion. Interestingly when you examine surveys such as KPMG’s GenAI Executive Survey, 83% of respondents say they are going to increase their Gen AI investments. It seems an apt time to start thinking about, as I call it, ‘the plumbing’. Before we all spend a lot of money on the latest AI tool.

The Importance of Data Lifecycle Management

At the heart of AI innovation lies data—vast, complex and ever-evolving. Effective data lifecycle management is essential for harnessing the true power of generative AI. The data lifecycle encompasses several stages: data collection, storage, processing, analysis and disposal. Each stage requires meticulous planning and execution to ensure data integrity, accuracy, and security.

According to Gartner, poor data quality costs organizations an average of $15 million annually (Gartner, 2020). Additionally, a study by IBM found that 80% of data professionals spend more time on data preparation than on data science (IBM, 2021). These statistics highlight the critical need for robust data lifecycle management practices.

When we extend the lens to the current lessons from generative AI, Hallucinations and customers even restricting AI due to fears there is some great learnings to be had. Microsoft CoPilot, being something close to my heart. I was surprised on reading a Gartner paper around CoPilot for M365. That stated that the steps to mitigate the risks of AI was to restrict who can leverage it. The so called Trusted people. ?I found this statistic of 57% of respondents doing a smaller rollout to ‘Trusted people’ illuminating. Instead of trying to ensure that the pool of data is secured or that people have access to the right materials the solution was to only give it to ‘trusted’ people. To me this is putting a bandaid over the larger problem.

Challenges in the Current Wave of AI Innovation

1. Data Quality and Consistency

One of the most significant challenges facing AI innovation is data quality and consistency. Generative AI models require high-quality, diverse and representative datasets to function optimally. However, many organizations struggle with data silos, fragmented systems and inconsistent data standards. A survey from KPMG gives a clear direction with 66% Citing Data Quality is a key concern. (2024 KPMG GenAI Executive Survey).

2. Data Security and Privacy

The proliferation of data and the increasing sophistication of AI models raise significant concerns about data security and privacy. Organizations must navigate complex regulatory landscapes, such as GDPR and CCPA, to protect sensitive information. A survey conducted by PwC found that 62% of executives are concerned about data privacy risks associated with AI (PwC, 2021). Ensuring robust data governance frameworks and adopting privacy-preserving techniques are crucial steps toward mitigating these risks.

3. Scalability and Infrastructure

Generative AI models, such as GPT-3, demand immense computational power and storage capacity. Organizations often lack the necessary infrastructure to scale AI initiatives effectively. According to a report by McKinsey, only 20% of companies have the infrastructure needed to support advanced AI capabilities (McKinsey, 2021). Investing in scalable cloud solutions and high-performance computing resources can help bridge this gap.

4. Talent and Expertise

The shortage of skilled AI professionals is another hurdle for organizations. Building and maintaining sophisticated AI systems require expertise in data science, machine learning and domain-specific knowledge. A LinkedIn report indicated that AI specialist roles have grown 74% annually over the past four years, yet the demand far outstrips the supply (LinkedIn, 2020). Upskilling existing workforces and fostering collaborations with academic institutions can alleviate this talent crunch.

Solutions for Overcoming These Challenges

To address these challenges and unlock the full potential of generative AI, organizations must adopt a holistic approach to data lifecycle management.

1. Establishing Data Governance Frameworks

Implementing comprehensive data governance frameworks ensures data quality, consistency and security. Organizations should define clear policies for data collection, storage and usage, alongside regular audits to maintain compliance with regulatory standards.

Let’s think about this.? How useful is a presentation that Dave completed 9 years ago? Or how much do I really need that file I saved on my first day? I have always found that by human nature we like to horde, (my hording of choice is Lego). Moving into the business world do we have a structure as an organisation that is allowing our AI to feed off the best information? Or are we infact feeding it garbage and then being shocked that Dave’s presentation from 9 years ago doesn’t have the answers we were after for our customer today?

Manage the lifecycle, think about realistic length of data value. If a project after completion has lessons to learn then separate that from the 9TB of other information and then move the rest out of the primary memory. Reducing the pool of out of date information will help us get better answers.

2. Break the silos

Siloed data is a killer, over the last 20 years we have evolved from a person storing information in certain storage and needing to remember where each is stored. Organisations should be looking more holistically; can we explore centralising storage in one location or can we hook in additional data sources to give our AI more of a chance of finding the right answer. Try viewing your organisation as a new person, try and find all the information they need for their day to day or week to week. How easy is it? The answer if we look with fresh eyes is usually illuminating.

3. Data access

People should be taking the opportunity to explore what users have access to and tying into the Data Governance. There are tools out there to help with this and it can be a good opportunity to put an end to those file print servers lurking in the environment like the monster from the black lagoon. If we think about how AI tools such at M365 CoPilot work, they go wherever the person can access. If we set clear guide rails and lay the track for the AI we can ensure people don’t stray into dangerous territory.

4. Fostering a Culture of Continuous Learning

A culture of continuous learning and development is essential for staying ahead in the AI landscape. Organizations should provide ongoing training and development opportunities for their employees, fostering a growth mindset and encouraging innovation.

I have seen some great examples of this, and I have had the pleasure of working with KPMG on their Summer of AI and EY’s Roadshows all around AI. Which were real stand out examples of how organisations can take a massive step forward in preparing their teams for AI. Other organisations have been organising prompt tips and guidance. This is a key aspect to think about especially for users, if I am not teaching people how to ask the question how can we get frustrated we can’t get the answer?

Let’s expand it further and take a project, we might have some amazing learnings during the project. We might make new discoveries and improvements that we can bring forward. How are we ensuring those are coming up to the surface? How do we manage the challenge of referenceability and expertise. How easy is it in your organisation to find who the corporate tax specialist who has done a divestiture project for an automotive company in the Gulf Region? The lessons we learn in our jobs are some of the key learnings but as organisations we don’t necessarily make it easy to leverage that collective knowledge. In a programmatic manner which makes it easier for others to leverage our knowledge and lessons learnt.

Working at Microsoft I had a wide array of mentors, leaders and peers that I learnt from. One of the most influential on me used to challenge me constantly on what are the lessons and how do we replicate? We must examine within ourselves and our organisations how we do this.

Wrapping up

The current wave of AI innovation holds immense promise, but it is hindered by organizations unprepared for the complexities of the data lifecycle. By addressing challenges related to data quality, security, infrastructure and talent, companies can unlock the transformative potential of generative AI. As we navigate this ever-evolving landscape, a strategic and holistic approach to data lifecycle management will be the key to sustained AI success.

Note: I have written this by hand, AI has been used to fix spelling punctuation and grammar. All sources cited are publicly available and belong to the owners of the documents.