Holding Consulting Firms Accountable for AI Project Outcomes

As organizations increasingly turn to artificial intelligence to drive innovation and efficiency, the challenge of successfully implementing AI projects looms large. Recent reports from Gartner, Boston Consulting Group (BCG), and O'Reilly highlight the high failure rates of AI initiatives, suggesting that despite the growing interest, many projects fall short of achieving their goals.

For this article, I’ll use generative AI and AI projects interchangeably, understanding that most net new AI projects are also generative AI projects.? However, your AI projects may be traditional machine learning or deep learning use cases.??

A crucial piece of this puzzle is the role of consulting firms, which are often tasked with guiding enterprises through the complexities of AI adoption. Ensuring these firms are held accountable for AI project outcomes is essential for maximizing business value.

According to Gartner, 80% of AI projects in 2023 failed to meet their full potential or business objectives. BCG reported that 70% faltered due to poor data quality, lack of collaboration, and a disconnect between AI insights and actionable results. O'Reilly found that only 26% of AI initiatives progressed beyond the pilot stage, with obstacles such as inadequate infrastructure and misalignment with business goals posing significant barriers.

Of course, small AI consulting teams manage many of these projects rather than the enterprises themselves.?? ?However, much like any explosive rise of technology, such as cloud computing previously, as well as data lakes, large ERP implementations, etc., enterprises are often turning to big consulting firms to carry out the first instance of these solutions.?? ?While there is no information on exactly the number of consulting firms that are pushing these AI projects to failure, it’s safe to say that it’s about the same percentage as with the other net new technology projects in the past.??

Why big consulting is failing at generative AI.

Several factors contributing to the avoidance of accountability in AI projects can be attributed to the responsibilities of consulting organizations:

Misalignment of expectations, meaning that the consulting firms are responsible for setting realistic client expectations. This includes communicating AI’s potential and limitations and establishing well-defined objectives and KPIs that align with business goals. ?

Sometimes, this occurs with the alignment of marketing efforts by consulting firms, sometimes aligned with AI technology providers.?? In many instances, GPU companies have deep relationships with consulting firms and approach mutual clients together.?? This leads to inflated expectations around the business outcome of using AI, specifically Generative AI, and this often leads to not meeting those set expectations, and the project is thus being considered a failure.? ?

Lack of clear metrics means that the consulting organization is responsible for helping define success metrics that align with the client's business objectives. This involves setting up appropriate performance indicators and ensuring that the success of AI initiatives is measurable in terms of business impact.

I don’t know how many meetings I’ve been in in the last six months where the question is asked: “How are we defining success.”? This often results in blank looks rather than a clear list of business values.??? If you can’t answer that question, you have little chance of success, considering we don’t know what “success” is.? It would be best if you pushed your consulting team to define this with the business and do this before any architecture or strategies are formed.?? I can’t stress the importance of this more.

Focus on technology over strategy, referring to the more significant problem that consulting firms should ensure that the AI solutions they propose and implement are not only cutting-edge but also strategically aligned with the client’s business goals. They should work to integrate AI into the broader business context, emphasizing strategic alignment over simply deploying technology.

The symptoms of this problem are easy to spot.?? Consultants that blurt out “GPUs” or a cloud provider brand in the initial meeting without understanding the issues to solve.? Indeed, even promoting AI or generative AI early in the process is also concerning, considering we may not need AI for this specific business use case.??

AI is powerful and necessary for many business use cases, but it also adds significant cost and complexity. If not needed, it should be used. The consulting firm will need to have the courage to point this out, even if it means leaving money on the table.

The proper role of consulting firms for AI system development.

I believe in the role of consulting firms in that enterprises are able to access a group of experts quickly and use these experts to find an optimal solution.? ??I assisted in starting and selling several of these firms in my career, like the space, because you have the potential to do so much good.??? That is if you understand your role and can guide your work with your clients with great ethics and focus on their success, not yours.?? Consulting firms often need to keep closer track of this.? This considers the need to find revenue for the firm as the primary focus, getting to those big bonusses, not client success.? They are coupled concepts.?? ?

Consulting firms are pivotal in helping organizations navigate the complex landscape of AI. Their expertise should provide technical solutions and strategic alignment, ensuring that AI initiatives are integrated seamlessly into business operations. However, the high failure rates signal a need to hold these firms accountable for the outcomes of their projects.? So, what measures should client organizations take to ensure they find value with consulting firms around AI design, building, deployment, and operations??? I have a few suggestions:

Clear Objectives and KPIs: Consulting firms should be required to set clear objectives and key performance indicators (KPIs) at the project's onset. These KPIs should be aligned with the client's business goals and provide measurable outcomes.? These should be done incrementally around deliverables.??

This includes requirements, success metrics creation, architecture, conceptual design, physical design, technology selection, technology alignment, testing, acceptance, final KPIs, and proof that the solution is optimized to return the most business value. ??You’ll often find that they have missed the business value if they are reluctant to provide these or that they are too rudimentary to be helpful.?? This could be for many reasons, but the most common is using technology that benefits them but does not benefit you, the enterprise.? ?

Data Quality Assurance: Given the problems associated with poor data quality, firms must implement robust data governance strategies. Ensuring high-quality data should be part of the contractual agreement, with firms held accountable if data issues lead to project failure.?? This is not hard to determine, and this assessment should be done before any work begins.?

Regular Audits and Reporting: Establish a schedule for regular audits and progress reports. This transparency allows for early identification of potential issues and course corrections, if necessary.? If you don’t have the expertise to do these audits and assessments, use outside consultants that don’t have conflicts.??? Yes, experts such as myself (shameless plug).

Outcome-Based Contracts: Pay for performance is a good thing.?? Consider shifting to outcome-based contracts, where a portion of the consulting fee is contingent on the AI project's success. This creates a financial incentive for firms to deliver tangible results.? Consider making these payments contingent on passing the audit, as previously stated.???

Ethical AI Practices: Consulting firms must ensure the AI systems they implement adhere to ethical guidelines, avoid biases, and ensure decision-making transparency. This responsibility includes conducting AI ethics audits and training the client’s workforce.

Consulting firms should act as true partners, deeply embedded in the client’s strategic planning. This strategic alliance should focus on long-term value rather than short-term wins. Engaging with clients to develop a tailored AI roadmap can help overcome strategic misalignment and achieve sustained success.

The responsibility for AI project success does not rest solely on the shoulders of enterprises. As key drivers of AI adoption, consulting firms must be held accountable to ensure positive business outcomes. By implementing accountability measures, fostering transparency, and aligning closely with their client's strategic goals, consulting firms can significantly lower the high failure rates of AI projects. Ultimately, this approach benefits enterprises and enhances the reputation and effectiveness of consulting firms in delivering AI-driven business transformations.

?