Generative AI: It's all fun and games until it becomes mission critical

Generative AI (GenAI) is transforming how we work. It’s evolving into an essential enterprise tool, capable of answering questions, creating images, summarizing information, and writing documents and source code. AI is augmenting human creativity and productivity with capabilities that were previously assumed to be the sole domain of humans.

GenAI isn’t just simplifying tedious work, but it’s doing things that are hard for humans to do at scale. In software development, for instance, GenAI tools can not only handle the tedious and time-consuming task of writing source code comments but also explain what the code does, a task that even experienced developers may find challenging.

And GenAI is addictive.

After experiencing the benefits of GenAI going back to the old way of doing things is challenging. Despite the acknowledged limitations of Large Language Models (LLMs), the substantial productivity gains they offer are sufficient motivation to justify continued use.

Gold Rush Thinking

Transformational technologies like GenAI emerge infrequently, yet when they do, they trigger a new kind of gold rush. Consider how the Web permanently changed how we share and discover information, as well as how we connect with one another. Forward thinking entrepreneurs recognize the potential to leverage the power of transformational technology to create new and uniquely differentiated products. Startups proliferate along with VC funding. At the same time, businesses see threat and opportunity.

Threats

An existential threat exists in the fear that other companies might move faster to embrace new technology, elevating their competitiveness. Bold public statements by the CEOs of other companies about their investment in AI adds fuel to the fear of being disrupted. In a time of transformative change, being a fast follower is not a good strategy; every company needs to be on the leading edge of evaluating emerging technologies.

Opportunities

Threats serve as motivation, compelling businesses to seek opportunities to achieve tasks previously deemed impossible, excessively labor-intensive, or financially impractical. As productivity improves and creative work becomes automated, considerations arise regarding the skills and personnel required to manage the business. The prospect of enhancing the bottom line by cutting headcount and overhead instead of investing in the re-skilling of existing employees leads to a predictable impact on employees.

Accelerated Development

Given this business-impacting context, it’s no surprise that enterprises are swiftly transitioning from experimenting with GenAI to actively adopting and deploying it. Initial experimentation showcased the technology's potential to transform the businesses, and the question no longer revolves around "if" but rather "how soon?"

Concerns regarding the use of public AI services, like ChatGPT, where employees might inadvertently disclose proprietary information in their queries, have led enterprises to opt for deploying and running GenAI services on-premise or in private clouds.?

Enterprises are also seeing tremendous value in training custom LLMs on proprietary data, and using these models to build custom applications that are aligned with business-specific use cases.

The unusual gold rush mentality is driving internal teams to rapidly develop and deploy GenAI solutions. The technology’s potential is not only driving innovation within traditional technology innovation teams but is also enabling creative solution development elsewhere. And it’s not just developers building on GenAI APIs, but also semi-technical and non-technical users that are crafting custom prompts for GenAI-based chat applications that address real world business problems.

Some enterprises are outpacing startups in their GenAI solution development, benefitting from ample funding and resources, and unburdened by the necessity of pitching to investors for the same. It's not uncommon to hear a startup pitch only to realize that the company’s internal team is further along. It's crucial to bear in mind that this is a gold rush, and opportunistic startups are hastily wrapping GenAI services, as they try to capitalize on enterprises' paranoia about falling behind.

That’s not to say that startups aren’t creating interesting products; they are, but the AI innovation arena is no longer the exclusive domain of startups. The availability of open source and commercial LLMs have leveled the playing field for everyone.

So far, so good. Innovation and the application of emergent technology are aligned.

But is there a problem lurking behind the scenes?

Mission-Critical Risk

The problem lies in the addictive nature of GenAI. Once adopted, breaking the habit becomes difficult. The enhanced day-to-day productivity from the use of GenAI tools creates a reluctance among employees to discontinue their use. Consequently, hastily developed prototypes and proof-of-concepts start to become a normal part of how the enterprise functions. Over time, these prototypical applications start to work their way into mission-critical workflows.

While it’s unlikely that someone would consciously choose to introduce a dependency on a proof-of-concept component into a mission-critical function, it can happen inadvertently. For instance, consider a GenAI application used to automate a business function that was previously a manual, time-consuming task susceptible to human error. A GenAI solution that saves time and reduces the risk of error is perceived as a net positive and, consequently, gets deployed and relied upon. It works great until it doesn't.

Unfortunately, an application rapidly developed as a proof-of-concept likely wasn’t architected for scalability, resilience to failure, quality, or performance. It probably wasn’t extensively stress tested in production workflows. It has no support team, SLA commitments, documentation, or run books for troubleshooting. The rationale behind not investing in these aspects is straightforward: the development team might not have anticipated the application being used in an environment requiring that level of rigor.?

So, when that application fails in a business-critical context, the blame inevitably falls on the team that developed it rather than investigating how it came to be used in that context. The team that developed the proof-of-concept are now on the line to rectify a business-impacting failure in an application that was never designed for use in a production environment.

This risk doesn't mean that enterprises shouldn't deploy experimental solutions or adopt Agile development of GenAI solutions. On the contrary, innovation should be encouraged, and solutions should be refined based on internal user feedback. However, it needs to be done thoughtfully and with a strategic approach.?

GenAI Strategies

Consider the following strategies to mitigate the risk of business-impacting issues from early adoption of GenAI:

1. Avoid Labels Implying Productization: Exercise caution with labels like "early access" or "beta" to prevent setting inaccurate support expectations and misguiding users about the path to productization.
2. Define Usage Boundaries: Clearly delineate where the GenAI solution can be used. Consider restricting usage to development environments ("dev") rather than production environments ("prod").
3. Implement Access Controls: Enforce access controls to limit usage to those actively participating in the proof-of-concept evaluation. This prevents unintentional deployment into business-critical functions through word-of-mouth referrals.
4. Impose Time Limits: Set explicit time limits for proof-of-concept usage. At the end of the evaluation period, shut down the application and gather user feedback through surveys.
5. Architect for Reuse: When creating foundational technologies intended for reuse, architect them with the same considerations as critical enterprise IT services. Prioritize scalability, availability, resilience to failure, performance, security, and overall engineering quality.
6. Instrument for Usage Tracking: Implement instrumentation in the application to monitor usage patterns. Sudden spikes in usage could signal a transition from limited proof-of-concept evaluation into broader production use.
7. Provide Guidelines for Use: Include clear guidelines about how the application can be used, establishing best practices, and promoting responsible deployment.
8. Create a GenAI Solutions Team: Establish a product team that can translate proof-of-concepts into production-ready applications and services. This should be a full-service team with a product manager, development team, a tech writer, QA, support engineer etc.

Summary

GenAI is a transformative technology with untapped potential, and enterprises should actively encourage and support experimentation. Creative ideas and valuable business use cases can come from anyone. However, it's crucial to incorporate guardrails to prevent proof-of-concepts from seamlessly integrating into mission-critical workflows, where their failure could have substantial business consequences. Embrace speed in innovation, but with a mindful approach to avoid unintended disruptions.