Enterprise IT in the Age of Generative AI

How to unleash the power of Gen AI to Transform the Enterprise (IT)

In 2010, Marc Andreessen famously proclaimed, "Software will eat the world," ushering in a new era of technological advancements. Today, we stand at the precipice of another transformative phase for technology -?where AI is set to eat software itself.

We officially entered a hype cycle with Generative AI, taking the world by storm. We witnessed records in tech product adoption with ChatGPT, ballooning and public and private market AI companies evaluations, and Gen AI becoming the most discussed topic in almost every media, conference and boardroom.?

After years of seeing tech fads, finally, enterprises are in front of a remarkable opportunity to leverage the tectonic shift in AI to innovate and deliver unparalleled business impact.?

This time it is different because, with Gen AI, the cost of cognition and creativity is dropping.

As we are just beginning the journey, everyone has more questions than clear answers. Today technology executives need to quickly navigate this new area to come up with updated strategies and prepare people and organisations for the future. That's why I wanted to share my perspective and recommendation on harnessing the AI tidal wave impacting Enterprise Technology and Organizations.

This time it is a real deal

The complexities, talent scarcity, inherent risk, and historical underinvestment in automating IT operations make enterprise technology an ideal area for automation and AI adoption. Gen AI has the potential to revolutionise the way enterprise technology works and creates value.

Through the broad adoption of Gen AI, IT organisations can build new avenues for growth and finally deliver on the long-time aspirations to create strategic competitive differentiators and advantages for the Enterprise.?

What is Generative AI

Before we move forward, let's take a moment to demystify Generative AI.?

Under the "brand" of Generative AI are Large Language Models based Foundational Models, like GPT-n powering ChatGPT, BERT behind Google Bard, DALL-E used by Midjourney, Anthropic Clude, Meta LLaMa etc.?

A foundation model is a large-scale, pre-trained, multi-modal, and transfer-learning-friendly machine learning model using neural network architecture. Foundation models are designed to be fine-tuned for a wide range of downstream tasks, such as natural language processing, computer vision, and speech recognition. The latest advancement pushing the performance and capabilities of Gen AI adopted the Auto Regressive-Large Language Models (AR-LLMs).

LLMs work so well by capturing wisdom-of-the-crowds through large-scale training corpus of text and images and further refined by human-supervised discriminative fine-tuning. It allows for generating outputs by predicting the most probable token - e.g. word, or pixel, one after another. The principle is very similar to the observation by?Francis Galton?used to guess the weight of an Ox at the English country fair in 1906. Today it is made possible through the implementation transformer architecture using billions of parameters, typically from 1B to 500B, and using mind-boggling training data sets between one to a few trillion tokens. Transformers are a type of neural network that can be used to learn relationships between different pieces of data.

The performance of LLMs seems magical, but they make mistakes such as factual errors, logical errors, inconsistency, limited reasoning, toxic bias, etc. LLMs have no knowledge of the underlying reality, no common sense, can't plan their answer, and do not understand the underlying world.

The current state of (AR-)LLMs are suitable for writing assistance, draft text generation, stylistic polishing, summarising, code writing assistance, generating the first set of ideas etc. However, they are not good at producing factual and consistent answers, reasoning, logical step-by-step planning, math, or using "tools" such as search engines, calculators, or database queries. Some of these limitations are being addressed through innovative engineering solutions, such as integrating 3rd party tools, e.g. via AutoGPT or building integrated AI systems around them.?

While modifying and adapting the foundational models is a deep research problem in the domain of Autonomous AI systems that will take quite some time to solve and address the inherent limitations of existing LLMs.?

But we have been using different flavours of AI for years, so how is Gen AI different? In the past, machine learning models were very specialised and often able to perform only one task, such as classifying objects in a photo or making a prediction. In contrast, one foundation model can perform both functions and generate content.?

A foundation model incorporating information about a company's products and operations could be used both to answer customers' questions and support engineers in developing updated versions of the products.?

Given the versatility of a foundation model, companies can use the same one to implement multiple business use cases; something rarely achieved using deep learning models.

To summarise, foundational models perform well in the following categories of tasks:

• Summarising: Large corpus of data or text, e.g. new regulation or policy, summarising the purpose of a legacy code base etc.
• Reformating: Convert unstructured messy data, e.g. system logs text and data into tables/structured.
• Classifying information and date: Help desk tickets based on the type of issues, users etc.
• Editing text or graphics: Quickly modifying UI design, proofreading release notes etc.
• Answering questions: About security policy and requirements.
• Drafting: Sample code when boilerplate is unavailable, incident root cause analysis report etc.?
• Brainstorming; Generate the first set of ideas for the project change management plan, critique board paper, etc.

As technology evolves, Generative AI capabilities are getting integrated into enterprise workflows to orchestrate tasks, automate specific parts of work and augment people.

AI Assistants - Augmentation, Automation, and reimagining work

Digital assistants, long promised and so far disappointing in delivery of the promise, are finally catching up. From MIT's chatbot Eliza in 1966 to today, progress has been steady, gradually improving digital bots' ability to understand and deliver nuanced responses. The big breakthrough that triggered people to believe in the potential of bots again happened in November 2022, with ChatGPT becoming GA and taking the world by storm.?

Google's AI Bots promise, "More than answers, we'll help you when there's no right answer," catches the essence of the step change that happened in the last months. The shift triggered the beginning of the augmentation phase. Now AI bots have started taking the role of every knowledge worker's personal assistant and augmenting their work.

It is happening fast with the proliferation of various co-pilots across different work domains. Big tech companies have already started embedding co-pilots into every possible platform and productivity tool, driving mass awareness. At the same time, we witness a Cambrian explosion of specialised AI startups creating AI assistants for every possible knowledge and white-collar work category.?

Despite not being widely discussed, it will equally impact all Enterprise IT organisations, both from the skills perspective and the nature of work. Technology organisations of all sizes need to develop relevant expertise and experience to capitalise on the changes, leverage Gen AI, and reimagine the organisation, nature of work and IT strategy.?

Machines will not replace humans in the short term; humans with machines will replace humans without machines.

Examples of application of Generative AI in IT Organisations

With Gen AI, English is becoming the de facto universal computer interaction language - programming, data science, admin etc., allowing non-technical users to develop fully functioning software and complete solutions to their specific business challenges.

The development of custom internal tools will explode. We will quickly see the move from a current state of shadow IT with citizen development and no-code solutions to an environment rich in custom full-stack business applications developed by business users with the help of Gen AI, replacing tools like RPA and specialised SaaS tools and apps.

Similarly, as English becomes the universal language of data analysis, the long-promised vision of data science and analytics for the masses will come to fruition. In this case, Gen AI is addressing one of the biggest bottlenecks: an acute shortage of data skill and talent, and finally making the concept of citizen data scientist possible.

Gen AI applications in Enterprise IT are broad and relevant across all IT domains. The use cases range from using out-of-the-shelf AI tools and specialised co-pilots to developing tailored AI-enabled systems depending on your organisation's priority and potential benefits.?

Below are real-world examples of what's possible so you can kick-start your exploration:?

Software Development:

• Generating code: using developer co-pilots to generate code saves developers time and effort, e.g., boilerplate code or code that implements specific algorithms or data structures.
• Code migration and refactoring: use specialised Gen AI tools to migrate existing codebases from a legacy language or framework to a modern tech stack.?
• Code testing: use co-pilots to generate unit tests and test data, optimise test cases by analysing the existing set of test cases, remove redundancies and improve the test coverage. Automate the testing execution, analyse the results, identify why a test failed and suggest potential fixes. AI will be essential in the implementation of the test-driven development process.?
• Predictive analysis:?can predict which areas of the codebase are more prone to bugs, based on historical data and the complexity of the code, guide developers in writing optimised code and more focused test cases for these areas.
• Documenting code: automate writing and update code comments, documentation and summaries across code base and modules, generate diagrams, identify areas with insufficient or outdated comments and create training and tutorials.??
• Enhancing security: Proactively assist developers in code development resistant to common and context-specific security vulnerabilities, update threat models, ingest code to identify vulnerabilities combined with context, e.g. Infra as a Code (IoC) setup, and recommend infra config baseline adjustments to counter specific attack vectors.
• Improving performance: Automate code optimisation for specific hardware platforms, use fewer resources or recommend resource allocation changes.
• Automating code reviews: Generative AI can automate code and IoC code reviews regarding adherence to best practices, standards and policy compliance.?
• Training: Provide personalised developer training based on the code contribution history and product contribution?

Information Security and IT Assurance:

• Threat intelligence: analyse large amounts of data to identify emerging threats and patterns of behaviour, enabling the deployment of proactive security measures.
• Vulnerability scanning: scan and analyse exposure to vulnerabilities based on the latest announcements, and provide prioritised plans and remediation recommendations.
• Incident response: automate issue or incident response tasks, such as triaging alerts, identifying affected systems, and coordinating with remediation teams, as well as summarising tasks based on incident response calls and drafting incident reports.
• Compliance: automate compliance tasks, generate reports, track remediation progress, and identify gaps in process or policy compliance.
• Security awareness: automate the generation of personalised security awareness and training in an always-on form tailored to specific individual needs.

IT Governance:

• Risk assessment: analyse large amounts of data to identify potential risks to the Enterprise's IT applications, systems and infrastructure, generate threat scenarios, and identify single points of failure in design, recurring incident patterns and resolution recommendations.
• Compliance: automate regular compliance tasks, generate reports, track remediation progress based on project data, identify gaps in compliance with policies and regulatory requirements, detect out-of-date info and automate updates.
• Audit: automate certain aspects of IT audits, such as data gathering and analysis, reviewing IT systems and infrastructure for compliance with policies and procedures based on unstructured data from systems and applications as well documents and artefacts provided by the users.
• Efficiency: reduce time and effort by automating reviews and approvals based on policy compliance, budget tracking, planning optimisation and uncovering dependencies, and transfer pricing optimisation.
• IT asset management: identity, update, and track IT assets, recommend asset utilisation optimisation, compare vendor solutions and offers, generate reports on asset usage and compliance; track SLA adherence; and recommend asset remediation plans based on patching announcements.
• Resourcing and talent management: Perform ongoing skills gap analysis, assess talent needs, and recommend training curriculums based on analysing current staff profiles, new strategies, and delivery roadmaps.

UI/UX Design:

• Design: generate design ideas and concepts for new UI/UX features based on natural language descriptions and examples, speeding up the design process and improving the quality of the final product.
• Testing: recommend best usability testing scenarios and strategies for new features and, identify potential problems with usability or accessibility, automate the generation of design variation for testing.
• Personalisation: Generate microsegment-based personalised UI and experiences for each user segment, improving user engagement and satisfaction.
• Analytics implementation: Generate analytics tagging plans and code based on simple prompts.
• Analytics: Analyse and summarise user behaviour and qualitative and quantitive feedback to improve the UI/UX experience and generate recommendations for testing and experiments.

IT Operations:

• Automating tasks: automate repetitive tasks in IT operations, such as ticket management, configuration management, change management, and communication. For example, L1 and part of L2 support can be automated through chats that understand the specific application or system context and past resolutions.
• Incident Response Automation: automate certain aspects of incident response, such as initial data gathering, impact analysis, communication and additional info request and even some remediation steps.
• Provide insights: insights into IT operations data, such as performance metrics, security logs, and user behaviour. It helps IT staff identify potential problems, troubleshoot issues, and optimise IT systems and processes.
• Improving communication: Generative AI can automate the generation of tailored and contextualised communication between IT staff and other stakeholders, such as announcements, regular updates, and user guides, ensuring everyone is on the same page and IT is meeting business needs.

Data Engineering:

• Data preparation: automate cleansing, transformation, and enrichment, freeing up the time of data engineers and analysts.
• Testing and Training: create synthetic data directly from product requirements for model testing and training purposes.
• Data exploration helps engineers and analysts explore data for patterns and insights using natural language prompts.
• Data visualisation: create and recommend engaging and informative data visualisations based on simple prompts.
• Model building: support in testing the model optimisation hypothesis, making predictions and recommending solution and optimisation strategies based on the latest available research.

Tech Product Management:

• Generating new product ideas: generate new insights and seed new product ideas by analysing data on customer behaviour, market trends, and competitor offering gaps.
• User insights: provide insights into user behaviour, helping product managers understand user preferences and needs.
• Prioritising product features: analyse data on customer feedback, market demand, and technical feasibility to prioritise product features, enabling product managers to focus on the most critical and impactful elements.
• Personalising product experiences: personalise product experiences by analysing data on customer preferences, usage patterns, and context, improving user engagement and relevance.
• Testing and evaluating products: Generative AI can simulate real-world use cases and generate feedback from simulated users, helping product managers identify potential problems and improve the user experience before releasing the product to the market.
• Customer success: Generative AI can analyse customer feedback, discovery call transcripts, and support tickets to identify areas where customers are struggling and provide actionable recommendations for improvement.
• Automating product lifecycle and team management tasks: Generative AI can automate tasks such as generating reports, reading Jira tickets, generating OKRs, and facilitating regular meetings and product communication and Training.
• Product planning: Generative AI can automate tracking and effort estimation and provide productivity and capacity planning recommendations.

Today, we use software to do things, but with AI, the nature of interaction will change, abstracting and reimagining how the user experience and interactions look.

IT Strategy in the Era of Generative AI

"Strategy is about making the future happen, not just reacting to it," said Gary Hamel.?

Now with generative AI, senior technology leaders suddenly have a fantastic set of new tools to shape the fortunes of their organisations.

In the next two years, we can expect organisations to gain 20%-30% efficiency improvements through Gen AI. Very soon; widespread adoption will become the expected standard. Investors will use it as a leading indicator and demand an articulated AI strategy and plan from each Enterprise. At first, focused on driving efficiency, followed by a product differentiation strategy, impacting market share, growth prospects, and, ultimately, the company's evaluations in private and public markets.?

In this new world, CIOs have no choice but to kick-start the work with the rest of the executive team to update their IT and AI Strategy.?

AI has the potential to close the gap between small and big organisations. Traditionally, small companies did not have access to great developers or could afford significant tech investments. With Gen AI, the promise is that even small organisations and communities will be able to develop advanced applications and solutions without massive investments or bringing in expensive consultants, democratising the impact delivered through technology.

As always, everything comes at a price. With Gen AI, it's essential to understand very early the economics of Generative AI and use it to develop a clear case for benefits realisation.?

At this initial stage of evolution, even services that are free or charged at a nominal cost to individual users, when used commercially, become one of the most expensive technology services available.

For example, the cost of a single request (inference) for a tool like ChatGPT on a variable cost basis is 10-100x more expensive than a regular search via commercial API.

The costs will come down over time, but they will always be much more expensive than the traditional services we have used.

One of the cost drivers is the high energy consumption of Generative AI applications. When developing the strategy, you must also consider how Gen AI-driven energy and carbon emissions align with the Enterprise's ESG ambitions and policies.

The pace of Gen AI advancements is also impacting IT and Enterprise strategy, requiring speed up the strategy cycles. The strategy needs to become genuinely iterative; an eighteen-month or longer horizon is way too long. At the same time, Gen AI can provide a good support strategy formulation and review. It allows us to synthesise information, prepare quick scenarios and trade-off comparisons, and make quick decisions and adjustments.

Revamping Enterprise Architecture and product management

A modern data and tech stack are key to nearly any successful business, and it becomes even more critical in the era of Generative AI. The space is rapidly evolving, with technology maturing; every day, the demand grows exponentially with new use cases, startups, and specialised vendor solutions.??

We are starting to see the first signs of the Enterprise Gen AI tech stack crystallising:

• Application layer: Includes end-to-end apps or third-party APIs that integrate generative AI models into user-facing products.
• Fine-tuning layer: Provides tools and ML Ops workflows for monitoring, quality assurance, continuous improvement, optimisation and governance.
• Model layer: Proprietary or open-source models (general, specific, hyper-local models) that power AI products via APIs.
• Data: A modern data platform enabling acquiring, refining, safeguarding, governing, and deploying data at scale and turning it into trusted, reusable data products.
• Infrastructure: Platforms and hardware responsible for running and training inference workloads for generative AI models.

With maturing tech stack, the AI product management domain is adapting to incorporate the latest thinking and best practices across hierarchical product categories:?

• AI model - foundational models, both private, open-source, specialised or hyper localised
• AI System - combines multiple models along with specialised AI components allowing access to private data, automating workflows etc.?
• AI Product - vertically integrated user workflow built around AI that allows the propagation of feedback and data back for training and continuous improvement of the underlying models, AI System, and the AI product as a whole.

Taking Action to Implement Gen AI in Enterprise Environment?

Step one - Review your technology and data ecosystem:

Most CIOs already have a sense of whether the company has the required infrastructure and technical capabilities in terms of computing resources, data systems, tools, and access to AI models, either commercial via big tech providers, open source, or via third-party model platforms and aggregators.

A clear data and infrastructure strategy incorporating Machine Learning requirements and use cases is critical.?

It all starts with a fundamental question. What do I need to do so my organisation's Enterprise Technology and Data platforms can be an effective data provider internally or externally?

The lifeblood of any AI is fluid access to domain-specific data sets fit for a specific business context or problem. Companies that have yet to find ways to harmonise generating data, capturing and providing easy access to their data sets will be unable to fine-tune AI models to unlock their transformative potential.?

Equally important is a scalable data architecture incorporating data governance, risk and security policies automation as a lubricant for efficient experimentation. Depending on the use case, the existing computing and tooling infrastructure (either cloud or set up in-house) might also need upgrading to support Ai-optimised computing platforms.

?

Step two - Review AI governance and risk frameworks:

Urgently assess whether the company's AI governance regime is sufficiently robust and covers expanded risk areas for Gen AI: Fairness, Intellectual property, Privacy, Security, Explainability, Model Reliability, Organisational impact, and Social and environmental impact.

Develop and build controls for assessing risks at the design stage and embed responsible AI principles into the AI architecture and products, e.g. via fine-tuning layer.

Step three - Develop hands-on experience:

Set up a cross-functional team that can tinker, drive the first experimentation efforts and provide a comprehensive perspective on the potential, risk and prioritisation of future work and investments.?

At the same time, you should start with:

• Educate and explain to as big as possible group of employees the fundamentals, purpose, examples and use cases where Generative AI is suitable and where it is not.
• Provide a safe sandbox environment where teams can experiment with existing and publicly available Generative AI tools and get hands-on experience. Allow teams to reimagine bottom up the future of work with Gen AI embedded by witnessing firsthand what it can do.
• Collect experiment results and use cases of how Generative AI can be applied to support day-to-day work and improve productivity across your organisation at the task and process level.
• Ask the cross-functional team to reimagine end-to-end processes and functions and define new experiences and business models that were impossible before the use of Gen AI.
• Leverage the cross-functional team to prioritise, consolidate into a program and determine the easiest way to validate and implement them as experiments or actual projects.?
• Double down on quick wins, capture learnings and refine your process, governance and strategy and communicate about success and education to build organisational enthusiasm and increase internal maturity.
• Based on the learnings and increasing maturity and setup of dedicated AI product organisation responsible for implementing the AI strategy, AI Sytems and AI Products with support from Data, Technology, UX and Risk functions.

Step Four - Be clear of the required level of sophistication:

Based on your organisation's experimentation, you should get a clear idea of where the organisation wants and needs to use Gen AI to extract the maximum benefits.?

There are three levels of increasing sophistication and maturity for companies that want to use Gen AI in technology, operations and products.?

• First - understanding and exploring how existing applications, e.g. ChatGPT, MidJourney etc., can be used to support individuals. It does not require deep expertise as it is about improving individual tasks, and only a basic level of company support and governance is needed.

??

• Second - Integrating 3rd part AI solutions into company workflows and products is where Gen AI starts to be really valuable for the organisation. Custom-build integrations open several opportunities to tailor and modify the product or service experience and maximise the value to offer a differentiated experience for your customers or employees. At this level, intermediate expertise is required to assess the suitability of the 3rd party solutions correctly, tailor and optimise it for the use case, e.g. set the context and engineer prompts, evaluate the risk and apply the governance framework.???

• Third - Organisations are considering developing their proprietary models, not necessarily from scratch but taking mature open source models, e.g. LLaMa developed by FB or others, and using your proprietary data to fine-tune the model for specific use cases. Naturally, as the market and ecosystem grow, more options are available, with buy vs build consideration for model development, foundational model training and optimisation. Naturally, having internal expert-level talent will be a crucial differentiator for success. Today AI talent is costly and very scarce.?

Step Five - Invest in People and Specialised Talent:

Moving forward, we need to rethink how work gets done. The focus must be on evolving operations and upskilling people as much as technology.?

If you want to gain an early mover advantage in your industry, the first step should be ramping up specialised talent and investments to provide fundamental knowledge to all employees on how Gen AI works and training on using available off-the-shelf solutions in the day-to-day work.

In the environment of austerity, hiring freeze or layoffs, the adoption of AI will accelerate. People at every level will have to think about optimising and automating the work with AI, as hiring an additional person, like in the past, is not an option. That's why training all employees is so important.

Investment in specialised technical capabilities and talent will depend on your company's ambition and the type of Gen AI implementation required.?

If the priority is to create products and solutions integrating pre-existing models and SaaS solutions, a team of seasoned data and software engineers could effectively integrate and manage 3rd party AI tools.?

Suppose the ambition is to work on optimising foundational Models. The critical skills required are training and optimising the models. It will require having AI Engineers, data architects and potentially machine learning experts with PhD-level qualifications. Optimisation experts need two sets of skills, hard and soft. Hard skills are about setting up and optimising infrastructure and processes efficiently, while soft skills involve model optimisation, which requires a keen eye and often involves making judgment calls based on intuition and experience.?

Conclusion

Generative AI is set to profoundly impact enterprise IT, revolutionising how organisations operate and achieve business outcomes. With the power of large language models and AI assistants, enterprise IT is entering a new era of automation, augmentation, and innovation.

By embracing Generative AI, organisations can leverage its potential across various domains, including software development, information security, IT governance, UI/UX design, IT operations, data engineering, and IT product management. The short-term impacts are evident, with efficiency improvements and enhanced capabilities across these areas.

Organisations must dedicate time and invest in technology architecture, talent development, and revise operating models to seize the opportunities Generative AI presents.

With Gen AI more than ever, ownership of unique large-scale data sets and data aggregation capabilities will drive value creation and create defensibility and business motes. A modern data and tech stack and scalable data architecture and infrastructure are essential.?

Equally important is nurturing a workforce equipped with the necessary skills to harness the power of Generative AI and reap benefits.?

Remember, machines will not replace humans in the short term, but humans with machines will replace humans without machines. So, are you ready to seize the opportunities and reimagine the possibilities?

The opinions and perspectives presented in this article are my personal views and do not necessarily reflect the stance or opinions of the author's employer.