Artificial Intelligence – Unraveling the Modern Chakravyuha - Part 2

In the first part of our article titled "Artificial Intelligence – Unraveling the Modern Chakravyuha", we introduced the various layers involved in constructing Artificial Intelligence (AI) systems. Now, let's dive right in and gain invaluable insight into their inner workings.

Layer 1: Data Delve: The Foundation for AI/ML Success: The Foundation for AI/ML Success: Gaining access to your own data (that represents your tacit knowledge) establishes the groundwork for your AI Journey. By doing so, you ensure the availability of high-quality data for training or fine tuning or doing RAG (iteratively). Be it Generative AI or classic ML, having access to data unique to your business (your tacit data) will set you apart from the rest.

Layer 2: Label Logic: Crafting Clarity in Data: Be it data activation or predictive analytics, labelling or annotating your tabular data or images is pivotal. Be it Gen AI or classic ML, supervising your model training is still a requirement (with varying degrees of frequencies). Only by labelling and annotating effectively, your models will be great at their task of predictions or classifications.

Layer 3: Feature Forge: Shaping Insights from Information: This step is an extension to the previous step and requires deliberate practice to push in the right representation of the data. For predictive analytics on top of tabular data, this step practically determines the success of the entire initiative.

Layer 4: Model Mastery: Crafting the Art of Intelligence: Be it Classic ML or Gen AI, modelling is the stage where the best of algorithms is married with the best of your data. This step is mostly left to the machines but requires human intelligence to maximize the value selecting appropriate algorithms, tuning hyperparameters, and evaluating model performance for desired accuracy.?

Layer 5: Validation Voyage: Navigating the Seas of Accuracy: Here models are validated on unseen data to assess performance and generalization ability. Be it LLMs or LVMs or Classic ML models, this is where your organization’s human talent needs to step in. As Models need to be validated on unseen data to assess performance and generalization ability, this step requires the most amount of human intervention.

Layer 6: Deployment Drive: Steering Models to Real-World Impact: Trained models are deployed into real-world environments for predictions or task automation. This involves integrating models into existing systems, monitoring performance, and addressing deployment issues.?

Layer 7: Maintenance Mode: Sustaining AI Excellence Over Time: Continuous monitoring, updating, and improving deployed models to maintain performance and adapt to changing conditions. This phase includes monitoring model drift, retraining with new data, and incorporating user feedback for ongoing effectiveness and relevance.?

It's crucial to recognize that each AI engagement is unique and so are the challenges. For instance, in a predictive analytics AI model we developed for a 150-year-old bank in the APAC region during 2021-22, significant effort was invested in the infrastructure, particularly on MLOps (applying DevOps and Software Engineering practices to the ML world). Although the bank possessed ample data, progressing beyond layer #2 posed challenges, with layer #6 being the most formidable.

Conversely, a similar AI model built to classify borrowers for a financial institution in Sioux Falls in 2019 faced data scarcity, hindering the accuracy of the system. Our client was not willing to accept that limited data quantity and model accuracy were connected despite us repeatedly emphasizing it. This illustrates the absence of a "one-size-fits-all" approach to AI system development. Each business context demands a unique approach, requiring prior experience, intuition, analytical skills, and resilience to navigate failures for a successful AI implementation.

Reflecting on the legend of the Chakravyuha from the Mahabharata, wherein only 9 warriors could unravel its complexities, it's evident that knowledge isn't merely about memorization but realization. Similarly, AI implementation demands expertise, intuition, and adaptability to navigate complexities effectively.

Now, how does one find the right experts who can help them enter the Chakravyuha called AI and bring them out safely by successfully deploying the AI model the business needs? Having spent 266112 hours on AI implementation since 2019, we've classified AI experts into two categories as "AI Charmers" and "AI Realists." What distinguishes them? Who do you choose for your AI implementation? Let's explore this further in the concluding part of this series next week.

P.S: Thanks to my colleagues Janarthanan Poornavel Dinesh Raman and Satish Chathanath for their valuable inputs here.