A Practical Framework for Understanding AI - Part 1

In this article I will present a simple framework to think about how AI works and how the frame work can be applied in any business process. It is not as important to know the details of engine on which the AI will run, or how the machine will cognize, but to leverage the specific calculations at scale. And to do that one benefits from understanding what must be provided in order to get an intelligent output (no pun intended). The way it is progressing, the need for understanding GPUS or cloud computing will soon become a reward less endeavor.

Before we go into AI, I want to emphasize that many problems in an organization where discreet results are expected can be solved with simple deterministic rule based logic. If decision ends up in predefine quantitative output, or if the output is predefined, rules are good— For a concrete defect criteria on a manufacturing line, rule based RPAs will suffice.

But most real world problems are non deterministic, otherwise world will be a different place and there will be no paradox of choice for us humans, and word like luck will not exist. We may need guard rails as deterministic boundaries within an AI system, but those don't change the core probabilistic nature of AI. It is almost certain that our results will be uncertain and our logic will be marred by random incomplete information, we can deterministically assume that cognitive outputs, ours or from AI will be probabilistic. There is no further dissertation I need to make here, but the simple case that even if the prediction looks solid as rock to our naked intelligence, it is the output with the highest probability (or a weighted random selection), chosen from a set of choices in multidimension probability space. In short we don't need to worry about if it is artificial or real. None of us can make the distinction between deterministic and probabilistic output of an optimally trained and tested AI output with gazillion parameters. Although I would say that context as an input to make the output context aware will be a good precautionary measure. Moving on then-

Things to keep in mind about AI :

1. AI in essence is pattern recognition in data
2. AI will repeat the pattern or reproduce in creative (yet) similar ways based on the recognized relations in data across dimensions.
3. AI will learn, adjusting its internal model based on implicit or explicit feedback.
4. AI can take actions

The pattern recognition is itself a type of learning, but then AI also "learns" after it has learned for the first time, or trained as we say. How do we distinguish learning that occurs to determine the pattern and the learning that occurs after the pattern is predicted. In our framework all we need to know is predicting the pattern, refining the pattern, and learning to refine the pattern are part of an AI system. As long as we understand which dimensions, features and data types concern our interests we will be able to define the problem in terms of AI solution.

Framework

There is a fundamental tension in designing any intuitive AI framework that we must contend with: simplicity vs. depth. On one hand, treating data as holistic inputs makes the framework easy to understand; on the other hand, certain applications demand a deeper understanding of features, especially when domain expertise biases the way features are interpreted or collected. In our approach we will gravitate towards simplicity for an overarching understanding that will still provide direction.

To work with AI we need the following concepts for our framework:

1. Data
2. Data Types
3. Feature Space
4. Core dimensions
5. Derived Dimensions
6. Pattern Recognition
7. Pattern Reproduction
8. Operational Layer

Delving into a brief description of each concept:

Data - The Raw Material, collection of data points

The input for the AI system to learn from, consisting of an aggregation of data points.

Data Types - Defining the Nature of Data

Assigns a single, aggregate identity to data (e.g. text, audio, images) and guides in thinking about possible patterns. These are not to be confused with programming data types.

Feature Space - Multi-Dimensional Properties of Data

Breaks down data points into measurable properties. Defines the structure within which patterns can be recognized (e.g., frequency for audio, pixel intensity for images). Think columns in spread sheet

Core Dimensions - Where Patterns Form

The fundamental relationships in the data are formed, typically across spatial (structure-based) and temporal (sequence-based) dimensions. Think time and space.

Derived Dimensions - Abstractions of Core Dimensions

Complex patterns that emerge from combining core dimensions of time and space (e.g., causal relationships, behavioral trends, or systemic effects). For most purposes AI will automatically data engineer where needed.

Pattern Recognition - Identifying Recurring Structures and Sequences

The AI’s ability to detect consistent relationships, trends, and anomalies across time and space dimensions. Think conventional AI.

Pattern Reproduction - Generating New Data or Predictions

The AI’s capacity to generate predictions, simulate new data, or reproduce recognized patterns to address future scenarios. Think generative AI, GANs.

Operational Layer - Take action to achieve a goal

Executes decisions based on recognized patterns and inputs. Taking actions autonomously to meet predefined objectives. Think Agentic AI.

Data

The sensorial advances guarantee that every moment in time and space can be recorded as a data point and there are inherent relationships among data points in feature space across time and space dimensions.

We may consider big data for some AI and it was a quintessential part of any digital transformation strategy, at least few years ago. I would like to add a "C" for "Compliant" to the 4Vs of big data, to encompass, data privacy and legalities of data collection, and usage.

While Big Data can power complex AI systems but it is not always necessary. For focused problems, smaller, high-quality datasets combined with smart algorithms can often deliver more efficient and effective results.

Thinking about data in our framework, we need to keep the following in mind:

Is the problem complex that requires deep learning AI patterns?

• Yes - Big Data might be necessary. (e.g. autonomous driving)
• No - Smaller, high-quality datasets might be sufficient. (linear regression, small sales forecast)

Is real-time data processing critical?

• Yes - We’ll need the Velocity aspect of Big Data. (IOT data in factories for predictive maintenance, high speed stock trading bot, fraud detection in payment system etc.)

Combining diverse data types?

• Yes - Variety is required, meaning Big Data will help (social media analysis, or multi-modal digital assistants.)

Is data quality (Veracity) crucial for decisions?

• Yes - Even small datasets need to be highly reliable (think critical environments like healthcare and finance.

There are applications in AI where big data may not be needed, or can be synthetically produced. Many domain specific AI models can be built with small data, or on top of existing models using small data. Simulations have also helped with creating data similar to real world.

This article is related to AI frame work, but I feel compelled to add few lines about data collection, for rounding out role of data in the framework. The three main sources of data are nature, humans and machines. Data is generated by natural events ( weather data, biological signals, cosmological events etc. ), human interactions( clickstreams, voice commands, social media activity, pretty much any human activity) and machine processes(sensor outputs, machine logs, telemetric data from various sources; autonomous vehicles to telecom networks. With IIoT all machine activity is bound to become data).

The three types of data can be collected as structured or tabular data (spreadsheets, databases), unstructured data( text, images, videos, audio) and semi-structured data (XML, JSON, log files). Since data is the raw material or fuel for AI engines, cleaner is better. A discussion of noise is out of scope of this article, we'll assume data collection and refinement as an implementation detail.

Data Types

The data most real world business applications deal with, typically falls under four major categories:

1. Numbers (e.g., sales figures, temperatures, timestamps)
2. Text (e.g., customer reviews, product descriptions)
3. Images (e.g., product photos, scanned documents)
4. Audio (e.g., call center recordings, machine noises)

We are purposely focusing on these core data types for simplicity and practical relevance. More semantically complex data types can emerge within the feature space, which we’ll discuss later. Each data type should guide our thinking to a specific pattern or relationship, and AI application. The role of data types to identify relationships between data points in core dimensions is listed below:

One can argue that from the machines' perspective all of the above are numbers and then basically just two numbers, (sometimes entangled-;). We can assuredly assume that for majority of people and organizations, computing resources are abstracted in cloud and easily accessible.

The other point is, we can define our own meta data types with combinations of numbers, text images, to expand feature space but this doesn't change our basic model of thinking.

Feature Space

Our concept of data is comprised of data points as the fundamental input to the AI system and for most AI applications the underlying implementation will automatically extract relevant features to recognize patterns without requiring the division of a data point into it's relevant feature space. For example an audio recording of a machine in manufacturing can be fed into a pre-trained audio model as an input or even used to train a model to generate audio. The feature space(which may consist of frequency, amplification and other sound attributes etc.) in this case will be abstracted.

In some cases, understanding the underlying features of multi dimensional data points allows domain experts to influence data collection strategies. Consider a product database. Each product exists in a feature space defined by its attributes (e.g., price, color, weight, brand, material etc.). We then we have numeric ratings, textual descriptions, or binary availability (in stock vs. out of stock). This creates a feature space with hundreds or thousands of dimensions. The product data in our frame work will then comprise of this feature space, which AI will use to find patterns across categories that wouldn’t be obvious manually. For instance, "certain products with longer lead times consistently sell more often than those with shorter lead times, regardless of category.”

In this frame work think of the feature space to help define the scope and complexity of problems AI can solve. The larger and more diverse the feature space, the more opportunities there are for the AI to recognize complex patterns and relationships.

Keeping in mind that a larger feature space increases the chances of finding hidden patterns relevant to business but at the same time too many irrelevant features can result in noise, which distorts pattern recognition hence focusing on quality features and reducing irrelevant features enhances performance without needing massive datasets.

The right feature space aligns AI with business objectives and that's where domain expertise and understanding of features in data points is required.

Dimensions

Before I begin on an explanation of dimensions, I want to clarify, we are not discussing the feature dimensionality as used in traditional AI/ML problems that leads to "curse of dimensionality". We are using dimensions to define the fundamental ways data varies across time and space. In our framework, core dimensions (spatial and temporal) define how data naturally forms patterns, while in AI modeling, feature dimensionality refers to the number of variables an algorithm considers when learning those patterns. We have defined a separate feature space as part of the framework, where data points form relationships along core and derived dimensions, providing structured guidance for AI planning and understanding.

Core Dimensions of Pattern recognition:

Putting aside the complexity of an AI model, we can think of two core dimensions, where the relations in data exist:

1. Temporal dimension (Time) - Patterns in sequences or changes over time (e.g., forecasting, prediction).
2. Spatial dimension (Space) - Patterns based on arrangement, similarity, or structural relationships in space or form (e.g., classification, clustering)

As the complexity increases the two dimensions above can be combined to form derived dimensions that are required for specific patterns like behavior, motion etc.

Derived Dimensions

To keep the frame work relatively complete, we need to consider some derived dimensions beside the core or "raw" dimensions of time and space. We can view them as derived attributes or contextual layers built upon the core dimensions. Most of the following fall under the "Spatio-Temporal" derived dimension.

Causal Dimension (Cause & Effect):

Understanding the direction of influence between temporal events (e.g., Policy change results in Market Shift). Consider "events" as a composite data point with feature space composed of multi-dimensional features—time, location, severity, and other properties that enable AI to uncover complex relational patterns in spatial and temporal dimensions.

Hierarchical Dimension (Scale or Granularity):

Zooming in and out between micro-level and macro-level patterns (e.g., global to local supply chain, or global to local market formation). This is mostly spatial but can have temporal interactions if the hierarchy changes over time. Example will be a disruption at the global supplier level impacts regional distribution centers, eventually leading to stockouts at the local store level, or a global trend in sustainable products is used for suggesting personalized eco friendly items in specific regions and user segments.

AI can learn to anticipate bottlenecks by recognizing patterns across the supply chain hierarchy or hone in from global to local trends in product marketing.

Relational Dimension (Networks & Graphs):

Connections between entities (e.g., social networks, citation graphs). Primarily spatial relationships but can have temporal evolution.

Frequency Dimension (Signal Patterns):

Time-based, but looks at cyclic patterns within the temporal dimension (e.g., audio signals, market cycles).

Without losing much benefit of dimensional details, our thinking can center around the feature space and the two core dimensions: spatial and temporal. All complex relationships within the data can be understood as interactions at intervals or similarities(or dissimilarities) in form and structure between points in this multi-dimensional space.

Patterns

AI brain in simplest terms recognizes patterns in data, mimics it in creative combinations and makes decisions based on a combination of decision rules and prediction. These patterns emerge from relationships across two core dimensions—time and space. These core dimensions as stated earlier form the foundation for prediction, classification, and decision-making.

Temporal Patterns (Time-Based) - Prediction & Forecasting

Temporal patterns focus on how data changes over time finding relations among sequences of data points to predict. Forecasting sales, predicting user behavior are examples of temporal analysis:

• Short-Term Predictions: Immediate next steps based on recent data (e.g., next-word prediction in text type data).
• Long-Term Forecasting: Extended future trends derived from historical data (e.g., stock market trends, climate modeling, inventory forecast, sales forecast).
• Sequence Prediction: Understanding sequences of events (e.g., user navigation paths on a website, DNA sequence prediction, next best offer).
• Temporal Anomalies: Detecting irregularities in sequences (e.g., fraud detection in transaction logs).

Spatial Patterns (Structure and Form - Based) - Classification & Clustering

Spatial patterns emerge from the structure or arrangement of data at a specific point in time. Unlike temporal patterns, they are not dependent on sequences but instead focus on how features relate in space or structure. AI systems use spatial analysis to classify, cluster, or detect anomalies in static data snapshots:

• Categorical Classification: Grouping data into pre-defined classes (e.g., spam vs. non-spam emails).
• Clustering: Grouping data into natural clusters without predefined labels (e.g., customer segmentation, ).
• Object Detection: Identifying objects or features within an environment (e.g., identifying tumors in medical scans, verify insurance claims with images).
• Spatial Anomalies: Finding outliers in spatial patterns (e.g., defect detection in manufacturing).

Besides the two core dimensions of time and space, we defined a set of derived dimensions earlier. Those dimensions are added to emphasize that AI also captures more abstract patterns across derived dimensions like causality, hierarchy, and relational structures. These patterns help explain complex relationships between events, entities, or features

Summarizing the core and derived dimensions along with types of commonly used patterns and examples of use are listed below. This is not an exhaustive table of examples by any means but it helps to keep the pattern buckets and few related examples in our frame of view.

Common types of AI,

To establish a shared understanding of the conceptual boundaries within AI systems, we’ll categorize some of the most commonly used terms. This will also help clarify implementation strategies when encountering specific AI terminology—namely, Generative AI, Conventional (or Discriminative) AI, and Agentic AI, which has recently gained broader recognition.

This article does not delve into the distinctions between machine learning (ML) and Artificial Intelligence (AI). For the purposes of this discussion, we assume that ML serves as an enabler for AI. In many cases, particularly with Conventional AI, the terms ML and AI can be considered interchangeable, and we will treat them as such for simplicity

Questions To Ask When Thinking Which AI

We have already gathered that AI itself is pattern recognition within dimensional data. Now to bring it all together the simple questions one can ask when thinking about any of the above AI systems.

When we are pondering "What is this? about data at hand then conventional AI comes to mind, which will helps us define predictive outcomes or similarity groups from patterns in our data.

Think of questions like "What can I create from this?", Generative AI should come to mind. It uses the recognized patterns in our data to create similar yet new data, be it text, or images, or audio etc.

Asking a question like "Is there a decision based action to achieve an activation?" and agentic AI should come to mind. This goes beyond pattern recognition and takes action, towards a predefined or self-learned goal. The agentic AI can use both types, generative AI or conventional AI in its system, example will be Agentic AI driver of a self driving car that can sense surroundings and predict (conventional AI), communicate with the driver (generative AI), make a decision and take next action.

Putting the Framework to work

This frame work is to help think about any task or process at hand and tie it with different types of AI capabilities. It is to help intuit a logical progression from basic data recognition to complex adaptive systems and set a foundation for thinking through industry-specific applications by plugging them into data, feature space, dimensions and pattern buckets.

To summarize,

• AI is pattern recognition (in space, time, or both) + reproduction (based on learned patterns), data types define the nature of these patterns.
• Dimensions determine the structure of (pattern)learned output using the data points(feature space).
• Understanding feature space helps aligning AI with business objectives
• AI can make decisions and take actions.
• AI can adapt or learn with feedback.

In the Part 2 of this article, I will put examples how the frame work can be applied to different industries.

Meanwhile I had some fun with my experiments with AI video, (thanks to LTX Studio, Pictory, PowerDirector, Sora, and Gemini).