Unstructured to Structured Data: Domain-Specific AI-Agents Driven Systems of Intelligence

Introduction

In enterprise software, it's long been believed that building a lasting competitive advantage—or moat—requires a system of record. Notable examples include:

• Salesforce: System of record for customer data.
• Workday: System of record for employee data.
• ServiceNow: System of record for IT and customer support data.

While these central repositories of critical business data are important, they aren't the whole story.

The True Value: Workflows and Human Interfaces

The real value—and the true lock-in—comes from the workflows and integrations built around these systems. These applications are not merely data repositories; they are tools designed for humans to input and manage data. Key characteristics include:

• Human Interfaces: Interfaces that help employees manually transfer information from one source to another.
• Manual Workflows: Systems that rely on users navigating UIs to input, process, and move data.
• Human Dependency: The design centers around people interacting with the system, making these applications essential for companies.

The significant switching cost lies not just in replacing the data but also in the workflows tied to these systems.

The AI-Driven Shift

With the advent of AI, particularly large foundation models, this dynamic could radically change. AI excels at processing unstructured data into structured data—the very data that humans currently input into systems of record.

Challenges in Current Systems

• Data Entry Burden: Sales reps often dislike entering data into systems like Salesforce after every call.
• Data Accuracy Issues: Managers find this data can be inaccurate or incomplete, complicating forecasting.

AI Solutions

Imagine an AI agents that can:

• Automate Data Capture (Unstructured to Structured): Listens to sales calls. Identifies the person on the other end. Extracts relevant details like company size, pain points, competitors, deal size.
• Eliminate Manual Entry: Removes the need for sales reps to input data manually.
• Enhance Data Quality: Provides more accurate and complete data for better forecasting.

This AI agent effectively replaces the human UI, seamlessly interacting with the system of record. Solutions like Skypoint AI Platform's AI agents (NorEntropy ) exemplify this approach by removing noise from data and utilizing AI agents for data processing.

Shifting Emphasis to Databases

This shift moves the focus away from the UI or front-end applications toward the databases themselves. In an AI-first future, the system's ability to autonomously:

• Gather Data: Collect information from various unstructured sources.
• Store Data: Efficiently manage large volumes of data.
• Process Data: Convert unstructured data into structured formats that BI, AI Copilot and traditional applications can interface with.

AI will not only capture data but also create and manage workflows without human intervention.

Transforming Enterprise Applications

We may witness a fundamental transformation in how enterprise applications are built:

• From: Traditional front-end applications tied to databases and manual workflows.
• To: AI-native applications built on AI-native databases, with the database taking center stage.

These AI applications will:

• Operate on centralized data repositories (e.g., Data Lakehouse / Delta Lake).
• Use AI agents to gather and process information from diverse unstructured sources.
• Automate workflows, reducing or eliminating the need for human data movement between systems.

Implications for Traditional Moats

In this new AI-driven landscape, traditional moats built by systems of record could erode:

• Diminished Reliance on UIs: Human-driven interfaces and manual processes will become less critical.
• Emergence of Data Apps: Value will shift to how efficiently AI can gather and act on data.
• Competitive Advantage: Companies embracing AI-powered systems can build more flexible, scalable solutions less reliant on manual workflows.
• Fundamental Change: This represents a fundamental change in how enterprise software operates.

The Need for New Infrastructure

To support AI-native applications, we'll need an entirely new set of tools and infrastructure, such as Skypoint AIP NorEntropy AI Agents and Pipelines. Key considerations include:

• Error Handling:What happens if a long-running workflow times out or encounters a processing error?Should the process restart from the beginning or resume from the last successful step?
• Workflow Monitoring: How do you monitor multiple workflows running in parallel?
• Output Evaluation: How do you assess the quality and accuracy of AI-generated outputs?

In essence, we'll see an explosion of new infrastructure requirements to support AI-native apps.

Conclusion

The world needs AI databases to power AI-native applications. As we shift from systems of record to systems of intelligence, companies must adapt to an AI-first future where data processing and workflow management are autonomously handled by AI agents. This transformation will redefine enterprise software, emphasizing the importance of efficient, flexible databases over traditional human-centered interfaces.

Credit: Jamin Ball (Clouded Judgement 10.18.24 - From Systems of Record to Systems of Intelligence)