Beyond Intuition: AI Systems Demand a Learning Organization
Charles Phiri, PhD, CITP
Executive Director | SME AI/ML Innovation at JPMorganChase | Gartner Peer Community Ambassador
The unique challenges posed by AI systems are fundamentally reshaping enterprise architecture thinking. These challenges stem from the unprecedented state spaces that AI systems operate within, transcending traditional system design paradigms.
Given the unique challenges of integrating emerging AI capabilities, the question that architects must grapple with is: how do we design and govern AI systems that evolve beyond their initial specifications, necessitating a new approach?
In architecting AI Systems, enterprise architects now confront systems whose behaviors emerge from complex interactions that defy prediction at design time. While effective for deterministic systems, conventional architectural patterns fail when architects apply them to AI-driven enterprises.
We are now in an era where the ability to assess and understand AI system behaviors is as crucial as our design capabilities. This shift mandates a fundamental evolution in our enterprise architecture approaches, signaling the need for significant changes.
Enterprise AI architecture reaches beyond technical implementation into organizational design.
Enterprise AI architecture enables the deployment of AI capabilities and supports continuous assessment and evolution of these systems. This evolution requires new architectural frameworks to manage emergent behaviors, maintain safety at scale, and facilitate systematic organizational learning.
Traditional enterprise architects design static systems with well-defined boundaries and behaviors.
Modern AI systems, however, behave like living organisms within enterprise ecosystems. They learn, adapt, and evolve through interactions, compelling architects to develop approaches supporting continuous assessment and adaptation. This necessity for continuous learning and adaptation stresses the importance of staying updated and flexible in AI system design.
This emerging reality forces us to reconstruct fundamental architectural principles and governance models.
We draw parallels from Kennedy’s Moon mission to modern enterprise AI challenges to demonstrate why conventional architectural approaches fail and why architects must revolutionize their approach to AI system design and governance. The stakes demand more than incremental architectural adaptation. Assessment maturity has become crucial for system safety, and enterprise architectures must transform to meet these challenges.
Success requires more than technical expertise or access to the latest AI models.
Architects must develop robust architectural frameworks that enable continuous learning and adaptation. This article charts the evolution enterprise architecture must undergo to support organizational learning and details how we can initiate this critical transformation.
Vision Despite Uncertainty
On September 12, 1962, President John F. Kennedy stood before ~45,000 people at Rice University stadium in Houston and declared:
“We choose to go to the Moon in this decade and do the other things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills.”
Kennedy declared this when the Soviet Union led in space achievements, launching the first satellite and putting the first human in orbit.
The Apollo program demanded unprecedented investment. According to the archives, funding totaled $28.7 billion in 1960s’ dollars—equivalent to approximately $280 billion when adjusted for inflation to 2024.
By fiscal year 1966, NASA’s budget reached its peak at 4.41% of federal spending, marking one of the largest peacetime mobilizations of public resources in American history. This massive investment culminated in Apollo 11’s successful Moon landing on July 20, 1969, fulfilling Kennedy’s pledge.
Kennedy’s statement captures the essence of correctly framing ambitious enterprise architecture goals in the face of incomplete information. It demonstrates how to rely on clear objectives to focus innovation, experimentation, and organizational energy.
Technical Realism
NASA did not have the luxury to fake the scientific rigor involved in addressing the challenges of the Moon mission! They had to deliver tangible results.
Kennedy acknowledged that achieving the Moon landing would require solutions to technical problems for which no immediate answers existed.
The mission demanded metal alloys that had not yet been invented, systems capable of withstanding unprecedented stresses, and precision engineering beyond known capabilities.
This level of executive transparency parallels the clarity and precision expected of enterprise AI architects.
Know Thy Data!
Clarity and precision are essential to understanding and solving complex problems; duplicity and ambiguity undermine both.
We must acknowledge the gaps in current AI methodologies while advancing systems capable of handling tomorrow’s complexities.
Scientists and engineers have developed robust methods of coping with system uncertainties.
The Apollo program established rigorous assessment frameworks before solutions existed. They developed testing criteria for materials not yet invented and evaluation systems for unprecedented engineering challenges.
Strategic Framework
Kennedy’s speech frames an architectural approach that must resonate powerfully today:
These fundamental elements remain vital in managing AI systems that evolve and scale under complex, uncertain conditions. This strategic framework is crucial in managing the complexities of AI systems and provides a structured approach to their design and governance.
Deterministic Safety vs. AI Safety - The Fundamental Divide
Traditional Systems Engineering
No bridge becomes “eventually safe.” System engineering achieves safety from inception by restricting the system’s degrees of freedom.
When bridges fail catastrophically—like the Baltimore Francis Scott Key Bridge in recent history—they fail outside defined operating parameters. The collision did not make the bridge retrospectively unsafe - while unexpected (within reason), the failure occurred within understood physical bounds.
Architects and engineers systematically reduce possibilities until every behavior becomes knowable.
Every interface, every boundary, and every interaction must resolve to a finite set of known states. The architecture succeeds within defined parameters or fails within mapped failure modes.
This systematic restriction of degrees of freedom enables exhaustive mapping of behaviors within system boundaries. Traditional enterprise architecture operates entirely within these knowable states.
AI Systems: Emergent Degrees of Freedom
AI systems fundamentally challenge this approach. Each capability added to an AI system inherently increases the degrees of freedom through complex interactions.
Most combinations of behaviors in AI systems extend beyond known states. Where traditional systems constrain possibilities, AI systems generate new degrees of freedom during operation.
Enterprise AI architects must prepare to manage systems where small, incremental parameter changes spawn behavioral spaces that exceed design-time understanding.
The degrees of freedom emerge through operation, expanding beyond initial constraints.
Beyond the Known States
The Psychology and Epistemology of AI Safety
The Nature of Our Angst
Traditional system safety emerges through restriction—deliberately constraining degrees of freedom until every potential outcome is mapped. This restriction creates an illusion of comprehensive understanding through exhaustive specifications.
Enterprise architects could previously validate safety through complete testing of enumerated states. The boundaries of possible behaviors remained knowable.
AI systems destroy this certainty. They force architects to confront systems where unknown unknowns dominate the behavioral space.
The Desire to Be Right vs Actually Being Right
The challenge transcends traditional architectural practices. Unknown unknowns demand scientific rigor in system design and operation.
Enterprise architects must now incorporate empirical methods, systematic observation, and hypothesis testing into their core practice. Testing becomes an exercise in scientific discovery rather than requirements validation.
This epistemological shift demands new architectural competencies. We must apply scientific methods to:
The architectural practice must embrace uncertainty while bringing scientific precision to its management.
The Multiplication of Complexity
Beyond Simple Emergence
Modern AI systems introduce architectural challenges that exceed traditional complexity models. Each domain of interaction multiplies the degrees of freedom.
Multimodal models simultaneously process text, images, and audio, generating cross-modal interactions that create novel behavioral spaces.
When AI systems interact within complex ecosystems—models feeding outputs into each other—feedback loops amplify emergent behaviors and risks beyond prediction.
Compounding Complexity Through Agency
Multi-agent systems shatter even these expanded boundaries. Agents adapt and strategize in response to each other, producing social and strategic emergent properties absent from their isolated components.
Agents may compete for resources!
Expanding action spaces, where systems simultaneously generate text, control robotics, and manipulate APIs, create combinatorial explosions of possibilities that transcend traditional design assumptions.
The integration of quantum computing will further multiply these complexity classes by introducing computational modes that defy classical architectural bounds.
领英推荐
The Fallacy of Architectural Familiarity
The "Fake-It-Till-You-Make-It" Approach
Traditional software-intensive systems operate within bounded complexity, where established patterns and heuristics suffice.
Enterprise architects typically rely on intuition built from past successes. This approach fails when AI systems present evolving state spaces and emergent behaviors.
We must abandon the comfort of perceived expertise. AI systems demand a scientific approach, not intuition or incremental improvements.
State Space Paradox and Open-Ended Creativity
The state space paradox arises when we attempt to define all possible states a system might enter. Enumerating these states becomes part of the state space itself.
This self-referential nature creates an infinite regress. Each attempt to completely specify the system’s behavior becomes part of what we must specify.
This paradox manifests in AI systems when we define safety bounds. Our specification of safety constraints becomes another dimension of the system’s behavior, potentially creating new unsafe states.
Expanding or redefining these state spaces demands more than architectural intuition. It requires rigorous experimentation and evaluation frameworks that acknowledge this fundamental limitation.
Design Time Risks and Data Quality
Poor data breaks systems. Bad training data skews models. Inconsistent labeling creates unintended behaviors.
Enterprise architects must advocate for strict data governance. Every pipeline needs monitoring, and every dataset demands validation.
Rigorous auditing must catch anomalies and bias before models reach production.
Architectural Decisions and System-Level Coordination
Enterprise AI combines multiple methods into unified systems, including rules, Bayesian inference, and other Machine Learning techniques.
Each addition introduces new failure modes. Bad interfaces compound errors, and problems cascade through pipelines.
Architects must champion formal verification and clear protocols to prevent design flaws. Reviews must catch integration risks early.
Runtime Risks from Emergent Behaviors
Emergent capabilities arise when local interactions within AI systems produce unintended outcomes. Knowledge-based systems may synthesize contradictory rules into surprising conclusions. Neural Networks learn unexpected strategies.
Architects must advocate for:
Redundancy designs and adaptation protocols support continuity when subsystems behave erratically. This is a knowledge-based endeavor.
Performance Issues and Scalability Constraints
Complex models strain computational resources, and high-throughput data streams amplify these constraints. Container orchestration and cloud computing offer partial solutions.
Architects must champion:
System design requires continuous load testing at the enterprise scale. As architects, we must understand and validate how.
Architectural Validation and Control
Enterprise architecture must evolve beyond documentation and governance into empirical practice. Each architectural decision requires testable hypotheses and measurable outcomes.
Architects must advocate for:
Adaptability in Enterprise Settings
Rigorous architecture demands precise measurement of adaptation mechanisms. Each change in the system state must support falsifiable hypotheses.
Enterprise architects must champion:
The practice of architecture must meet the standards of scientific investigation. This must be a deliberate journey.
Human-AI Collaboration and Beneficial Outcomes
Enterprise AI augments human decision-making across critical domains. Medical diagnosis, financial modeling, and large-scale analytics require us to understand precisely where and how augmentation occurs.
The architectural challenge exceeds interface design. We must measure and validate each claim of augmented capability. Simple accuracy metrics fail to capture how humans and AI interact.
Scientific Assessment of Value
Measuring AI value demands experimental rigor. Enterprise AI architects must champion experimental design in deployment strategy, and statistical methods must validate each claim of improved performance.
We cannot trust correlation or anecdotal evidence. Control groups must validate business outcomes over time, and causal frameworks must prove how specific architectural decisions drive improvements.
Measuring value demands the same scientific rigor we apply to system behavior. The scientific method must verify claims of effectiveness. Architects must defend this rigor against the pressure to deploy rapidly.
The Learning Imperative
Enterprise AI architecture faces distinct challenges in integrating AI systems. Each architectural decision necessitates new methods, deeper understanding, and increased rigor, requiring deliberate and sustained effort. The complexity of AI systems renders traditional architectural paradigms inadequate.
Daniel Kahneman’s assertion, “The quality of our thinking cannot be higher than the quality of the information on which it is based,” highlights the core of this challenge. Achieving robust architectural decisions demands an understanding of systems and data at their most fundamental level. Without this, the quality of decisions is inevitably compromised.
Chip Huyen aptly observes, “ML algorithms don’t predict the future, but rather encode the past, thus perpetuating the biases in the data and more.” This underscores the necessity for architects to master data quality assessment, bias detection, and rigorous validation techniques—failure to do so risks embedding systemic flaws into AI-driven architectures.
Conscious effort parallels Kahneman’s System 2 thinking:
“In all these cases, you are asked to do something that does not come naturally, and you will find that the consistent maintenance of a set requires continuous exertion of at least some effort.”
Architectural decisions in AI systems demand this same deliberate, analytical engagement. Learning in AI architecture is neither passive nor automatic; it requires sustained intellectual rigor.
The evolving nature of AI systems compounds the challenge.
“True intuitive expertise is learned from prolonged experience with good feedback on mistakes.”
In AI, however, expertise remains temporary. The rapid pace of technological advancement and the escalating complexity of systems ensure that no architect maintains comprehensive mastery.
Success in this domain hinges on proficiency in:
A cessation of critical inquiry and learning transforms valid principles into unexamined dogma. Success depends on sustaining deliberate, systematic investigation of every architectural assumption and decision.
Conclusion
We need to improve our AI architecture practice by incorporating scientific principles.
The evolution of enterprise architecture into a scientific discipline is no longer optional; it is an imperative dictated by the unprecedented complexity of modern AI systems.
Traditional architectural patterns, rooted in assumptions suited for deterministic systems, are insufficient to address the multidimensional challenges posed by AI’s dynamic and expansive capabilities.
Our analysis highlights the necessity of transcending intuition-driven approaches. AI systems introduce a vast array of degrees of freedom, invalidating legacy methodologies and demanding a rigorous, evidence-based approach to design and decision-making.
Success in this domain requires a fundamental shift in principles and practices:
? Empirical Validation: Architectural decisions must be grounded in measurable evidence, ensuring that choices are not merely speculative but demonstrably effective.
? Outcome Quantification: System performance and behavior must be continuously evaluated against objective, quantifiable benchmarks.
? Statistical Rigor: Improvements must be substantiated through robust statistical analysis, minimizing bias and ensuring repeatability.
AI systems are characterized by emergent behavior and non-linear dynamics, making reliance on past experiences inadequate. Effective architecture must be adaptive, incorporating iterative cycles of hypothesis generation, empirical testing, and refinement.
The capacity to succeed or fail in AI architecture is directly proportional to the commitment to rigorous learning. Learning entails questioning assumptions, systematically measuring performance, and deeply understanding the causal relationships within systems. Without this foundation, architectural efforts risk stagnation or irrelevance.
We must combine innovation with precision, leveraging scientific rigor to navigate uncertainty and unlock the full potential of AI systems.
Executive Director @ JPMorgan Chase | MBA, AI/ML, Kubestronaut, Cloud Architecture, DevOps
2 个月Charles, this was a great read! Thank you for sharing it.
Technology and Talent Enabler | Passionate about Positive Change & Impacts | Trusted Partner & President's Club Achiever | Ex-Solarwinds Ex-Oracle Ex-Gartner Ex-Xerox
2 个月Adapt, change, evolve, learn, strategize, protect. And getting to the moon. Thank you for sharing your knowledge and these very detailed insights, Charles, & Happy New Year!