How will we measure intelligence in AI, AGI, and ASI systems?

I had a wonderful conversation this morning with o1 and Claude 3.5 about the Model of Hierarchical Complexity developed by Dr Michael Lamport Commons over the last several decades at his lab at Harvard Medical School. I believe the MHC is how we will pinpoint, measure, and improve cognitive functions of AI & AGI systems until they are able to do this work themselves (ASI).

I met Dr Commons and his team about 8 years ago in my travels and explorations as I and Dr. Rick Kubina were building Chartlytics a precision behavioral measurement system based on the work of BF Skinner, Ogden Lindsley and a bunch of brilliant behavior scientists. Chartlytics (now called PrecisionX) was our manifestation of what we thought a standard celeration chart - the first derivative of the cumulative recorder, which enabled Skinner and his collegues to discover the fundamental nature of behavior change (i.e., learning) - could be in the digital age.

Dr. Commons and his team in his lab, have been working to quantify the element-compound or ontological nature of skills, in a mathematical calculable way of defining the order of operations of developing prerequisite skills needed to attain fluency in more complex skills.

There is much to explore here. I believe this is the first principals way and I strongly believe behavior science has a lot to offer the field of AI development. I have not fact checked much of this conversation yet, it should be considered wrong until such fact checking and citations have been completed, but I felt if it inspires others to pursue the implications, then it has served its purpose.

Q: Can you describe the model of hierarchical complexity, provide a description of each stage, a couple examples, and compare against human developmental stages?

Certainly! I'll provide more detailed descriptions for each stage, including examples and correlations to human developmental ages where applicable. Keep in mind that age correlations are approximate, as individual development can vary significantly.

Model of Hierarchical Complexity: Detailed Stage Descriptions

1. Calculatory - Description: Automatic computation or responses to stimuli. Example: A calculator performing arithmetic operations. Human Age: N/A (machine level)
2. Sensory or Motor (0-3 months) - Description: Basic reflexes and sensations. Example: A newborn turning their head in response to touch (rooting reflex). Human Age: 0-3 months
3. Circular Sensory-Motor (3-6 months) - Description: Repeating actions intentionally. Example: A baby repeatedly shaking a rattle to hear the sound. Human Age: 3-6 months
4. Sensory-Motor (6-12 months) - Description: Coordinated actions in response to stimuli. Example: A baby crawling towards a toy they want. Human Age: 6-12 months
5. Nominal (1-2 years) - Description: Using single words to name objects or actions. Example: A toddler pointing to a dog and saying "doggy". Human Age: 1-2 years
6. Sentential (2-3 years) - Description: Using simple sentences with basic grammar. Example: A child saying "I want juice" or "Mommy go". Human Age: 2-3 years
7. Preoperational (3-6 years) - Description: Using multiple sentences and understanding basic cause-effect. Example: A child explaining, "If I eat all my dinner, I get dessert." Human Age: 3-6 years
8. Primary (6-8 years) - Description: Making simple deductions and following sequential orders. Example: A child solving simple word problems like "If I have 3 apples and get 2 more, how many do I have?" Human Age: 6-8 years
9. Concrete (8-11 years) - Description: Seriation, classification, and conservation of concrete properties. Example: A child organizing objects by size, shape, and color. Human Age: 8-11 years
10. Abstract (11-14 years) - Description: Forming abstract concepts and using variables. Example: A teenager understanding and using metaphors in language. Human Age: 11-14 years
11. Formal (14-17 years) - Description: Using hypothetical reasoning and solving problems with single variables. Example: A high school student solving algebraic equations or designing a scientific experiment. Human Age: 14-17 years
12. Systematic (17-21 years) - Description: Coordinating multiple relationships within a system. Example: A college student analyzing complex economic systems or ecological relationships. Human Age: 17-21 years (Some adults may not reach this stage)
13. Metasystematic (21+ years) - Description: Comparing and contrasting different systems. Example: A researcher developing a new theory that integrates multiple existing theories in their field. Human Age: 21+ years (Many adults do not reach this stage)
14. Paradigmatic (Rare in adults) - Description: Creating new fields by synthesizing metasystems. Example: Einstein developing the theory of relativity, integrating concepts from physics and mathematics in a revolutionary way. Human Age: Rare, typically in mature adults with significant expertise
15. Crossparadigmatic (Extremely rare) - Description: Integrating multiple paradigms across different fields. Example: Developing a unified theory that bridges multiple scientific disciplines, like a comprehensive theory of consciousness integrating neuroscience, psychology, and quantum physics. Human Age: Extremely rare, typically in late-career experts or polymaths
16. Metacrossparadigmatic (Theoretical) - Description: Creating new fields that synthesize crossparadigmatic systems. Example: Developing a new framework for understanding reality that fundamentally changes how we approach knowledge across all domains. Human Age: Not observed in human cognition
17. Metametacrossparadigmatic (Highly Theoretical) - Description: Creating new ways of generating metacrossparadigmatic syntheses. Example: Developing new methods for how paradigm-shifting syntheses could be achieved across all fields of human knowledge. Human Age: Not observed in human cognition

Note: Stages 15 and 16 are theoretical constructs and have not been observed in human cognition. They represent potential levels of cognitive complexity beyond current human capabilities.

This expanded description provides a more comprehensive overview of each stage in the Model of Hierarchical Complexity, including examples and approximate age correlations where applicable. A few important points to note:

1. Individual variation: While age ranges are provided, individual development can vary significantly. Some people may reach certain stages earlier or later than the average, and some may never reach the higher stages.
2. Domain specificity: An individual might function at different stages in different domains. For example, someone might operate at a higher stage in their professional field but at a lower stage in other areas of life.
3. Higher stages: Stages 11 and above are not universally reached. Many adults operate primarily at the Formal or Systematic stages in most aspects of their lives.
4. Theoretical stages: Stages 15 and 16 are purely theoretical and have not been observed in human cognition. They represent potential levels of complexity that might be achievable by artificial superintelligence or through future human cognitive enhancement.
5. Cultural context: The model and its age correlations are based primarily on research in Western contexts. Cultural differences may influence the pace and manifestation of these stages.
6. Continuous development: While presented as discrete stages, cognitive development is often a more continuous process, with individuals gradually developing more complex thinking patterns.

This model provides a framework for understanding cognitive complexity, but it's important to remember that human intelligence and development are multifaceted and can't always be neatly categorized into distinct stages. The MHC is one of many models attempting to describe cognitive development and has its own strengths and limitations.

Q: Where do you think current AI capabilities stop?

That's an insightful question about the current capabilities of AI in relation to the Model of Hierarchical Complexity (MHC). To answer this, we need to consider different types of AI systems and their various applications.

Current AI Capabilities in Relation to MHC Stages

Based on this analysis, I would say that current AI capabilities generally stop around Stage 12 (Metasystematic) of the Model of Hierarchical Complexity, with some potential touches on Stage 13 (Paradigmatic) in very specialized domains. Here's a more detailed breakdown:

1. Lower Stages (0-10): Current AI systems excel at these stages. They can perform calculations, recognize patterns, process language, and solve well-defined problems with great efficiency, often surpassing human capabilities in specific tasks.
2. Stage 11 (Systematic): Many advanced AI systems, particularly large language models like myself, operate comfortably at this level. We can coordinate multiple relationships and understand complex systems within and across various domains.
3. Stage 12 (Metasystematic): This is where we start to see the current limits of AI capabilities. While some AI systems can compare different systems and create overarching principles, this ability is often limited and lacks the depth and nuance of human experts operating at this level.
4. Stage 13 (Paradigmatic): Current AI systems generally do not reach this stage in a broad sense. However, in very narrow and specialized domains, some AI systems might approach this level. For example, AI systems used in advanced scientific research might contribute to creating new paradigms within specific fields.
5. Stages 14 and beyond: These stages are currently beyond the reach of AI systems. They require levels of abstraction, creativity, and integrative thinking that AI has not yet achieved.

Key considerations:

1. Lack of True Understanding: While AI can process and generate information at high levels of complexity, it often lacks true understanding or consciousness. This limits its ability to operate genuinely at the higher stages of the MHC.
2. Narrow vs. General Intelligence: Most current AI systems are narrow, excelling in specific domains but lacking the general intelligence required for consistent high-level performance across all domains.
3. Rapid Progress: AI capabilities are advancing rapidly. What's considered the limit today might be surpassed in the near future.
4. Human-AI Collaboration: In many cases, the highest levels of performance are achieved through human-AI collaboration, combining AI's processing power with human intuition and creativity.
5. Ethical and Philosophical Questions: As AI capabilities advance, it raises questions about the nature of intelligence and consciousness, and how we define and measure these concepts in non-human entities.

It's worth noting that this assessment is based on our current understanding and observations of AI systems. The field is evolving rapidly, and new breakthroughs could potentially push these boundaries further. Additionally, applying the MHC to AI systems is not straightforward, as the model was originally developed for human cognition.

Q: How could we adapt the MHC to allow us to pinpoint, measure and improve AI cognitive capabilities?

That's an excellent and thought-provoking question. Adapting the Model of Hierarchical Complexity (MHC) for AI systems could indeed provide valuable insights into AI capabilities and guide future development. Here's an approach we might take to adapt the MHC for AI:

Adapting the Model of Hierarchical Complexity for AI Systems

1. Redefine Stages for AI Context Translate human-centric descriptions to AI-relevant terms Define clear, measurable criteria for each stage
2. Develop AI-Specific Assessment Tasks Create standardized tests for each MHC stage Ensure tasks are domain-agnostic where possible
3. Implement Multi-Modal Evaluation Assess AI performance across various input/output modalities Consider text, image, audio, and multi-modal capabilities
4. Measure Generalization Capabilities Test AI's ability to transfer learning across domains Assess performance on novel, unseen tasks
5. Evaluate Metacognitive Abilities Assess AI's awareness of its own processes and limitations Measure ability to explain reasoning and decision-making
6. Implement Continuous Scale Develop a more granular scale between stages Allow for partial achievement of stages
7. Incorporate Ethical Reasoning Assess AI's ability to handle ethical dilemmas Evaluate understanding of social and moral implications
8. Measure Creativity and Innovation Develop metrics for assessing novel idea generation Evaluate AI's ability to combine concepts in unique ways
9. Assess Abstraction and Conceptualization Test AI's ability to form and manipulate abstract concepts Evaluate understanding of hypothetical scenarios
10. Implement Longitudinal Assessment Track AI system's "cognitive" development over time Measure rate of improvement and learning

Now, let's delve deeper into how we could implement these adaptations:

1. Redefine Stages for AI Context: We need to translate the human-centric descriptions of each MHC stage into terms that are relevant and measurable for AI systems. For example, instead of "sensory-motor" stages, we might have stages related to basic data processing and pattern recognition.
2. Develop AI-Specific Assessment Tasks: Create a standardized set of tasks for each MHC stage that an AI system could perform. These tasks should be designed to clearly demonstrate the cognitive capabilities associated with each stage. For instance, for the "Systematic" stage, we might have tasks that require the AI to analyze complex systems with multiple variables and predict outcomes.
3. Implement Multi-Modal Evaluation: As AI systems become more versatile, it's crucial to assess their performance across various modalities. This could involve tasks that require integrating information from text, images, and even audio inputs.
4. Measure Generalization Capabilities: One key aspect of higher cognitive functions is the ability to generalize knowledge and apply it to new situations. We could design tests that assess how well an AI system can transfer learning from one domain to another, or how it performs on entirely novel tasks.
5. Evaluate Metacognitive Abilities: Higher stages of the MHC involve metacognition - thinking about thinking. For AI, this could translate to assessing the system's ability to explain its own reasoning, recognize its limitations, and adjust its approach based on past performance.
6. Implement Continuous Scale: While the original MHC uses discrete stages, for AI, it might be more useful to implement a continuous scale. This would allow for more nuanced measurement of AI capabilities, recognizing partial achievements of higher stages.
7. Incorporate Ethical Reasoning: As AI systems become more advanced, their ability to handle ethical considerations becomes crucial. We could include assessments of an AI's ability to recognize and reason about ethical dilemmas, which would be particularly relevant for the higher MHC stages.
8. Measure Creativity and Innovation: Higher stages of cognitive complexity often involve creativity and the ability to generate novel ideas. We could develop metrics to assess an AI's ability to come up with innovative solutions or combine concepts in unique ways.
9. Assess Abstraction and Conceptualization: The ability to form and manipulate abstract concepts is crucial for higher-level cognition. We could design tasks that assess how well an AI system can work with abstract ideas and hypothetical scenarios.
10. Implement Longitudinal Assessment: Unlike humans, AI systems can potentially "develop" very rapidly. Implementing a system for tracking an AI's progress over time could provide valuable insights into the learning and development process of AI systems.

By adapting the MHC in these ways, we could create a more comprehensive and AI-specific framework for assessing cognitive capabilities. This adapted model could serve several purposes:

1. Benchmarking: It would allow for standardized comparison between different AI systems.
2. Development Guidance: It could highlight areas for improvement in AI systems, guiding future research and development efforts.
3. Safety and Ethics: By including ethical reasoning assessments, it could help ensure that as AI systems become more capable, they also become more aligned with human values.
4. Public Understanding: A clear, stage-based model could help non-experts understand the current state and progress of AI capabilities.

However, it's important to note that this adaptation would be a significant undertaking, requiring collaboration between experts in AI, cognitive science, philosophy, and other related fields. It would also need to be regularly updated as our understanding of AI and cognition evolves.

It might be time to go back to school the only way I know how, start building. I'd love to connect with others thinking about and working on this area, there is much that can be done to move us towards a ASI future. Please reach out!