Assessing Gaps in MITRE ATLAS (Oct 2024)

Missing MITRE ATLAS Attack Techniques for Modern Symbolic AI

The October 2024 update to MITRE ATLAS , the Adversarial Threat Landscape for Artificial-Intelligence Systems, offers new insights into evolving threats facing AI-enabled systems. However, upon a detailed review, significant gaps remain—particularly concerning attacks against symbolic AI systems, such as those using OWL, RDF, or hybrid AI paradigms. These systems, foundational to many advanced knowledge-based applications, appear to be underrepresented in ATLAS' current catalog of adversarial tactics and techniques. By highlighting these gaps, this review aims to underscore the need for enhanced coverage of symbolic AI vulnerabilities, addressing complex threats like ontology hijacking, rule manipulation, and inference engine attacks, which are crucial for securing modern hybrid AI solutions. Symbolic AI is also commonly known as Knowledge Representation and Reasoning, Knowledge Engineering, Ontology Engineering, Rules-Based AI, and Machine Understanding systems.

When discussing common techniques for attacking modern symbolic AI systems—such as those based on OWL, RDF, Inference Engines, or RML—there are unique considerations that differ from the attacks on machine learning systems. Here are some common attack techniques that are relevant to symbolic AI systems but are missing from the current ATLAS landscape:

1. Ontology Hijacking and Injection Attacks

• Ontology Hijacking: Attackers may introduce malicious ontologies into a knowledge graph or manipulate existing ontologies. By injecting incorrect or compromised ontological definitions, attackers can force symbolic reasoning systems to reach erroneous or misleading conclusions. This form of attack is similar to SQL injection in databases but specifically targets the integrity of the semantic model.
• Ontology Injection: Attackers may add invalid or misleading triples into an RDF graph to cause incorrect inference. This is particularly problematic in reasoning processes that rely on these relationships to make critical decisions.

2. Inconsistency and Contradiction Attacks

• Symbolic AI systems heavily rely on consistency within ontologies for inference and reasoning. Attackers could introduce contradictory or inconsistent data to create logical conflicts, effectively sabotaging the reasoning process. These inconsistencies could lead to system crashes, false decisions, or the inability to make decisions. Such attacks could be particularly harmful if they target automated decision-making in critical systems, leading to confusion or paralysis of system functions.

3. Ontology Poisoning

• Knowledge Graph Poisoning: This involves maliciously altering the knowledge within a knowledge graph. By adding or removing RDF triples, attackers can mislead the inference engine into drawing incorrect or misleading conclusions. The poisoning can affect downstream applications that rely on the knowledge graph for tasks like automated reasoning or recommendation.

4. Rule Injection and Modification

• In symbolic AI systems that use rule-based engines (such as SWRL rules in OWL), attackers may add or modify rules in a way that subverts the intended function of the system. This type of attack could result in false inferences, incorrect alerts, or recommendations. Since reasoning engines use these rules to derive new knowledge, the attacker could cause the AI to exhibit unintended or malicious behavior.

5. Semantic Misalignment Attacks

• Symbolic AI systems often integrate different ontologies and knowledge graphs. Attackers could exploit differences in ontology definitions or mismatches between different schema standards to manipulate the data interpretation. This semantic misalignment can cause knowledge graphs to integrate incorrectly, leading to faulty inferences. This kind of vulnerability is especially relevant in systems attempting to integrate data from diverse domains without proper reconciliation mechanisms.

6. Knowledge Graph Link Prediction Attacks

• Link Manipulation: Attackers can exploit link prediction algorithms used for adding inferred relationships to a knowledge graph. By subtly altering graph data, attackers can lead the link prediction algorithm to generate connections that should not exist. This approach can indirectly create vulnerabilities in inference chains or in automated decision-making tasks that rely on relationships derived from manipulated graphs.

7. Denial of Service (DoS) through Reasoning Load

• Symbolic reasoning often has a high computational cost, especially with large ontologies. Attackers could exploit this by introducing overly complex structures or triggering computationally intensive reasoning processes, leading to a denial of service. This "reasoning overload" attack forces the system to spend excessive time processing certain logic, thereby degrading performance or making it unavailable.

8. Adversarial Schema Evolution

• In systems where ontologies evolve, attackers can force malicious schema evolution. By modifying or replacing classes and relationships in evolving RDF/OWL ontologies, attackers can cause downstream applications to interpret the updated ontology in unintended ways, which leads to incorrect data integration and misclassification of data entities.

9. Knowledge Graph Integrity Attacks

• Unauthorized Triple Updates: Altering the values within existing RDF triples to incorrect or harmful data can lead to wrong inferences. Unlike data corruption in traditional databases, such changes can propagate incorrect knowledge to multiple entities, amplifying the effect.
• Inferred Threat Propagation: Once the ontology has been compromised, attackers can manipulate how inferences are derived, allowing the incorrect information to spread across related entities. This threat propagation can be highly damaging, as it affects all systems that consume or depend on the compromised knowledge.

10. Ontology Merging Attacks

• Semantic Misleading: When integrating ontologies from different sources, attackers could exploit semantic ambiguities to inject misleading mappings. This results in merged ontologies that are vulnerable to incorrect inferences. Attackers could exploit this especially in cases of federated knowledge graphs or in distributed semantic systems.

11. Inference Engine Attacks

Inference engines are core components of symbolic AI systems, responsible for deriving new information from existing knowledge by reasoning over ontologies and rules. They are thus critical targets for adversaries looking to sabotage the decision-making capabilities of AI systems. Some common attack techniques include:

• Inference Process Manipulation: Attackers may manipulate rules or knowledge to hijack the reasoning process, forcing the inference engine to take altered reasoning paths that result in incorrect conclusions. For instance, if critical rules or logical axioms are compromised, the system may output misleading or harmful decisions.
• Inference Loop Creation: By introducing circular rules, attackers can create loops in the inference logic that effectively trap the engine in endless processing. This "inference loop" could result in denial of service, consuming valuable resources without producing any valid outputs, particularly in time-sensitive applications.
• Rule Complexity Bombs: Attackers may add overly complex nested rules to the knowledge base to exploit the high computational cost of symbolic reasoning. These attacks, often targeting the complexity of OWL-based systems, can lead to reasoning overload and eventual degradation of system performance, potentially making the AI engine unresponsive.
• Faulty Rule Deletion: Specific rule deletions or modifications may impair an inference engine's ability to perform consistency checks, leading to incomplete reasoning that results in suboptimal or incorrect system behaviors.

These techniques highlight potential weaknesses in modern symbolic AI systems that differ significantly from traditional machine learning vulnerabilities. Unlike adversarial attacks against neural networks—such as those focused on adding noise or manipulating training data—attacks against symbolic AI target the structure and semantics of the knowledge representation itself.

These are areas not currently addressed in the MITRE ATLAS framework, which largely focuses on vulnerabilities typical in machine learning models and neglects the specific challenges associated with knowledge-based and symbolic AI systems. Addressing these concerns would require unique sets of defense mechanisms, such as ontology validation, rule consistency checks, graph integrity monitoring, and consistency verification techniques, which are essential to secure symbolic AI effectively.

Exploring the gap in MITRE ATLAS

The gap in MITRE ATLAS related to ontologies, symbolic AI, and the integration of these technologies with other AI systems is significant, especially when considering the increasing convergence of different AI paradigms. Let’s examine why this gap is important and what it means in the context of modern hybrid AI systems, where symbolic AI is paired with technologies like graph analytics, machine learning, deep learning, and neuro-symbolic AI.

1. Symbolic and Hybrid AI in Broader AI Contexts

• Complex Integration Scenarios: Modern AI systems often use symbolic AI in conjunction with other types of AI, such as graph machine learning and generative models, to gain benefits that neither symbolic reasoning nor ML alone can offer. Ontologies can serve as structured representations of domain knowledge, allowing reasoning processes to support deep learning models with enhanced semantic context. In neuro-symbolic AI, symbolic reasoning is paired with neural networks to achieve more explainable and contextually aware systems.
• The current ATLAS framework predominantly addresses adversarial threats specific to machine learning models, like adversarial perturbations, poisoning attacks, and model inversion. This leaves a considerable gap for hybrid systems, where attacks on symbolic components can undermine the AI’s performance and cause failures at the system level. Attacks against ontologies, such as ontology hijacking or semantic misalignment, could have cascading impacts on the overall hybrid AI system, affecting decision-making, recommendation engines, and more.

2. Graph Analytics and Knowledge Representation

• Graph Analytics & Graph ML: With increasing adoption of graph analytics and graph machine learning in fields like cybersecurity, financial fraud detection, and biological research, ontologies and symbolic knowledge representation play a critical role in defining relationships and enhancing graph-based inferences. Graph machine learning models often rely on underlying semantic knowledge represented through RDF or ontologies to define relationships in graphs.
• MITRE ATLAS currently does not fully account for adversarial risks that emerge when graph ML is used with symbolic reasoning. Graph poisoning attacks, where relationships between nodes are manipulated, are one form of attack. When combined with symbolic AI, these manipulated graphs can lead to flawed ontological inferences, disrupting the whole chain of reasoning in an AI-driven system.

3. Neuro-symbolic AI

• Adversarial Risks in Neuro-symbolic Systems: Neuro-symbolic AI aims to combine the deep learning capabilities of neural networks with the rigorous, human-readable representations of symbolic AI. Attacks against symbolic representations can directly affect the quality of the overall system's performance, leading to incorrect conclusions that, unlike pure ML models, are not easily detectable by metrics like accuracy or precision scores.
• Attacks targeting symbolic reasoning in neuro-symbolic systems could, for example, inject malicious rules that misguide neural models' operations, affecting applications such as autonomous systems, medical diagnostics, and other safety-critical use cases. Since MITRE ATLAS primarily considers vulnerabilities in deep learning models, these hybrid threats remain underrepresented.

4. Integration with Generative AI and AI Agents

• Generative AI Agents: When generative AI agents, such as language models, are integrated with symbolic AI, they can provide contextual responses enhanced by ontologies. For instance, using OWL ontologies can help a language model understand domain-specific knowledge more thoroughly.
• An adversary might manipulate the symbolic AI side, such as introducing ontology poisoning, leading the generative agent to deliver systematically misleading information. This can pose a risk in fields where factual correctness is crucial, such as healthcare or finance. ATLAS presently lacks the techniques for understanding these risks, focusing instead on conventional attacks like adversarial text perturbations for natural language models.

5. Potential Systemic Impact

• Increased Attack Surface: By incorporating ontologies and symbolic reasoning, AI systems expand the attack surface beyond the typical ML vulnerabilities, adding risks related to data quality, ontological consistency, rule integrity, and logical coherence. These risks, if exploited, can have consequences across the board—affecting the robustness, reliability, and explainability of AI.
• Sophisticated Adversaries: Real-world adversaries may exploit this integration, targeting the symbolic component to maximize impact. Since ontologies are often at the core of understanding relationships and domain-specific logic, their manipulation can directly affect any dependent AI systems—such as generative agents that use these ontologies to drive conversations, diagnostic tools, or automated decision systems.

What This Gap Represents

This gap in MITRE ATLAS represents a serious limitation, particularly given the direction AI systems are heading. Hybrid AI approaches—whether through graph-based reasoning, generative models, or neuro-symbolic integrations—are increasingly being deployed across industries. This confluence of different AI types creates vulnerabilities that exist at the intersection of traditional symbolic reasoning and machine learning/deep learning.

• Attack Complexity: Unlike traditional attacks on machine learning models that exploit statistical properties, attacks on symbolic AI often require manipulation of logical structures and inference mechanisms. The absence of comprehensive techniques to address these attacks in MITRE ATLAS leaves an entire class of AI systems vulnerable to sophisticated attacks that blend different types of exploit strategies.
• Lack of Defense Recommendations: MITRE ATLAS also lacks corresponding mitigation techniques that are specific to symbolic AI and hybrid approaches. Effective mitigation for attacks on symbolic AI might involve measures like semantic integrity checking, inconsistency detection, schema validation, and rule-based auditing, none of which are extensively covered in the current framework.

Recommendations to Address the Gap

To close the gap, MITRE ATLAS should consider expanding its threat taxonomy to include:

1. Specific Threat Vectors that target ontological structures, such as ontology poisoning, inconsistency attacks, and schema manipulation.
2. Hybrid Threat Models that cover neuro-symbolic AI, addressing both the symbolic and neural network components simultaneously.
3. Mitigation Strategies that focus on semantic validation, logical consistency checks, and attack detection in rule-based systems, which are crucial for defending symbolic AI components.

The expanding integration of symbolic AI with other AI types necessitates an update to frameworks like ATLAS to properly address vulnerabilities that could significantly undermine AI assurance and safety across a variety of critical domains. This would allow organizations using complex AI systems to better understand and defend against the full spectrum of adversarial techniques targeting their systems.

Mitigating Symbolic Attack Techniques

1. Ontology Hijacking and Injection Attacks

• Access Control and Authentication: Restrict access to ontology editing using robust role-based access control (RBAC). Only authorized users or systems should be able to make changes to ontologies.
• Ontological Integrity Checking: Use digital signatures and integrity verification to ensure the authenticity of ontological definitions. Any modifications must be authenticated and validated.
• Change Tracking: Employ systems to track changes to ontologies, enabling the detection of unauthorized alterations. Version control tools can help manage different ontology versions and detect unexpected updates.

2. Inconsistency and Contradiction Attacks

• Consistency Validation Tools: Use ontology validation tools, such as Protégé with plugins like Hermit or Pellet, to perform logical consistency checks regularly.
• Triple Validation Procedures: Implement semantic validation to ensure that newly added RDF triples do not introduce inconsistencies or contradictions into the ontology.

3. Ontology Poisoning

• Graph Integrity Monitoring: Apply integrity monitoring tools that detect unauthorized changes in the knowledge graph by comparing current states to trusted baselines.
• Anomaly Detection: Employ graph-based anomaly detection to identify suspicious modifications. These methods can flag newly added triples that do not fit existing patterns or relationships.

4. Rule Injection and Modification

• Rule Verification and Testing: Conduct unit testing for rules added to the system, ensuring they work as expected without malicious or unintended outcomes.
• Immutable Rule Sets: Make critical rules immutable after they are thoroughly vetted. This prevents unauthorized modifications that can affect system reasoning.
• Change Notification Mechanism: Notify administrators or system auditors when changes are made to rule sets so that they can verify authenticity.

5. Semantic Misalignment Attacks

• Ontology Alignment Tools: Use ontology alignment tools like AgreementMaker or LogMap that help ensure consistent mappings between integrated ontologies.
• Schema Validation: Validate ontologies and schemas before integration to ensure consistent terminology, definitions, and data structure.

6. Knowledge Graph Link Prediction Attacks

• Reputation Systems for Triples: Assign confidence scores to each RDF triple based on their source's reputation, which allows the system to filter out potentially malicious or low-confidence connections.
• Graph Consistency Checking: Perform continuous graph consistency checks and evaluate link predictions against predefined business rules to ensure no erroneous connections are introduced.

7. Denial of Service (DoS) through Reasoning Load

• Complexity Limiting Mechanisms: Limit the complexity of input queries or the depth of nested reasoning allowed in the system to prevent computational overload.
• Rate Limiting: Implement rate-limiting mechanisms to control the frequency of reasoning requests, which reduces the potential for overload due to maliciously complex inputs.
• Task Scheduling and Priority Handling: Implement intelligent scheduling of reasoning tasks, where less critical or computationally intensive tasks are given lower priority to prevent overload of essential processes.

8. Adversarial Schema Evolution

• Schema Versioning and Auditing: Introduce strict version control of schema and ontology definitions. Track all schema evolution with well-documented changes and audit trails.
• Automated Regression Testing: Conduct regression testing whenever schemas evolve to ensure that new classes and relationships do not break existing functionality or introduce inconsistencies.

9. Knowledge Graph Integrity Attacks

• Triple Authorization: Apply role-based or attribute-based authorization for updating triples to ensure that only trusted entities can make changes.
• Integrity Constraints: Define and enforce integrity constraints, such as referential integrity, to prevent unauthorized or harmful changes to RDF triples.
• Monitoring and Anomaly Detection: Continuously monitor RDF updates and employ anomaly detection to spot unusual changes in graph relationships.

10. Ontology Merging Attacks

• Mapping Validation: Validate ontology mappings before merging them using ontology mapping tools like AML (AgreementMakerLight) to ensure semantic alignment between ontologies.
• Automated Conflict Resolution: Use automated conflict resolution algorithms that can identify semantic conflicts in ontology merging and either flag or attempt to resolve them before integration.
• Human-in-the-Loop Verification: In cases of complex ontology merging, ensure that subject matter experts verify the merged ontologies to catch potential semantic ambiguities or misleading mappings.

11. Inference Engine Attacks

• Rule Complexity Constraints: Implement constraints on the complexity of rules that can be added to prevent complexity bombs and inference loops from being introduced.
• Circular Reasoning Detection: Use rule validation tools to detect circular logic before rules are introduced into production environments.
• Audit Trail for Rule Changes: Maintain an audit trail for changes made to reasoning rules, with notifications for administrators to ensure rule integrity.
• Timeout Mechanisms: Use timeout mechanisms during inference to avoid infinite processing loops that can occur due to inference loop creation attacks.

General Mitigation Recommendations for Symbolic AI:

• Regular Audits and Verification: Periodically audit the ontologies, rules, and knowledge graphs to ensure that they have not been compromised and are logically sound.
• AI-Driven Intrusion Detection: Implement AI-driven intrusion detection systems specifically designed to detect anomalies in symbolic AI data and reasoning processes.
• Redundant Consistency Checks: Incorporate consistency checks into various stages of data integration and reasoning to quickly identify and mitigate any unauthorized changes or inconsistencies.

These mitigations require a combination of technical tools, strict access controls, and validation procedures to maintain the integrity of symbolic AI systems, which rely heavily on structured relationships and logical inference. Deploying these defenses will help address the specific challenges of defending symbolic AI against the unique vulnerabilities we've highlighted.

Conclusion

The October 2024 update to MITRE ATLAS represents an important step forward in cataloging adversarial threats to AI systems, but significant gaps remain in addressing vulnerabilities unique to symbolic AI. As hybrid AI systems become increasingly prevalent, integrating ontologies, knowledge graphs, and deep learning, it is crucial that threat frameworks evolve accordingly. The absence of techniques such as ontology poisoning, inference engine attacks, and semantic misalignment leaves symbolic AI systems exposed to sophisticated adversaries. By expanding ATLAS to encompass these symbolic AI threats and recommending corresponding mitigations, organizations can better protect their advanced AI solutions, ensuring robustness, reliability, and security in an increasingly interconnected world.