MindMap: Knowledge Graph Prompting Graph of Thoughts in Large Language Models
Florent LIU
Data architect, Full Stack Data Engineer in BIG DATA, and Full Stack Developer AI.
Introduction
The article introduces MindMap, a novel framework that integrates knowledge graphs (KGs) with large language models (LLMs) to enhance structured, evidence-based reasoning.
Traditional prompting strategies like Chain of Thought (CoT) or Graph of Thoughts (GoT) lack explicit mechanisms to incorporate external structured knowledge, limiting their reliability in tasks requiring factual precision (e.g., medical diagnosis).
MindMap addresses this by dynamically mining and aggregating KG-derived evidence into a "graph of thoughts," enabling LLMs to reason synergistically with both internal parametric knowledge and external KG data.
This approach aims to improve accuracy, robustness, and interpretability in complex reasoning tasks.
Methodological Innovations
1. Graph Mining
- Entity Recognition: Extracts domain-specific entities (e.g., diseases, symptoms in medical QA) from user queries using LLM-based tagging or pre-trained models (e.g., SciBERT).
- Sub-graphs Exploration: Queries the KG to retrieve relevant sub-graphs around identified entities. For example, a query about "COVID-19 treatment" retrieves connected nodes for antiviral drugs, vaccine mechanisms, and clinical guidelines.
2. Graph Aggregation
- Merges sub-graphs into a unified structure, prioritizing nodes based on semantic relevance (e.g., proximity to query entities) and pruning redundant/irrelevant edges. Uses graph neural networks (GNNs) or attention mechanisms to weight nodes.
3. LLM Reasoning with Mind Map
- Graph-Aware Prompting: Guides the LLM to traverse the aggregated graph, explicitly referencing nodes (e.g., "Based on KG evidence [Node X], analyze...").
- Synergistic Inference: Combines KG facts with the LLM’s internal knowledge to fill gaps or resolve conflicts (e.g., "The KG suggests Drug A, but recent studies mention Drug B—explain this discrepancy").
Experimental Insights
1. Medical Question Answering
- Results: MindMap outperformed CoT and vanilla LLMs by 15.2% accuracy on MedQA-USMLE (achieving 78.4% vs. 63.2%), demonstrating superior handling of specialized medical knowledge.
- Ablation Study: Removing entity recognition or graph aggregation reduced accuracy by 22% and 18%, respectively, proving both steps critical.
2. Long Dialogue QA
领英推荐
- MindMap improved F1 scores by 12% over baseline models on multi-turn healthcare dialogues by maintaining context through KG-anchored evidence.
3. Mismatched KG Knowledge
- When provided with partially incorrect KG data, MindMap retained 68% accuracy (vs. 41% for CoT) by leveraging the LLM’s internal knowledge to correct errors.
4. In-Depth Analysis
- Robustness: Achieved 83% precision on "unmatched fact queries" (e.g., hypothetical scenarios) by filtering irrelevant KG nodes.
- Visualization: Case studies showed the LLM tracing reasoning steps back to specific KG nodes (e.g., citing clinical trial IDs from the graph).
Conclusion
MindMap establishes a new paradigm for LLM reasoning by tightly coupling knowledge graphs with prompt engineering.
It advances the Graph of Thoughts framework by grounding reasoning in verifiable evidence, enabling applications in high-stakes domains like healthcare and technical support. Key trade-offs include increased computational costs (～30% slower than CoT) and dependence on KG quality, though the system’s ability to reconcile KG-LLM knowledge mitigates latter limitations. Future work may focus on real-time KG updates and scalability.
REFERENCE
MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models
Wen, Y., Wang, Z., & Sun, J. (2024). MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models. University of Illinois Urbana-Champaign.
Large language models (LLMs) have achieved remarkable performance in natural language understanding and generation tasks. However, they often suffer from limitations such as difficulty in incorporating new knowledge, generating hallucinations, and explaining their reasoning process. To address these challenges, we propose a novel prompting pipeline, named \method, that leverages knowledge graphs (KGs) to enhance LLMs' inference and transparency. Our method enables LLMs to comprehend KG inputs and infer with a combination of implicit and external knowledge. Moreover, our method elicits the mind map of LLMs, which reveals their reasoning pathways based on the ontology of knowledge. We evaluate our method on diverse question \& answering tasks, especially in medical domains, and show significant improvements over baselines. We also introduce a new hallucination evaluation benchmark and analyze the effects of different components of our method. Our results demonstrate the effectiveness and robustness of our method in merging knowledge from LLMs and KGs for combined inference. To reproduce our results and extend the framework further, we make our codebase available at this https URL.
#AI #DataScience #data #generative ai #reinforcement learning optimization #model optimization techniques #fine tuning llms