Case-Based Reasoning (CBR) with Large Language Models (LLMs)

Using Case-Based Reasoning (CBR) with Large Language Models (LLMs) involves integrating CBR methodologies into the framework of LLMs to enhance their ability to solve problems by leveraging past cases. The CBR framework operates by retrieving similar past problems, reusing their solutions for the current problem, evaluating the effectiveness, revising the solution, and retaining successful solutions. Utilizing CBR enables LLM agents to analyze, extract and reuse solution patterns from these human insights, and to iteratively revise the solution based on the execution feedback to attain consistently improved performance. The integration of CBR into LLM agents not only enhances their problem-solving abilities in tasks but also achieves high efficiency in both sample and computation resources.

Here’s a step-by-step approach on how to integrate and use CBR with LLMs:

Case Retrieval:

Definition of a Case: First, define what constitutes a "case" in the context of your application. A case typically includes a problem, a solution, and possibly an outcome or feedback.

Building a Case Database: Collect and store cases in a structured format that an LLM can access. This database should be rich and diverse to cover various scenarios the LLM might encounter.

Retrieval Mechanism: Implement a retrieval mechanism that allows the LLM to query and retrieve relevant cases based on the current problem. This can be achieved through similarity search, where the current problem is compared with stored cases to find matches.

Adaptation:

Relevance Analysis: Once relevant cases are retrieved, analyze them to determine how closely they match the current problem. This step may involve LLMs interpreting and extracting key elements or patterns from the cases.

Solution Adaptation: Adapt the solutions from past cases to fit the current problem. This might involve modifying parameters, combining elements from multiple cases, or generating new solutions under the guidance of past examples.

Solution Testing and Revision:

Apply and Evaluate: Apply the adapted solution to the current problem and evaluate its effectiveness. This evaluation can be based on predefined criteria or feedback mechanisms.

Revision: If the solution is not satisfactory, revise it by further adapting or seeking additional cases. This iterative process continues until an acceptable solution is found.

Learning and Feedback:

Feedback Integration: Incorporate feedback from the effectiveness of the applied solution back into the system. This can help in refining the solution strategies and updating the case database.

Case Update: Successful solutions can be documented as new cases in the database, thus enriching the resource for future problem-solving.

Integration with LLMs:

Modular Integration: Treat the CBR system as a modular component in the LLM architecture, where the LLM can call upon the CBR system when needed.

Seamless Interaction: Ensure seamless interaction between the LLM and CBR components so that the LLM can automatically use CBR techniques during its reasoning and generation processes.

By following these steps, you can effectively integrate CBR with LLMs to create a robust AI system capable of learning from past experiences and improving its problem-solving capabilities over time.

For example:

DS-Agent, a novel framework that harnesses LLM agent and CBR to facilitate model-centric automated data science.

Architecture:

When CBR used with LLM for data science tasks the DS-Agent with GPT-4 achieves an unprecedented 100% success rate in the development stage. https://arxiv.org/pdf/2402.17453

You need to build automated data science tasks using CBR? Please check this out - https://github.com/guosyjlu/DS-Agent

Thank you.