An Analysis of LangChain's Reusability in LLMs: Challenges and Insights

LangChain, despite its ambitious goals, falls short in delivering reusability in the context of large language models (LLMs). The core issue is that LangChain tries to build abstractions on top of technical foundations that simply aren’t capable of supporting them effectively. Here's my perspective on why this is the case and what it means for using LLMs like GPT-4 and GPT-3.5 in complex applications.

The Core Argument: Lack of Reusability

In my experience, the current generation of LLMs doesn’t support reusability. Even with the impressive capabilities of models like GPT-4 and GPT-3.5, the need for custom prompt engineering and specific data formatting for each feature or chain step is a significant hurdle. Over several months, I’ve been building features using sophisticated LLM chains that perform various types of reasoning. While the human-like outputs are impressive, each feature requires meticulously crafted prompts and precise data formatting.

The reality is stark: 95% of my work in setting up LLM chains involves fine-tuning prompts and formatting data correctly. Only about 5% is dedicated to orchestrating Directed Acyclic Graphs (DAGs) to run these chains. For those unfamiliar, a DAG is a Directed Acyclic Graph, a structure used to model workflows where tasks are executed based on dependencies, ensuring that each task runs only after its preceding tasks have completed. This makes LangChain’s attempt to create reusable abstractions result in a "mediocre DAG framework" where data and instructions degrade as the chain grows longer.

Practical Experiences and Internal Solutions

I even tried building my own internal version of LangChain in TypeScript. I found it easier to develop and refine a framework I could improve over time. In my version, all internal prompts come from an external library of prompts, inspired by Daniel Miessler's Fabric library. Despite these efforts, the fundamental issue remained: the bespoke nature of prompt engineering and data formatting required for each feature.

LangChain’s lack of transparency is another significant frustration. I had to dig into the source code and manipulate private variables just to tweak a single prompt. The default prompts they use are often suboptimal, relying heavily on the intelligence of models like GPT-4 and GPT-3.5. This approach falls apart with less advanced open LLMs, requiring significant prompt rewriting.

Insights into Prompt Engineering

One key insight I’ve gained is understanding how LLMs process and generate responses. For example, LangChain’s MULTI_PROMPT_ROUTER_TEMPLATE is overly complex and doesn’t align with the LLM’s auto-completion nature. Instead of using unconventional formatting and numerous rules, I found that a simpler approach, leveraging common patterns and clear examples, works much better.

By simplifying the prompt, it became more effective even for less advanced models like GPT4All. This highlights the need to adapt prompt engineering strategies to the specific capabilities and limitations of the LLMs in use.

Broader Implications

These challenges have broader implications for developing and deploying LLM-based applications. The heavy reliance on custom prompts and data formats suggests that, while LLMs have advanced, their practical application still requires substantial manual intervention and fine-tuning. This limits the scalability and reusability of solutions built on these models, posing a significant hurdle for frameworks like LangChain.

Moreover, developers need flexibility and transparency in LLM frameworks. The ability to easily customise and optimise prompts without being hindered by the framework’s abstractions is crucial. There needs to be a balance between providing useful abstractions and maintaining the flexibility necessary for effective prompt engineering.

What can be done ?

LangChain’s approach to reusability in LLMs is fundamentally flawed. The bespoke nature of prompt engineering and data formatting, combined with the limitations of current LLMs, makes reusability impractical. As LLM technology evolves, frameworks like LangChain must adapt, balancing abstraction with the necessary flexibility for effective application development. Through my own efforts in developing a TypeScript-based framework with external prompt libraries, I’ve seen firsthand the importance of customisation and the challenges of achieving true reusability in LLM applications.

Future Directions: Embracing the Infancy of LLM Technology

It's crucial to acknowledge that we are still in the infancy of this technology. This early stage presents an opportunity for everyone in the field to explore and find better ways to build chains of thought and implement robust tooling. Among these, Retrieval-Augmented Generation (RAG) stands out as particularly important. RAG combines retrieval mechanisms with generative models, enhancing the accuracy and relevance of outputs by grounding them in external knowledge. Embracing these advancements will help us overcome current limitations and pave the way for more scalable and reusable solutions in the future.