Building Compound AI systems

Introduction

In this article, I will explain how I think about building and optimizing compound AI pipelines with Langtrace , DSPy and CrewAI . I strongly believe that this is how compound AI systems optimized for high performance and reliability will be built in the future.

Composing the pipeline

Create individual projects for each block of the pipeline in Langtrace. This represents the individual blocks of my pipeline.

Setting up DSPy projects

I think of each block as a set of inputs and outputs. For tasks that have well defined/structured input(s) and output(s), I use DSPy where I pick an optimizer and write an evaluation function that optimizes for the given metric. I create 'DSPy' projects on Langtrace for this and run experiments until I get what I want.

The nice thing about Langtrace is, each time I run my experiments, it automatically captures the checkpoints. Checkpoints can be directly copied and deployed to production once I achieve the desired metric. Additionally, I can also keep track of my cost so I know when to switch to a lower cost model if I need to lower my cost.

Setting up CrewAI projects

For open ended tasks like doing a research on a particular topic where I don't look for a specific format, I use CrewAI and I create a 'CrewAI' project on Langtrace to capture sessions and tweak/iterate it until I get what I need consistently.

Within the CrewAI dash, I can track every session, the agent(s) deployed and it's associated tasks and tools, along with a session drill down of what's actually going under the hood. This helps me identify issues like prompt front loading, latency etc. and iterate until I achieve the ideal outcome. Once I get there, I just deploy the CrewAI agent out to production.

Conclusion

Finally, if you are interested in building Compound AI pipelines like these, check out Langtrace and leave any feedback you have. It's free to get started and I am happy to provide additional credits if needed.