Letta

Introducing the AI agents stack: breaking down today’s tech stack for building AI agents into three key layers: (1) agent hosting/serving, (2) agent frameworks, (3) LLM models & storage. Sarah Wooders and I got so tired of seeing bad “market maps” for LLM / AI agents shoved in our feeds that either had layouts that made no sense or were littered with random companies on them (ie that don’t have serious community adoption), or both. As researchers / engineers actually working in the agents space, we decided to make a serious attempt at making one of these market map diagrams that actually reflects real world usage by today’s developers building AI agents. Basically - if you’re starting a vertical agents company today (November 2024), what software are you most likely to use to build out your “agents stack”? In our opinion, the AI/LLM agents stack is a significant departure from the standard LLM stack. The key difference between the two lies in managing state: LLM serving platforms are generally stateless, whereas agent serving platforms need to be stateful (retain the state of the agent server-side). Building stateful services is a lot harder of an engineering challenge compared to building developer SDKs, so unsurprisingly very few agent serving platforms actually exist today (Letta being one of them). For a full breakdown of the stack, check out our full post (link in comments).

New short course: LLMs as Operating Systems: Agent Memory, created with Letta, and taught by its founders Charles Packer and Sarah Wooders. An LLM's input context window has limited space. Using a longer input context also costs more and results in slower processing. So, managing what's stored in this context window is important. In the innovative paper "MemGPT: Towards LLMs as Operating Systems," its authors (which include Charles and Sarah) proposed using an LLM agent to manage this context window. Their system uses a large persistent memory that stores everything that could be included in the input context, and an agent decides what is actually included. Take the example of building a chatbot that needs to remember what's been said earlier in a conversation (perhaps over many days of interaction with a user). As the conversation's length grows, the memory management agent will move information from the input context to a persistent searchable database, summarize information to keep relevant facts in the input context; and restore relevant conversation elements from further back in time. This allows a chatbot to keep what's currently most relevant in its input context memory to generate the next response. When I read the original MemGPT paper, I thought it was an innovative technique for handling memory for LLMs. The open-source Letta framework, which we'll use in this course, makes MemGPT easy to implement. It adds memory to your LLM agents and gives them transparent long-term memory. In detail, you’ll learn: - How to build an agent that can edit its own limited input context memory, using tools and multi-step reasoning? - What is a memory hierarchy (an idea from computer operating systems, which use a cache to speed up memory access), and how these ideas apply to managing the LLM input context (where the input context window is a "cache" storing the most relevant information; and an agent decides what to move in and out of this to/from a larger persistent storage system)? - How to implement multi-agent collaboration by letting different agents share blocks of memory This course will give you a sophisticated understanding of memory management for LLMs, which is important for chatbots having long conversations, and for complex agentic workflows. Please sign up here!? https://lnkd.in/gu2BgqJE

Kevin Lin is joining our research team from Berkeley NLP! Kevin was previously a PhD student at University of California, Berkeley, and prior to the PhD worked as a researcher at the Allen Institute for AI (Ai2). Kevin has pioneered context management for language agents, with research spanning retrieval, long-context, and evaluation. Kevin was also the third author on the MemGPT paper and shared Joseph Gonzalez as an advisor with me and Sarah Wooders (please see attached photo of the three of us enjoying a pint at the famous Joey Lab Social ?? ). We are incredibly excited to have him back on the team at Letta! ??

Matt is joining our engineering team from the platform team at Databricks (and previously University of California, Berkeley). At Letta, Matt will be working on the open source and cloud platform to take bleeding-edge agent architectures from research into production. Welcome Matt!! ??

shubham I???????????????????????????????????????????? naik is joining our engineering team from Snorkel AI / Ancestry / Klaviyo, and previously took multiple LLM playgrounds from 0→1 including at Together AI. Shubham is leading development on the ADE (Agent Development Environment), a new kind of interface to develop, debug, and deploy advanced LLM agents. We can’t wait to show you what he’s been working on at Letta. ?? Welcome Shubham!! ??