Building an AI agent

Lately, our news and LinkedIn feeds are full of talk of “AI agents”, promising sliced-bread level of improvements to everything software. Intrigued (and definitely a tad skeptical), Rafal Kapela and I embarked on our own agent-building journey, to see what all the fuss was about. This essay chronicles our experience, both technical observations and some product "aha" moments we encountered along the way.

What is an agent?

First, let's demystify the term "agent." Simply put, it's a new (or fancy, if you will) way to say “a software program that creates a guided and controlled conversation with a Large Language Model (LLM) like ChatGPT, and other software”. If you've ever used ChatGPT to write an article or an email by asking it to write something, and then refining its output through successive chat messages (a.k.a. prompts), congratulations! You've already experienced the power of an "agentic flow."?

Building and agent = writing a program that creates a structured back-and-forth, whereby the program takes the LLM's output, processes it, and uses it to formulate the next prompt to the LLM, or a step not involving an LLM, such as performing calculations, searching the web, or accessing external data (called “tools”). So it’s just a new way to write software, and because we have an LLM in the mix, it is more powerful, less predictable, and more conducive to writing blog posts about and creating startups out of ;)

If you’ve never written software, or dealt with a software dev team, here’s one way to visualize an agent: Imagine a highly motivated intern with a large rote memory and unwavering focus. They are eager to please but need explicit, step-by-step instructions to complete any task. How detailed depends on how complex the task.Oh, and their judgment is still under development, so you wouldn't trust them with critical decision-making (yet!).

Our toy project: An Essay-Writing Agent

For our foray into agent creation, we chose a familiar task: writing an essay. It felt like a manageable first project, seemingly just a few coding steps away from our everyday ChatGPT interactions. You know, like creating emails with our POV on some long article a co-worker shared on a mailing list that also has our manager on it.

To build our agent, we broke the essay-writing process into discrete stages:

1. Plan: Given the essay topic, come up with the key points to make.
2. Research: Gather relevant information and supporting evidence for these points.
3. Draft: Create the first draft of the essay.
4. Review: Evaluate the draft for clarity, coherence, accuracy, style, etc.
5. Edit: Incorporate feedback and refine the essay.
6. Iterate: Review and again, and Edit again, until satisfied.

Thinking in Graphs

As it turns out, our boxes-and-arrows workflow diagram isn't just a visual aid; it defines the program structure of an agent perfectly. And LangGraph, an open source framework for building agent applications, utilizes the concept of graphs like this diagram to represent agent workflows and make it easy to code them up.

For those interested a little in the details, here are the core components of these graphs:

• Nodes: Each node in a graph represents a distinct step in the agent's workflow (e.g., "Plan," "Research," "Review"). START and END are special nodes that… well, tell the program where to start and end.?
• Edges: Edges connect these nodes, defining the flow of information and sequence of actions. For example, in the graph above, the “Draft” step is followed by the “Review” step, and typically, the output of the "Draft" node would be fed as one of the inputs to the "Review" node as well.?
• State: State refers to all the things that we want to keep track of e.g. The user input topic, a “current draft” of the essay, a list of snippets of info from an internet search, etc. Back to our intern analogy, this is everything you want the intern to be tracking, either in their head or on paper.

Using LangGraph, we could translate our diagram above into functional working python code, with each stage represented as a node. Each node is just a function, that deals with:

• Input: The information required to execute the task (e.g., essay topic, research keywords).
• Output: The result of the task (e.g., an outline, a paragraph of text).
• State: The ability to store intermediate results, and track overall progress of our agent.?

From Concept to Code and the messy reality

With our LangGraph agent designed, it seemed straightforward to bring our agent to life through code. And yes, there was sample code available online too, so we didn’t have to actually start from scratch. But we chose to, because it seemed simple and we know we’ll want to experience the pain in small steps ?? . Also, we wanted to be able to create an actual web-app where anyone could run this agent, not just a proof-of-concept that ran on our own machines (for that, there are Juypter notebooks with working code available a short Perplexity search away. But what’s the fun in that!). So here we are.

And as with any software project, this too came with its fair share of setup grunge work and hurdles. From a cold start (i.e. a mac-mini with only default software dev packages, we went through installing homebrew, pip, 3 versions of python, some click-and-shoot Docker container setups, LLM API keys, and had something that worked. As a side note Anthropic and OpenAI are hugely ?easier to use than Google’s Gemini. It’s almost like Google doesn’t want developers to quickly get going ??. For folks intending to get into this seriously, we highly recommend going through the basic documentation of the LangGraph library. Some concepts (like reducers) can be tricky and are very important to understand before starting to work on AI agents even on a basic level [1], [2].

Further on, we wired up LangSmith (a platform for debugging and monitoring agents) and LangGraph Studio (a visual tool for building LangChain graphs, currently working only on Mac). While powerful, these tools came with learning curves of their own. We think it’s going to be worth it. LangSmith is quite flaky though, and every time it updates it makes me feel like the code will break. But, as with any new project, these issues should settle down with time.

Showtime!

After all the above setup, we finally had a running agent! With anticipation and ??, we hit the "execute" button in LangGraph Studio. The graph sprang to life, nodes lighting up sequentially as our agent diligently progressed through each stage of the essay-writing process.

Seeing our creation come alive was satisfying, but the real work was just beginning.

The first essays produced by our agent, while structurally sound, were far from Pulitzer-worthy. This is where the iterative nature of agent development comes into play. We experimented with different prompts, fiddled with agent state variables and steps, and explored competing LLM providers and configurations. Each tweak aimed to enhance the quality of our agent's output, focusing on three key areas:

• Improved Planning: How do we generate more creative and insightful essay outlines?
• Enhanced Drafting: How could we push the LLM to produce more engaging and well-argued paragraphs?
• Sharper Critique: Can we develop more nuanced and insightful feedback mechanisms within the agent's review process?

Each node in our graph is a playground for experimentation, offering opportunities to inject domain knowledge, refine instructions, and leverage the ever-evolving capabilities of LLMs.?

What’s next??

We have in mind a few other aspects of making this agent more useful, like?

• Style and Voice: What, or who, should the output sound like? Today, the outputs are still disappointingly smelling-like-LLM. What kind of work would allow us to style-transfer from our own writing, or that or someone we admire?
• The Stopping Problem: Our agent can critique whatever draft it is given. Like an employee out of Dilber’s office (or yours), we seem to have transferred the human-like capability to “give inputs” on anything. In real-life situations, we stop when we think something is good enough, or when the deadline approaches, or when people are sick of meeting on this topic one more time. What’s the equivalent in agent-speak? How might we formulate a stopping problem for our program?

References:

[1] Low Level LangGraph concepts https://langchain-ai.github.io/langgraph/concepts/low_level/#reducers?

[2] LangGraph: Context Objects in State https://langchain-ai.github.io/langgraph/how-tos/state-context-key/?

That’s all for now! Here’s a picture of Niki and Nora playing cat-in-the-bag, as a thank you for reading this far.