A New Frontier in AI Research: Automated Design of Agentic Systems

I want to share something with you today, about where I think AI research is headed in the near future. As a computer programmer with a background in AI, data engineering, software development, and bots, I’ve been fascinated for years by the potential of AI to automate complex tasks. Recently, I’ve been exploring a new area of research within AI that I believe has the potential to revolutionize the way we approach AI development: It’s called Automated Design of Agentic Systems (ADAS).

The Case for ADAS: Bridging the Gap Between Manual and Automated

The current landscape of agentic systems requires significant manual effort, heavy reliance on domain expertise, and often leads to bottlenecks and inefficiencies. Yet, history tells us that systems yielding to learned experiences consistently outstrip their handcrafted counterparts. ADAS offers us a way to automate these cumbersome processes, streamlining the journey toward innovative solutions.

Some problems are too complex for a one-step query to an LLM. For example, writing a book.

Yes, LLM can write a book, but, hmmm, you might not be satisfied with the result!

For complex problems, it’s best to break down into sub-tasks. But, human domain expertise can create the processes with subject matter expertise. But what would be better, is for AI system to do its own research, discover what agent should do, and create it… and so, the area of new research into AI is creating process where the AI can self-discover what subject matter expertise it needs, and create an agent to perform the necessary processes.

For writing a book, to use our example… human writers use a process like this:

1. Start with a rough story idea 2. Develop the world. This is called word-building. You define the technology, customs, point in time (is it the future, the past, or the present), et cetera. 3. Develop characters (backstory, personality, how they speak, education, income, ethnicity, profession, …) 4. For each character, develop their story, using beats, into a beat sheet. 5. Merge each character’s beat sheet into one unified beat sheet. 6. Using the unified beat sheet, create sub-beats for each beat 6. Work through each sub-beat, and expand to two or three sentences describing what happens. 8. For each sub-beat, write the action, dialogue, narration.

And then, there’s other details, like, choosing a writing style. You probably wouldn’t like the default writing style of the AI. For example, recently, when asking AI to write a preface to a computer book about AI I am working on, it gave me hyped talk like this: “Get ready! Buckle your seat-belts! We’re about to take you on a journey into the exciting world of AI. Boy oh boy, you’ll be truly amazed at the wonderful things you will discover in this book!”

Nobody I know talks like that. It’s a tell the author is not a normal person. It’s so enthusiastic, and promotional. A dead-giveaway. Totally useless.

The foundation of ADAS

ADAS is built on three foundational components:

1. Search Space: This defines the universe of potential agentic systems, whether they’re based on code structures, graphical representations, or neuron-like networks.

2. Search Algorithm: Here, we explore how the search space is navigated. By applying techniques like reinforcement learning and the intelligence of Foundation Models, ADAS dynamically generates new solutions based on learned experiences.

3. Evaluation Function: This mechanism assesses candidates for their ability to meet criteria such as performance, cost-efficiency, and safety.

Meta Agent Search: Unlocking New Pathways

One of the pioneering algorithms within ADAS is Meta Agent Search. This meta agent crafts new agents by utilizing a growing archive of existing discoveries, making full use of the expressive capabilities of programming languages. Through this lens, the potential to discover an extensive range of agentic systems, including innovative prompts and operational flows, becomes evident.

Research into Meta Agent Search reveals a compelling narrative. Agents emerging from this process have shown superior performance compared to traditional handcrafted systems. In trials using reading comprehension and mathematical tasks, Meta Agent Search demonstrated improvements of in evaluation scores. These readings not only underscore the technical edge of ADAS but also hint at its ability to catalyze a fundamental shift in our methodologies.

Here’s how ADAS works…

Agents, as they are created, are stored in a library. And a list of the agents, (e.g. their name, what they do, how to call them, what’s the output) is maintained.

An LLM is asked to propose a new agent. But, we want the LLM to avoid prososing something the system already has in its library, and also we want the LLM to be aware of what agents in the library can be used as building blocks as compentnets or something new.

Soooo, as we prepare to ask the LLM for help in proposing a new agent, the list is agents is loaded into context for a prompt, so the LLM knows what agents are already in the library collection, and that also informs the LLM about building blocks that can be used as compoentns in building a new agent.

Code calls upon an LLM with prompt something like this: “Your goal is to propose interesting new agents. Use your knowledge of my collection of existing agents. That knowledge of existing agents will help you avoid proposing something that duplicates existing agent unless the new agent somehow improves upon an existing agent. That knowledge of existing agents will also make you aware of potentials for using those existing agents as building block components for new agents for inspiration, you may draw upon academic literature for ideas for new agents.”

Until now, creating apps requires human subject matter expertise. but, combining LLM and web search, the ADAS can discover/obtain/retireve the sme info by itself without a human involved guiding it

Future Directions

The horizon for ADAS is filled with potential. Here are some paths that could lead to exciting advancements:

Expanding Search Spaces: Current frameworks could benefit from exploration beyond code-centric spaces, delving into graph-based models and other architectures.

Enhancing Search Algorithms: The development of more efficient algorithms to navigate large search territories while evading local minima will significantly enhance agent discovery.

Broader Evaluation Criteria: By establishing more comprehensive evaluation metrics, we can obtain a nuanced picture of agent performance, which in turn enables more informed decision-making.

The Broader Impact: Societal Implications of ADAS

While technical innovations are fascinating, we must also consider the broader societal implications. By relieving humans from mundane design processes, ADAS heralds new opportunities — whether streamlining software engineering, reshaping educational approaches, or revolutionizing creative fields such as music and film. Imagine an AI not just scripting video games but composing entire movies in real time — this is not far from the horizon.

Conclusion: Embracing the Future of ADAS

ADAS serves as a beacon at the frontier of AI research, set to redefine our engagement with intelligent systems. By grasping its complex mechanisms and recognizing far-reaching implications, programmers can better navigate the evolving landscape of AI. Whether you’re a seasoned expert or a newcomer, the exploration of ADAS opens up a world rich with possibilities.

#ADAS, #AIResearch, #AgenticSystems, #AutomatedDesign, #AIInnovation, #FutureOfAI, #IntelligentSystems, #MachineLearning, #AIApplications, #SoftwareDevelopment, #AIEngineering, #ArtificialIntelligence, #TechAdvancements, #AIRevolution, #DigitalTransformation, #AIforGood, #AIinSociety, #AIandEthics, #AIandInnovation