Building with AI Engineers

Hey builders! This month, we're diving into the world of AI Engineers and the tools they're using to build the future. But first, let's clarify what we mean by an AI Engineer. It's a builder who leverages AI products to create solutions.

These are the folks at the forefront of integrating AI into practical, real-world applications. I recently attended the AI Engineer World Fair and the AWS New York Summit, and I'm excited to share my thoughts on where I think we're heading!

AI Engineer World Fair: A Hub of Innovation

The AI Engineer World Fair was buzzing with excitement about creating AI-native workflows. It was incredible to see so many passionate builders eager to push the boundaries of what's possible. I had the pleasure of presenting, a workshop on building GenAI applications and running the demo booth for the Model Brawl League the LLM chatbot arena.

As I went through the fair, exchanging ideas with fellow builders, several key themes emerged that I believe will shape the future of AI engineering. Let me share my main takeaways:

1. The Elusive "Agent" Definition: One of the most striking observations was the lack of a unified framework for what constitutes an "Agent" or an "Agentic Framework." This ambiguity highlights the rapid evolution and diverse approaches in the field. Builders are exploring various frameworks, from simple task-completion bots to more complex, goal-oriented systems. Many talks and vendors were focused on how their product tackles this problem.

2. RAG's Continued Relevance: Despite the increasing size of context windows, Retrieval-Augmented Generation (RAG) remains important. It provides just-in-time information, to provide context aware responses for LLMs. Many vendors are trying to automate much of the "undifferentiated heavy lifting" such as chunking and ingestion. The end goal is to be able to imbue AI systems with "memory" so we can embed knowledge directly in their responses.

3. The Multimodal Future: There's a growing focus on multimodal AI. For models to move beyond simple chatbots, they need to process and understand multiple types of data inputs. Multimodal AI promises to bridge the gap between different forms of communication and data representation, allowing for more natural and comprehensive interactions between humans and AI systems. I attended a workshop where I got to build a real time voice agent and saw an OpenAI demo of how GPT-4o can interact with the world through a webcam.

This excitement set the stage perfectly for what I experienced next at the AWS NYC Summit.

AWS NYC Summit: Generative AI Takes Center Stage

The AWS NYC Summit was a one-day extravaganza packed with insights on building with Generative AI and some new launches like AWS App Studio, which is a new low code app builder powered by AI.

While technologies like web, mobile, and cloud have reached a plateau of productivity, Generative AI is just beginning its ascent. Some may argue that Gen AI is at the Peak of Inflated Expectations, but I firmly believe we're just scratching the surface of its potential.

The focus on Gen AI at the summit underscores this sentiment. We're witnessing the early stages of a transformative technology that will reshape how we build and interact with software.

When I talk to builders on the ground, their interests are clear:

• They want to learn how to incorporate GenAI into their workflow or build new apps, not how to set up a load balancer on EC2.
• They're eager to learn more prompt engineering to extract information from mountains of PDFs at scale, not set up S3 lifecycle migration.
• They're focused on automating tasks and placing guardrails around LLM hallucinations, not migrating apps to Kubernetes clusters.

This shift in focus doesn't mean that the "solved" use cases are unimportant. However, my hunch is that the vast majority of builders today will not need to care about these solved problems, just like many devs today don't focus on building compilers, setting up logic gates, or other low-level tasks.

The Age of the AI Engineer

All this points to one clear conclusion: the age of the AI Engineer is here, and it's time to learn and build together. Future applications are going to look dramatically different from what we're creating today.

If we take a page from OpenAI's Stages of AI, all Gen AI applications are at level 1 today (yes, even the agent workflows we have now). As AI becomes more capable, and we can offload more intelligence to these systems, we'll unlock whole new categories of applications and empower more people to become builders.

Andrej Karpathy's vision of a 100% Fully Software 2.0 computer is a great way to frame what type of system we are building. A computer with a single neural net with no classical software at all. Device inputs (audio, video, touch, etc.) would feed directly into a neural net, and its outputs would display directly as audio/video on speakers and screens.

Now, imagine what a cloud composed of millions of these 2.0 computers would look like, that's the future I'm excited about!

Embraced the Gen AI Future

As AI Engineers, we have the opportunity to shape the future of computing, to create tools and applications that were previously the stuff of science fiction. Yes, there are challenges to overcome and ethical considerations to navigate, but the potential benefits are immense.

So, to all the builders out there, let's embrace this Gen AI future. Let's learn, experiment, and build together. The applications we build today may seem basic compared to what's coming, but they're crucial steps on the path to that exciting future.

Now, let's build!