The AI Genie, Inference, Demos, and Demons

Hi, I’m Robert, CEO & Founder of deepsense.ai. I’m not here to tell you that AI is great and is going to solve your 83 problems, I’m not here to scare you with AGI either. My goal is to help you look below high-level hype and above low-level details to see the useful middle ground and wrap your head around it. I would like to share my thoughts and insights, hopefully helping you learn something and learning something from you as well.??

For those who don’t know me, allow me to start with the boring part—a few sentences to give context to my journey before we jump right into it.

I’ve spent over ten years working in the AI space. With my team, we’ve delivered more than 200 projects across various verticals, ranging from research to applied AI, from San Francisco to Clermont-Ferrand (Where? Exactly.), and from working with the biggest companies to helping founders shape and build their initial ideas. I’ve also spent years educating myself, clients, and students, creating and teaching machine and deep learning courses at a university level, connecting recent developments with practical applications. I’ve built models, teams, and companies; won Kaggle competitions (saving whales with deep learning is still one of my fondest project memories); underfitted a few times and overfitted more than I’d like to admit.?

With this experience, I’ve seen successes and failures, watched trends and hopes emerge and fade (remember when everyone was supposed to become a Data Scientist?), and witnessed toolkits change completely (once, at a course I taught, I changed the “go-to deep learning library” three years in a row…).

What better place to start than reflecting on what has changed since we founded deepsense.ai ten years ago? Not in terms of all the exciting developments (there are many) but in terms of how our thinking about AI has changed. Spoiler alert: a lot and the best is yet to come!

The AI Genie—Can You Do This for Me, Please?

In the past, I’ve had countless conversations explaining that AI isn’t something you can just ask to do stuff. Especially not different tasks, and even less so numerous tasks across various domains. The prevailing concept was about a single-task model trained on neatly prepared, problem-specific data in the form of (input, output) pairs, with no “communication channel” other than getting the predictions as raw numbers. Now the AI genie is out of the bottle—ready for your three wishes (well, depending on the rate limit and general availability ??).

This has greatly reduced the entry and play barrier and provided a less rigid interface with a human touch—something we couldn’t figure out for years.

Yet, rigidity still has its place in the overall system architecture. We want to know what can happen, follow predefined scenarios, limit options, or just be able to understand and interpret results. Sometimes, limiting the possible options and knowing your errors is what makes a good sleep.??

Training is Important, but Inference is “Importanter” (sic!)

It used to be all about training. With supervised learning and single-purpose models leading the charge, it was all about how “smart” your model could get on a particular dataset and how accurate its predictions were when put to the task. Besides a few technicalities like averaging over a few runs due to introduced randomness or some ensembling, it was all about a single output given the input—whether your model could get this particular case right. You could maybe do some pre/post-processing here and there, but more or less, it either got it right or it didn’t—you were pretty much stuck with what you had. No prompt tuning, no system-level prompts, no context, and hitting retry wouldn’t help in any way either.

That was the reality, and still is, in some places - where single-purpose models and supervised learning rule the world. Yet, Generative AI changed the game's rules by making outputs much richer and shifting our focus on what we can get out of the same model without changing it in any way. Now, with prompting and similar techniques, we have another layer (of capability and complexity), giving us more space to play with. Not only can we manipulate the query and context to squeeze something new out of the model, but also we can flip the task completely. Not to forget the option to roll the dice again and generate another output (or fix some issues). And all this happens long after the training is done. That’s incredible. We can now reap and re-reap what we sow during training in different forms without repeating the big training procedure.

This has been reiterated with the recent o1 model from OpenAI, where the model gets significant performance (accuracy) gains with more time/compute at the test-time (so - inference). That’s a very useful property - not only does it allow to keep improving after the very costly training is done, but it also gives you flexibility in cost/performance control. You can think about it as having more time to figure out your next move in chess - more time thinking = more depth & breadth (both for humans and chess engines). This trend is very likely to continue.

From Demos to Demons

Building a demo with a working ML model inside used to be unreasonably effortful. You’d need to collect the training data, define the problem/classes well, skip all the corner cases, and select the subproblem carefully so that the model could learn something from the limited data you had (with very limited quality). At least the good part was that you’d need to do most of it anyway to move forward.

Welcome to 2024, where most of the AI part of a demo can usually be just “prompted away.” That’s incredibly useful for showing how something could look. The demonic part starts soon after you like what you see and want to move to an MVP or (hell forbid) production.

The issue is that to go from 0 to 1, you need to do X, but to go from 1 to 100, you may need to forget X and come back to 0 first. You may need to rebuild your solution completely, kill the flexibility with rigidity, and invest in MLOps. Compared to the past, that still seems like a good tradeoff - at least you should be able to test more ideas cheaply.

Aye-aye, Captain, What’s Next?

Summing up, the GenAI wave allowed us to hop over some of the challenges that barred us for years. A gini-like interface, ability to handle uncertainty and add a “human touch”, escaping the cumbersome data collection - train - test loop, to name a few. We also have a lot to look for ahead of us - recent progress in reasoning, as seen in math/coding, should eventually translate into less defined tasks with no formal verification. All this is redefining how we can think and build with AI. This also points to where I believe we haven’t made enough progress - the models are already very powerful, but the tools, UX, and the way of working with them are lagging. This is less important if you believe we can outpace this with increased model capabilities, but we certainly need it to reap more benefits now.

These are just a few highlights that stuck with me. Coming up, I’d like to do deep dives, focusing on a single topic within applied AI or exciting tech buzz. To give you a taste, here are just a few themes I find interesting: RAGs, AI on Edge, SLMs, LLM evaluation, grounding, alignment, general vs. specific solutions, and practical limits… So, just bear with me ??

P.S. Feedback, disagreements, and insightful comments are always welcome below!?

Robert