Techno-Realistic Optimism in AI: A Realo Look at Technology's Frontiers

I am blessed to work with many bright people around the world, all genuinly interested in understanding and working out AI tech. During our deep-dives in AI tech, newly gained tech-capabilities and also the reaction of (basically all of) us on some fabulous new AI stuff the dialogue often focussed around the “makability” of it all. But also around the huge investments in money, energy and time needed to create and run it. Especially if you complete the equation with the downsides on errorness, trust, honesty of how models are build and what they actually contain. Optimism is roaring with some, while others lift an eyebrow and seem to ask “really?”. While discussing the possibilities and potential blockers with our clients for well over a year now, it shows sometimes hard to convey a realistic message.

Following the money is in such cases often a good idea for building the message. Businesses understand investment. When looking at the marked last 1,5 year, there has been a huge investment in early-stage models and tools with (yet to be proven) market value, in the expectation that much will fail and some will prevail and become unicorns. That is normal and it is good, while it provides money to develop use cases and capabilities. On the other hand, high-profile VCs often do not join the first wave of investments and start to invest if they see large potential revenue. In that context TechCrunchs′ Kyle Wiggers had a nice sump on the behaviour of the more institutional VCs in current AI tech and where they put their money:

“Several high-profile VCs, including Meritech Capital?—?whose bets include Facebook and Salesforce?—?TCV, General Atlantic and Blackstone, have steered clear of generative AI so far. And generative AI’s largest customers, corporations, seem increasingly skeptical of the tech’s promises, and whether it can deliver on them.” (Kyle Wiggers, TechCrunch: https://techcrunch.com/2024/04/15/investors-are-growing-increasingly-wary-of-ai/)?

But also Gary Marcus (after TED 2024, in a sum-up) said on the topic of investement in AI: “dreamers who promise far more than they are likely to deliver are getting almost all the money, and (at least in AI) they are getting so much money for a path that it is unlikely to work that they are leaving too little oxygen for the exploration of new ideas that might work.”

The latter might be an interesting statement to look a bit closer at. Current models have a huge lead time (several months to train), require much data (often with unknown legal quality, unfortunately) and the capabilities seem to improve in a linear manner, whereas the volume of data needed seem to increase exponentially. And mind you, with exponential data needs, and with a raising understanding that current training data might have legal issues (and thus has to be excluded next time), while at the same time current models “pollute” the internet with auto-generated content. So no wonder that these VCs start to ask questions about the ROI of Generative AI for various use cases.?

But is it all that dark? Indeed, with a current paradigm that “bigger is better” and “more data will solve many issues”, we might have a problem. Making sure we do not “stand still with the idea of scale” but look in several directions for the next step, might be instrumental. While at the same time utilizing what we have now in a realistic way, which will require some guideance and ruling. But this is not a popular discussion on americas west-coast, where much of the investment in current AI Tech has been done. Again the ROI thing, maybe? A lot of money is invested, and talking about some serious problems with AI (like hallucination and trust) is not serving the cause of finding a good payback opty.?

But on the optimistic site, we do start to see some interesting patterns and possibilities popping up:

• A growing understanding that current AI models (of various types, not only LLMs) struggle if they enter the real world, while real-world context is utterly subtile and complex, and that this probably requires better abstraction capabilities. Mammals with bigger brains than ours (and thus more connections), but a lesser neo-cortex, have not developed the same abstraction and self-awareness as humans have developed. Algorithms like Liquid Neural Networks and Neuromorphic Computing gain in popularity. Maybe while initial tests hold a promise that it is possible to do better with less?
• A growing recognition that a generic let′s-do-all-tasks-in-one-model approach is probably not always the way to go: some tasks (like math, physics, logics) are perfectly solvable without GenAI, with 100% correctness, and in much more efficient ways. For other tasks LNNs, RNNs show great performance for much less compute and training. Contemporary AI agent systems indeed start to offload such tasks through APIs, while also being able to use other functionality already implemented more effectively elsewhere (like booking systems, weather prediction, business operation systems, etc).
• A trend to build smaller models that are modelling and executing rather specific tasks very well, can be trained and run with a low environmental/energy footprint, work quickly for the task at hand, and do not have to forward “unnecessary world knowledge”.
• An increasing interest in Multi-A(I)gent systems, combining smaller-AI-model-agents with “capability agents” (the math, physical, logics, booking task specific agents) can come to the rescue to mesh together the minimum of functionality needed to solve a problem. By that such systems can stay capable within their context, while carrying minimal overhead. And the quality of the system as a whole will increase if tasks that can be performed with 100% correctness (while good abstraction models exist, again math, physics and the like), actually are offloaded to capability agents.
• Multi-agent approaches also solve other issues, e.g the issue of “forgetting”. When new insights become available, it could happen that old knowledge should be “unlearned”. But current models do not really have that capability. Or imagine the case where some data-set used for initial training has to be excluded while a court decides some IP is infringed. Retrain the whole LLM from scratch? Anyone?
• In addition there is a growing number of initiatives that aim at improving the core of AI models, and make the basic of the used algorithms smarter and more flexible with the aim to do a better job with (much) less resources (data, energy, water, hardware). Some companies, like LiquidAI, have started to explore alternative routes and we expect to see more of such soon.
• And when we are talking about the algorithms, one can also have a look at the hardware. If you do more probabilistic types of tasks, do you still want to use a binary (0/1) paradigm?? Or can you do something different? Approaches like neuro-morphic computing, analogue- and ionic processors all try to become alternatives to current hardware.

The field of AI has always been exciting, with waves coming and going, and every time we gained something more. Having been able to really build something that can help us to naturally interact with the digital world is already a breakthrough, and the coming years will bring very many new inventions. So let’s take the current consolidation for what it is: great functionality for some use cases, an opportunity to develop further on the basis of the tech and improve. At the same time we need some serious effort and progress on how we can and should tackle real-world context. Current systems are fooled way too easily, something which might prevent them from becoming mainstream, if not solved in a satisfying and sustainable manner. Throwing more compute, data and larger models at it does not seem to be able to solve those issues. New and other ways of thinking might be required.

The best strategy for dealing with AI nowadays might simple be short-term Techno-Realism and long-term Techno-Optimism. That would help us finding the real gems, and allow us to cut out the crap.