How to think about LLMs and The Cloud

We’re on the verge of the next great digital transformation. LLMs are making their way into every aspect of enterprise software. But every paradigm shift brings new business realities, and we must be careful not to thoughtlessly apply lessons from other technologies to the new world.

When cloud computing proved itself, it made sense to transfer nearly all enterprise compute. Our software stacks were mature, they weren’t compute-bound and the user experience was horizontally scalable.

But AI technology has a very different profile to “traditional computing”. AI compute is expensive, latency is high and custom user experiences are hard to scale. In some ways, this “new kind of computer” is more like Video Game software (and not just because they both use GPUs). Cloud Gaming has its place, but most customers prefer to run their games on their own hardware, where they can control cost and performance.

Of course, AWS, Azure and Nvidia want you to do AI in their clouds, under the same old model. They’re salivating at the potential margins! But I wonder if, in a world where customers are have powerful devices in their pockets, capable of running LLMs locally, it won’t be more efficient to push compute past the edge, into the device.

As an example, I recently helped an enterprise client to deploy an LLM stack. They had previously moved all of their software development into the cloud, using AWS development environments. At first, it seemed to make sense to develop the LLM product in the same way. But we soon realised that the cost of iterating and running tests against staging LLMs in the cloud was unnecessary. Every developer had a laptop capable of running open source models like Llama3, so we built an offline test-harness to take advantage of that under-utilised in-office compute.

Nobody wants to hear that we’re going back to on-prem! But I wouldn’t be surprised if the technical realities of AI — cost, performance, data security — mean large changes to the cloud computing model we’re used to, likely towards on-device.