Increased complexity, competition, and big prices are on the way of training machines.

The fundamental assumption of the computing industry is that number-crunching gets cheaper all the time. Moore’s law, the industry’s master metronome, predicts that the number of components that can be squeezed onto a microchip of a given size — and thus, loosely, the amount of computational power available at a given cost — doubles every two years. For many comparatively simple AI applications, that means that the cost of training a computer is falling. A combination of ballooning complexity and competition means costs at the cutting edge are rising sharply.

Bert, an AI language model built by Google in 2018 and used in the firm’s search engine. It had more than 350m internal parameters and a prodigious appetite for data. It was trained using 3.3bn words of text culled mostly from Wikipedia, an online encyclopedia. 

But these days Wikipedia is not such a large data-set. If you can train a system on 30bn words it’s going to perform better than one trained on 3bn. And more data means more computing power to crunch it all.

A bit cloudy

Facebook, which turned a profit of $18.5bn in 2019, can afford those bills. Those less flush with cash are feeling the pinch. But AI startups rent their processing power from cloud-computing firms like Amazon and Microsoft. The resulting bills—sometimes 25% of revenue or more—are one reason, it says, that ai startups may make for less attractive investments than old-style software companies. 

The growing demand for computing power has fuelled a boom in chip design and specialized devices that can perform the calculations used in ai efficiently. The first wave of specialist chips were graphics processing units (GPUS), designed in the 1990s to boost video-game graphics. As luck would have it, GPUS are also fairly well-suited to the sort of mathematics found in ai.

Quantum solutions and neuromantics

Other researchers are therefore looking at more exotic ideas. One is quantum computing, which uses the counter-intuitive properties of quantum mechanics to provide big speed-ups for some sorts of computation. One way to think about machine learning is as an optimization problem, in which a computer is trying to make trade-offs between millions of variables to arrive at a solution that minimizes as many as possible. A quantum-computing technique called Grover’s algorithm offers big potential speed-ups.

Another idea is to take inspiration from biology, which proves that current brute-force approaches are not the only way. Cerebra's chips consume around 15kw when running flat-out, enough to power dozens of houses (an equivalent number of GPUs consumes many times more). A human brain, by contrast, uses about 20w of energy—about a thousandth as much—and is in many ways cleverer than its silicon counterpart. Firms such as Intel and IBM are therefore investigating “neuromorphic” chips, which contain components designed to mimic more closely the electrical behavior of the neurons that make up biological brains.

For now, though, all that is far off. Quantum computers are relatively well-understood in theory, but despite billions of dollars in funding from tech giants such as Google, Microsoft, and IBM, actually building them remains an engineering challenge. Neuromorphic chips have been built with existing technologies, but their designers are hamstrung by the fact that neuroscientists still do not understand what exactly brains do, or how they do it.

That means that, for the foreseeable future, ai researchers will have to squeeze every drop of performance from existing computing technologies.

The Economist

Kind Regards,

Denis