Paper Review: Scalable MatMul-free Language Modeling

MatMul, shorthand in the AI world for matrix multiplication, is a backbone operation of AI models everywhere and often one that is easily dispatched down to acceleration hardware has been removed from a model as shown in "Scalable MatMul-free Language Modeling" by Zhu et al.

As the paper notes, MatMuls dominate the overall computational power of LLMs, often reinforcing a larger amount of memory and computational usage than without the MatMul free variety.

By removing these operations, the paper showed incredible speedups in both memory and compute usage.

Why is this interesting (and why should we care)?

We've seen in many places that AI and environmental sustainability are at odds:

with one example being the growing demand on the electric grid.

BBC quantified this for us late last month as data center usage going from 460TW in 2022 to 1000TW in 2026:

Imagine accelerating model training by 25.6% and reducing memory consumption by 61.0% over unoptimized GPU baseline. Similarly, employing lower bit-optimized CUDA kernels, inference speed is increased by 4.57 times and memory usage reduced by a factor of 10 scaled up to a 13B parameter model.

Results from Section 5 "FPGA Implementation and Results" shows performance as almost indistinguishable from an idle run when handled in the experiment:

which when deployed at scale could make a huge dent in the overall computational needs of an AI deployment, thus rendering significantly less electricity usage.

Take this at face value (I'm not a computer or power engineer), but my suspicion would hazard a solid guess of 30-40% reduction in overall usage from data center demand on the electrical grid at scale.

Enabling sustainability through innovation

When properly handled, innovations like this can make a huge dent in core problems that occur within areas like climate change. By reducing power consumption, we protect natural resources and allow for a more sustainable world to build itself up without teetering into unhealthy environmental dynamics.