AI Newsletter 01.16.2024

Welcome to the first edition of the AI performance newsletter for 2024. In this edition, we look at summaries of two interesting techniques that both try to make better use of system resources for LLMs.

1) LLM in a flash - The key innovation discussed in the attached paper is using flash memory to store large language model (LLM) parameters and selectively loading them into DRAM during inference. This allows running models that are larger than the available DRAM capacity.

2) PowerInfer - Exploiting the high locality and power-law distribution of neuron activations in LLMs to design a GPU-CPU hybrid inference engine. Frequently activated "hot" neurons are preloaded onto the GPU, while less frequently activated "cold" neurons are computed on the CPU. This significantly reduces GPU memory demands.?

And finally this blog post that starts from first principles on how to make LLMs go fast.?

Other notables readings?

Ten Noteworthy AI Research Papers of 2023?

2023, year of open LLMs

Detailed AI generated summaries are below. Happy reading, and if you have any feedback please send it my way.

Apple Develops Breakthrough Method for Running LLMs on iPhones / LLM in a flash: Efficient Large Language Model Inference with Limited Memory

The key innovation discussed in the attached paper is using flash memory to store large language model (LLM) parameters and selectively loading them into DRAM during inference. This allows running models that are larger than the available DRAM capacity.?

Specifically, the paper proposes two main techniques:

1. Windowing: Only loading LLM parameters for the past few tokens, reusing activations from recently computed tokens. This reduces the amount of data transferred from flash.?

2. Row-column bundling: Storing concatenated rows and columns of the LLM matrices to read larger contiguous chunks from flash memory, increasing throughput.

Together, these methods enable running LLMs up to 2x larger than DRAM capacity. They speed up inference by 4-5x on CPU and 20-25x on GPU compared to naively loading from flash.

The innovation is important for AI performance because larger LLMs tend to have better capabilities. But model size has been limited by device DRAM capacity. By storing models in flash and loading subsets into DRAM, much larger models can be run efficiently. This allows more powerful AI on resource-constrained devices.

GitHub - SJTU-IPADS/PowerInfer: High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

The key innovation in this paper is a system called PowerInfer for fast large language model (LLM) inference on a personal computer with a consumer-grade GPU. The main innovations of PowerInfer are:

1. Exploiting the high locality and power-law distribution of neuron activations in LLMs to design a GPU-CPU hybrid inference engine. Frequently activated "hot" neurons are preloaded onto the GPU, while less frequently activated "cold" neurons are computed on the CPU. This significantly reduces GPU memory demands.

2. Using adaptive predictors to forecast neuron activation, allowing the system to skip computing inactive neurons. This reduces computational load while maintaining accuracy.

3. Introducing neuron-aware sparse operators that focus on individual neurons within matrices rather than entire matrices. This avoids overhead from tracking and converting sparse formats.

4. Formulating an optimized neuron placement policy using integer linear programming to assign neurons between the GPU and CPU to maximize GPU impact while balancing workloads.

The main relevance of PowerInfer to AI performance is enabling fast and affordable deployment of large language models on consumer-grade hardware like personal computers. By effectively utilizing properties like sparsity and locality in LLMs, PowerInfer can accelerate inference by up to 11.69x compared to existing systems based on model compression or layer-wise offloading. This helps make high-performance LLMs accessible to broader applications and users. Overall, the innovations in PowerInfer advance the efficient delivery of AI capabilities on more widely available computing platforms.

How to make LLMs go fast

Here is a summary of the key ideas in the text about how to make large language models faster:

1. There are two main reasons plain autoregressive generation is slow: the algorithmic cost grows with the number of tokens generated, and large model weights do not fit well in hardware caches.

2. Techniques like batching, continuous batching, key-value caching, speculative decoding with a smaller model, threshold decoding, paged attention, and guided generation can help optimize speed.

3. Architectural improvements like multi-query attention, sparse attention patterns, and non-Transformer models can increase speed. Quantization to smaller number formats like fp16 helps as well.

4. Optimizations target metrics like time-to-first-token, throughput, latency, and hardware utilization. The optimal balance depends on the application.

5. Inference speed improvements enable the deployment of larger, more capable models while staying within hardware and cost constraints. Faster decoding directly translates to better end user experiences.

The text covers a wide range of software and hardware techniques to accelerate transformer-based language models. Speeding up these very large neural networks is essential for productizing and deploying AI systems that can understand language and generate human-like text.