Tokens Per Second is Not All You Need

The growing battle around high tokens per second (t/s) continues to heat up on X. But for the enterprise, is t/s the only metric that matters? We believe that first token time is equally, if not more crucial for complex, document-heavy, and agentic use cases.

The Hype and Limitations of Tokens per Second

Pursuing high t/s, some AI vendors cache model weights on SRAM and employ numerous chips in their inference systems. While this yields impressive decoding speeds, it brings significant drawbacks - learn more.

In real-world scenarios, these SRAM-based inference engines can take over 1.4 seconds to generate the first token for a 3K token input (see table), which makes the first token latency 85% of total inference time. For enterprises dealing with document intelligence, search, retrieval, and retrieval augmented generation, that's simply not fast enough.

In a recent Artificial Analysis AI report, a 900 tokens/s solution actually performs worse than the 200-300 tokens/s solution even for a medium input of 1000 tokens and output of 100 tokens simply simply because of the large TTFT.?

The SambaNova Solution: Designed for Enterprise Needs

We've taken a different approach to address the unique challenges of enterprise LLM inference. Our goal is to deliver the fast, flexible, and scalable inference. Our unique architecture delivers the best of two worlds:

• Blazing 0.2s 1st token
• Fastest 8-socket 450 t/s gen speed

The secret? RDU’s unique 3-tier memory hierarchy with reconfigurable dataflow:

• Larger SRAM than typical accelerators
• HBM for fast access
• DRAM for spillover params as model sizes get bigger
• Flexible dataflow optimize for compute intensive and memory intensive graphs

This setup also enables:

• Stellar perf with minimal sockets
• Orchestration of expert models beyond 8B

SambaNova's solution also:

• Avoids 8-bit quantization pitfalls
• Scales seamlessly with batch size

Conclusion

At SambaNova, first token time is equally - if not more - crucial for the complex, document-heavy, and agentic use cases that businesses rely on.

To explore why t/s doesn't paint the full picture of enterprise LLM inference performance, read our latest blog.