AI SoCs : the critical part in enabling the future of AI

Mark Hayter, Chief Strategy Officer and Co-founder, Rivos Inc.

I was in Taiwan last week at the RISC-V Taipei day. This high energy event brought together the Taiwan-based ecosystem who are focused on the evolution of RISC-V. The theme this year was focusing on AI SoCs and over the day there was a general agreement about the applicability of RISC-V for these.

As an Open Standard, covering both ISA and system (non-ISA) components, RISC-V gives a stable base for software development. The permitted device specific extensions give the flexibility to provide optimized device drivers and libraries for a particular SoC. Within the guardrails provided by the standard, it was clear each vendor was innovating in different ways, for different targets.

For anyone who is not familiar with Rivos, we are a 3.5 years old company bringing RISC-V to the data center for LLM and Data Analytic workloads. At the Taipei event, I presented a keynote that discussed our workload-defined-hardware approach to building SoCs.

The highest priority is to consider the needs of the software. In the current environment, AI SoCs must adapt to the continuous flow of new models, new tooling and new stacks. Even the entrenched Transformer architecture is being threatened by State Space approaches, such as mamba. It is also important to respect the investment in older models being used in production. Users expect new hardware to support the past, present and future. This gives rise to the Rivos mantra “Recompile not redesign”. Moving to Rivos hardware should be no more difficult than moving between GPU generations.

In addition, the needs of the system must be considered. The matrix multiply performance is important, but is far from the only thing. Models, and associated vector databases for RAG, are getting huge so the memory system capacity and bandwidth must be considered. Chips work together in nodes, nodes in racks and racks in data centers, so communication protocols and bandwidth are critical. And soon the CFO will be asking about the costs of power and cooling, as well as the value to the organization.

One of the key benefits of a community is being able to partner with other hardware IP providers to utilize their expertise, whilst we focus on our differentiators. We recently jointly announced that Rivos has licensed the Andes NX45 RISC-V core for use as a control processor in our SoCs. Dr Charlie Su, President & CTO of Andes, revealed this in his keynote and associated press release and I expanded on it when discussing how our SoCs can be RISC-V “all the way down”, making use of the full-range of the RISC-V ecosystem. We are building our own high-performance server-class 64-bit OoO RISC-V CPU; licensing the Andes 64-bit RISC-V control core; using the lowRISC supported open-source Ibex RISC-V microcontroller; and the boot process starts from the lowRISC OpenTitan Root-of-Trust with the secure version of Ibex.

I also got to spend time with the growing team at the Rivos Taiwan office near the High Speed Rail station in Hsinchu. It was good to see the collaboration as the team worked to finalize the first implementation of our SoC, combining Rivos’ high-performance RISC-V CPUs and Data Parallel Accelerators. Rivos is continuing to grow the team in Taiwan and across the globe, see our job listings: https://jobs.lever.co/rivosinc/

As AI and data analytics needs continue to evolve, the scalable software-driven approach that Rivos is adopting will enable a range of performance points, through efficient solutions that have been designed with these workloads in mind. Having a diversity of AI SoCs and systems is a critical part of the future of AI.