When is it ok to lie to your DUT? A risc-v example

As a simple rule of thumb: when it helps you gain speed/coverage and when you don’t end up getting tangled in your own lies, which goes down to keeping them simple and at least somewhat based on reality reality. 

Let’s use Google’s riscv-dv, an SV/UVM based RISC-V program generator as a demo vehicle. The flow for using google’s riscv-dv taken from this crystal clear slide-set, is shown in the diagram below. Riscv-dv generates a constrained-random, 100% legal, meaningless program that can be compiled and linked by the standard toolchain and run by an ISS, RTL, emulation, FPGA, a real board, and any other platform on the planet. Beyond being platform agnostic, this approach has other advantages: it can use any ISS as the reference model with no modifications, it can use developer tools such as debuggers seamlessly, and it can be run once on a commercial simulator to generate 1000’s of programs, and then run all simulation runs on free open-source tools, such as verilator, which likely resonates well with the RISC-V community.

For argument’s sake, however, let’s contrast this straightforward super-useful solution with the proposal shown in the diagram below. With this second testbench architecture, the simulator generates instructions on demand at runtime, and feeds them into the design via the instruction-cache interface (or instruction memory if the cache is not available). Compared to riscv-dv, this solution seems to have only dis-advantages to offer: it can run only on simulation, and only on commercial simulators equipped with a constraint solver; it doesn’t allow use of the standard toolchain or standard debugger; and it requires hacking an ISS to make it run one instruction at a time, instead of a full executable.

The only things that makes up for some of those disadvantages is that this architecture supports lying. Since the generator output doesn’t have to be a legal problem, many constraints are thrown out the window. For example, we can jump to the exact same address, and each time have some other instruction fetched. We can have loops, and recursions of any depth we choose. We can create instruction sequences that would be otherwise optimized away by a compiler. And, and that’s probably the best part, we can do state-aware generation, peaking at architectural register values and even at internal design state to make the next instruction hit a corner case. Using that we can make different addressing modes hit the max/min values of their address ranges, or even the exact same address, and arithmetic operations overflow at will. If you’re looking to torture your architecture to the max, what you’ve got here is the iron-maiden testbench you need. 

This kind of testing infrastructure does require significant work: we need to have an ISS which runs one command at a time, we need to have an instruction generator that generates instruction bits and bytes rather than assembly code, and a cache agent that models the cache, a prefetch buffer if one exists, and the rest of the memory system behind them. Of this list, the hardest and most fun part is the generator, though it is likely somewhat easier and less fun than a full program generator in the riscv-dv style. The least fun part is stubbing out the cache and memory system, which can range from easy (for example here in swerv) to a bit more complicated (for example here in ariane), and maybe to hard in some other designs. However, it is nothing that will stop determined verification engineers on the way to their coverage.

Though discussed here for a RISC-V example, this kind of architecture applies to many different types of designs, most notably in the networking/routing domain, where you would often generate a packet together with the specific configuration that the DUT will read from memory when this packet is processed, and then push the packet on the interface, and the configuration into a stubbed out memory, just like the one above. At the price of limited whitebox-ness, this "memory implant" design pattern, allows you to hit corner cases and coverage faster. The choice between lying and sticking to the rules, should really be decided on a case by case basis, but it is important to remember that you can and should choose.

(Thanks a lot to Matthew Ballance, an ex-colleague, and 2nd prize winner of the RISC-V soft CPU context for taking the time to review this post)