The NVDIMM Challenge

NVDIMM is not new. They’ve been around for at least 6 years, but it’s taken a more holistic view of system architecture, coupled with very fast NVMe SSDs to make the case for an NVDIMM storage layer loud enough to move to the mainstream.

NVDIMMs come in several flavors of memory and flash configuration. With on-board DRAM buffers, most models can accept both block I/O writes (the traditional storage I/O scheme) and byte-addressable transfers. Reality today is that the Holy Grail of being able to write a byte of data directly from a single CPU-memory instruction is still a future feature, but block I/O is a mature access mode.

Done properly, 4K block transfers to NVDIMM can be much faster than old-style RAMDisk. This involves superseding the industry standard memcopy() routine with a highly tuned tool that uses SIMD extended data instructions, for example. This can crank the memory-mapped copy operation up by as much as 1000X, making an overall NVDIMM version of RAMDisk a truly powerful tool.

That speed boost, which is way higher than NVMe talking to SSDs can achieve, necessitates changes in the way the I/O operation is managed. Much of the I/O operation, such as address translation, is best done in the CPU and not in the NVDIMM, which has only a small-scale processor. This is a very positive thing, since it opens up the opportunity to apply much more sophisticated performance tuning and error management, while adding data compression options to maximize effective capacity.

Let’s put this into the real world. An app quickly wants to store data into a persistent storage location. For speed, the data is written from DRAM to NVDIMM. Since NVDIMM memory is still in the low 10’s of gigabytes, some of the data should be compressed before writing to the NVDIMM, while other more transient data may just be left raw. Data type influences the compression decision, too. Media files just aren’t worth compressing, for instance.

After a while, the NVDIMM fills up … this happens much faster than with an SSD … bringing the question of caching the file as a front-end to a much larger SSD space or dedicating more of the NVDIMM to that file at the expense of other files. It is also likely that some files not in current use will be paged out to SSD. All of this requires a metrics-driven near-real-time analysis of data optimization. Manual methods, and admin guesses, won’t fly at the dynamism levels the NVDIMM-NVMe combination brings.

Graphic above shows a Virtual SSD made from Micron NVDIMM and NVMe
drives.  Hot data dynamically promoted to the NVDIMM in real-time.
Virtual SSD created using Enmotus Virtual SSD Software

The answer is to add storage analytics tools to the equation. These tools gather much more information about what’s happening in the system than traditional approaches. While critical event detection and correction is still a key feature, inferential analysis on the “big data” gathered on storage flows allows more long-term data management, path management through the virtual networks and instance creation and deletion in the software-defined storage ecosystem.

Better traffic flows and data placement will have a significant impact on overall cluster productivity. The approach will work with traditional-style networked storage, but one might expect a much larger impact in hyper-converged systems with virtualized storage data services. In other words, storage analytics fit in really well with the HCI/SDS world that looks to be the future of cluster design.

From an admin perspective, storage analytics reduce the control interface for the storage pool considerably, in the sense that much of what occurs operationally is handled by automated orchestration. The tools will provide substantial guidance on how to best optimize the cluster and, more importantly, provide insights into rewriting and tuning apps for efficiency. This will have a substantial financial impact on the datacenter, as there is plenty of evidence that writing code aiming to optimize the efficiency of compute resources will move up the priority list in the near future.

When the “Next Wave” of NVDIMM software hits us, in 2019, byte-addressability will be a massive challenge to app writers. Being able to save a byte or word or field directly without pausing to enter the I/O stack is a massive step forward in performance potential, but there are major changes to operating systems, compilers and link editors before app changes can really begin. These changes take time to coalesce in a standards-driven industry and are no small technical challenge.

Once we byte-address NVDIMM, tuning becomes even more complex and the guidance of use-case driven storage analytics is critical in making the best of a potentially huge boost in cluster throughput.