Software-defined storage finds its feet

Software-defined storage (SDS) is a very promising way to virtualize and scale storage services in a cloud or cluster. Utilizing storage services embedded in containers or VMs, functionality is no longer tied to the compute capability in a storage appliance. Instead multiple service instances can be created in the server virtual space as needed, allowing for great flexibility and much more powerful storage solutions.

As with many early-stage technologies, SDS is hard to install and difficult to manage. We also face the reality that we are dealing with hardware and tuning and monitoring of operations are a major factor in long-term success.

To address the first problems, Red Hat has created a product, Red Hat Storage One, which greatly improves integration of SDS modules using an Ansible-based quick install tool. This allows stable zero-day configurations and rapid scaling and must count as a major milestone in the storage industry.

The solution has limitations, of course. Node count is limited to 24 max., while solutions in the near-term will be offered only bundled with fixed hardware solutions from major vendors.  This is becoming a common practice for major new initiatives … Nutanix entered the market bundled, for example. I would expect these limitations to disappear within a year or so, as maturity and experience build.

Given that the code-base for Storage One is open-source, I sense that Red Hat expects 3rd-party software vendors to attach to the framework, achieving the SDS aim of more flexible mashups and lowered costs for solutions. It doesn’t yet address the issues of API interoperability between services, but recent experience is that leading choices appear and we end up with self-standardization, if not a formal process.

Red Hat is building on Gluster for a storage system, but Ceph is also containerisable and we can expect an expansion to object storage to be fairly near-term. In the process, missing services such as compression, deduplication and encryption can be designed into the data service sequences to build very powerful solutions.

Red Hat Storage One will accelerate SDS adoption significantly and this will change storage forever, much as software-defined networking altered the direction and dynamics of the switching and routing segment. It’s a commendable effort that all IT shops should understand.

RHS1 is not without limitations, however. Apart from the node number and the fact that it only comes bundled, there are key areas not covered (yet). We live in an evolving storage space, where new layers of device types such as NVDIMMs and byte-addressable persistent memory (3D X-Point) are competing already with traditional drive storage, where ultrafast SSDs are demanding much flatter fie systems and where memory usage optimization is paramount to reaching maximum performance.

Tiering technology, such as that pioneered by Enmotus, is essential to overcome the limitations in size for NVDIMMs that make moving of cold data out and anticipating what will hot data an issue for OS or driver writers. 5 minutes of study will make it clear that manual data placement is a no-go in such a complex high-speed environment. The Enmotus approach, embedded in a number of OEM offerings, provides a tool for rapidly moving data up and down the storage pyramid. 

Beyond tiering, the SDS stack has to be conscious of the cluster in which it exists and how data is moved, stored and protected. Replication is a simple example of a common need. In thinking this through, it isn’t clear where the boundaries between OS functionality and SDS are. Many OS functions could migrate into the SDS pool. I suspect we’ll reset bounds to limit OS functions to anything that enters kernel mode.

This would involve a systems architecture where storage services arise and die all the time, which is fine in a perfect word, but not one where things break.  This is the second major area the RHS1 is yet to cover. Very complex, automated systems need automated monitoring and problem correction. Just like the freeway systems at rush hour, prompt recognition of problems and rapid reaction keep things moving.

Automated dataflow monitoring and reactions to issues are needed to make RHS1 truly an SDS master-solution. Agents within services reporting health and performance could feed an analytics engine to evoke rapid response. Eventually AI will tackle the job and move QOS to a new level altogether.

Enmotus is already looking to the monitoring of service data and the analytics issues. This promises to be a key part of systems future.