Scientific Data Compression with ZFP

Scientific applications can produce, and increasingly produce, astounding amounts of data, from Gigabytes, to Terabytes and even Petabytes per applications. Typical scientific applications rely on floating points to encode information about the real world, which provide a computationally efficient way to represent approximately continuous values.

When receiving floating point data directly, e.g. sensor data or computer simulation data, the data can be generated in fairly fine grids, using fp32 or fp64, occupying huge amounts of space, possibly several 100s of GB or TBs. The reason for this is that 3D grids can have a fairly large number of points in each direction.

Take, for example, the case of spatially and temporally turbulence simulations. In order to fully resolve all scales, it is necessary that the smallest scales, represented by the grid spacing, to be of the order of the Kolmogorov scale. This is achieved in the so called Direct Numerical Simulations, as the ones made available by the John Hopkins Turbulence Databases. One such dataset is the Forced Isotropic Turbulence, which has a Reynolds number of ~ 433 (using Taylor Scale).

To perform this simulation, the group used a grid of 1024 x 1024 x 1024 points, which is over 1 billion points. For a reference, this means that each field, such as velocity in the x direction or pressure, in the simulation consumes 4 GB of RAM, and just holding a single time-step can consume 16 GB of RAM with single precision FP32 floating point. Actually running the simulation consumes considerably more memory, potentially more than 50 GB of RAM.

More up to date simulation datasets, also made available by John Hopkins, are the forced Isotropic Turbulence 40963 and 81923 datasets, which have more than 65 billion and 500 billion points respectively. Just holding a single field in memory of the 40963 can consume almost 280 GB of RAM, which is just astounding. The case for the 81923 dataset is even more severe, with a single field consuming upwards of 2 TB, and the whole dataset requiring almost 9 TB per time-step.

Dealing with such massive simulations is very difficult, since not only holding, but also moving such massive amounts of data is a very challenging endeavor. If one needs to reduce the storage footprint of a dataset like this, decimation and quantization like these are available, but they might introduce artifacts. Essentially: simply discarding trailing bits or down-sampling are very aggressive strategies.

One way to overcome these limitations is to use specialized compression algorithms designed to deal with floating point data.

Scientific data poses particular challenges for traditional compression algorithms for computer science. Data is not immediately sparse, and truncation strategies that would be acceptable for applications in video or computer vision may introduce errors that are unacceptable for scientific applications.

This gave rise to a new class of of algorithms designed to efficiently compress floating point data, while also providing bounds on the errors incurred during the compression process. These algorithms comprise a relatively new class of algorithms, like ZFP and SZ, which aimed to create lossy, error bounded, floating point compression, something that was unavailable till fairly recently.

The ZFP library, developed by Peter Lindstrom, compress floating point data by compressing floating point blocks of data, i.e. blocks of 4^d floats taken together, similarly to what a JPEG algorithm would do, but in a cleverer way. It aims compress efficiently by exploiting smoothness that is present in real fields, which appear in sensor and simulation data.

It creates a wavelet representation of the block, which it then re-orders in order to get the lowest frequency first, and then creates a bitstream which represents the data in increasing levels of resolution. In truncating the bitstream, one is able to create the compression.

I have successfully used this library in several projects I participated in, and it provides a surprising amount of compression, with very little degradation of the data, especially when the data is well resolved.

The first time I was presented to this library was through the work of Stephen Hamilton and others, in "Extreme Event Analysis in Next Generation Simulation Architectures", in which the authors compress data exactly this kind of turbulence simulation data successfully, for later analysis.

ZFP is a very efficient and flexible library, having both CPU and GPU execution modes. In CPU execution, it is able to achieve up to 2 GB/s per core of compression, and can use GPU compression up to 800 GB/s. I have used their CUDA implementation for GPU execution, but they also provide HIP.

Additionally, the ZFP libary also provides an execution mode that functions as array sub-library, similar to C++ std::vector classes. In this mode, the vector looks like a normal std::vector, but with built-in compression. They have demonstrated its use for solving partial differential equations with reduced memory consumption, which in itself can be a very attractive proposition for many applications.

Fields are not the only kind of data that can be compressed with this kind of library. Di et al. describe the use of the SZ library for compressing particle data generated by the N-Body cosmology code HACC. They were able to efficiently compress dozens of Petabytes, corresponding to the data of 3.5 trillion particles.

Data Driven Science and Engineering are ever more present in the daily workflow of many professionals. Dealing with the associated data deluge requires knowing techniques to efficiently lowering the storage and memory footprints without excessively damaging the information therein. This kind of libraries can be an avenue to do so for many kinds of applications involving floating point data.

References:

zfp: Compressed Floating-Point and Integer Arrays

Peter Lindstrom, "Fixed-Rate Compressed Floating-Point Arrays" IEEE Transactions on Visualization and Computer Graphics, 20(12): 2674–2683, December 2014, doi:10.1109/TVCG.2014.2346458

A. Fox, J. Diffenderfer, J. Hittinger, G. Sanders, and P. Lindstrom", Stability Analysis of Inline ZFP Compression for Floating-Point Data in Iterative Methods" SIAM Journal on Scientific Computing, 42(5):A2701-A2730, September 2020. doi:10.1137/19M126904X

Sheng Di, Franck Cappello, Fast Error-bounded Lossy HPC Data Compression with SZ, in IEEE International Parallel and Distributed Processing Symposium (IPDPS 2016), Chicago, IL, USA, 2016.

Hamilton, S. et al. (2017). Extreme Event Analysis in Next Generation Simulation Architectures. In: Kunkel, J.M., Yokota, R., Balaji, P., Keyes, D. (eds) High Performance Computing. ISC High Performance 2017. Lecture Notes in Computer Science, vol 10266. Springer, Cham. https://doi.org/10.1007/978-3-319-58667-0_15

Zeyen, M., Ahrens, J., Hagen, H., Heitmann, K., & Habib, S. (2017). Cosmological particle data compression in practice. In Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization (ISAV'17) (pp. 12–16). Association for Computing Machinery. https://doi.org/10.1145/3144769.3144776

S. Di, D. Tao, X. Liang and F. Cappello, "Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound," in IEEE Transactions on Parallel and Distributed Systems_, vol. 30, no. 2, pp. 331-345, 1 Feb. 2019, doi: 10.1109/TPDS.2018.2859932