File-backed allocator for STL containers

The STL is renowned for its modular design, allowing universal algorithms to be applied to a wide range of containers. Even the behaviour of a container is not set in stone and can be customized, including its memory management.

While there are many different types of allocators, let's focus on one in particular—memory allocation from a file. Sound crazy? Welcome to C++.

The concept is straightforward: map a file to the process memory and allocate memory blocks from that file instead of from the heap. As a result, if an element of the container is modified, those changes are written directly to disk. The overall picture looks like this:

But first of all, I have to say that this option is useful only for a limited number of applications where the persistence of objects is required.

I was able to find an allocator called mmap-allocator, which I’ll use for demonstration purposes. The code is a bit outdated and likely has some bugs, but it serves well for illustrating the concept.

In my example, I declare a vector to store a user-defined type and use the mmap-allocator with it. Then, I push several elements into the container, and we’ll observe what happens.

The data stored in the container should be a POD (Plain Old Data) object, without any references or pointers inside, so a vector of string is not an option here:

Then, the allocator can be created along with the vector:

After the program execution, the resulting file test.storage contains the following content:

So all data has been written to the disk without any extra modifications to the container's behavior.

Of course, in such a trivial demonstration, there are some blind spots. For example, data cannot be read back when the program starts up. This is a reasonable concern, and solving it requires additional logic, such as a factory method to construct and initialize the vector with the content of the mapped file.

Let's sum up:

• Using a file to allocate memory can be implemented for STL containers without modifying them.
• It may be necessary to store additional meta-information about the current status of the container, such as free and occupied blocks and the current size.
• This approach is suitable for a limited number of applications.
• There is a strong limitation on data types: they should be POD (Plain Old Data) types without references or pointers.
• Constructing a container from an existing file requires extra logic and meta-data.