IvalDB - simple and tiny flash database for embedded devices
Embedded systems are a special, separate niche of small computers on a chip (microcontrollers). They consume very little power, have a small size and limited capabilities, but they can control other devices (relays, motors, LEDs), so they are widely used in various devices from irons, drones, video surveillance systems, toys to life support systems and prosthetics. ?
?
One of the important functions of such systems is the permanent storage of information during periods when the power of the device is turned off (e.g., during battery replacement, emergency or scheduled power outage).?
?
Each time the device is turned on, it is necessary to constantly restore the previously saved state in order to continue the interrupted work. What kind of data is usually worth storing in embedded systems? Often, it is configuration information or calibration data that speeds up the system or increases its accuracy. This data can have different size and different structure. Often, they are related to each other (must be stored as part of a certain complex structure) and must be changed synchronously within a single transaction - either all the combined data is stored, or nothing is stored.?
?
However, the first problem arises here. The fact is that often such systems are part of larger systems and can suddenly turn on and off without warning, even at the moment of writing data to the media. This can damage the structure of the persistent database and requires the implementation of algorithms for reliable recovery of the structure and integrity of the stored data. ?
?
The second problem is the features of the "storage medium" in embedded systems. The fact is that flash memory chips are usually used for permanent data storage in embedded systems. This is a cheap and quite reliable solution. However, their use has a number of limitations that should be overcome. For example, modern flash chips have different write/read speeds, which sometimes differ from each other tenfold, and allow only a limited number of write/erase cycles, after which certain memory cells in the chip "break" and stop changing. ?
?
The third problem is to maintain autonomy. Embedded systems are often a "black box" that should work on a "plug and forget" principle. Therefore, such a system should solve all its problems independently, without human intervention. ?
?And the fourth problem is a limited amount of resources.?
?Due to limited resources, the amount of information that can be stored and processed by such a micro-computer is small. As we have found, this often requires a simple and compact embedded database to store small amounts of information: from one to several hundred bytes maximum. Usually, it is necessary to store only a few or dozens of structures ranging in size from 1 byte to several kilobytes, and their total size does not exceed tens of kilobytes. All this must work in a rather limited RAM. This imposes certain requirements on the interface of the system for storing and reproducing previously stored data. The API should be simple and concise, the database should be (if necessary) multithreaded, be able to cache (temporarily store in the controller's memory) the most important data, be able to recover from failures and allow the system to "survive" in the most adverse circumstances, such as the appearance of bad bits in the flash chip or shutdown during data recording. An important condition is the ability to create several such databases in memory on one or different flash chips.?
领英推荐
?
To cope with all these tasks, we have developed an embedded cross-platform thread-safe index-value-based flash database (IvalDB).?
?
Why, when there are already dozens of projects for persistent data storage for embedded systems in the world, there is a need for a new library for persistent data storage???
?
After analyzing many other projects, we concluded that most of them did not simultaneously cover all our requirements, which often seemed mutually exclusive. We needed a cross-platform library that should be small, fast, easy to use, multi-threaded, transaction-based, with the ability to autonomously recover from failures and maximize survival in case of degradation of individual bytes of flash chips.?
?The IvalDB interface consists of only 11 methods that allow you to create a database structure on a flash device and implement a full CRUD (Create-Read-Update-Delete) model. Data recovery from failures, support for multi-threading, transactional data is completely autonomous and does not require human intervention.?
?For the library to work in RAM, you need to allocate a space of at least 8.5Kb (however, in future versions this number may be reduced to 1-2Kb). The size of a single stored data structure can reach 3000 bytes (up to 4096 bytes in version 2). Only eight to sixteen bytes are required to store one to eight bytes on the flash device, and with a larger data size only a small service block with a constant size of 16 bytes is written along with it. All data is protected by CRC that allows you to quickly detect failures in the read data and either report a read error or try to restore the previous version of the data (at the request of the programmer).?
As part of the implementation of the IvalDB project, 4 libraries were implemented, which were created to consider cross-platforming and flexibility. In total, the following libraries and subprojects were developed:?
The combination of all four libraries (Note: Locker_lib is not used to build single-thread applications) makes it possible to cover many use-cases using different controllers, flash chips and frameworks/RTOS.??
if you would like to know more please contact us