Data storage to external flash memory in embedded devices

Embedded devices require one form or another of data storage. Let us consider a hypothetical consumer device that reads the CO2 levels of a specialized room. It is required to send these values over the network to some storage server for remote monitoring. Now, what typically happens is that once the data leaves the device, it is no longer under your control. It can be lost anywhere over the network during transmission, relies on 3rd party servers that you actually know nothing about, even though they are built for maximum reliability.??

To keep the data in your control, we need to store the data in some sort of non-volatile memory for later retrieval, or retrieval any time we need it. There are a myriad ways to do the storage, from SD cards to external flash chips. This article will discuss basic usage of external flash memories.

Why flash storage

Now why should you choose flash memory storage? First of all flash memory is non-volatile, meaning it can retain its contents even after power is lost. Most flash memories use floating-gate technology, which allows the data retaining mechanism.?

Secondly, flash memories can store huge amounts of data, while at the same time preserving a small form factor, which is ideal for space-constrained devices. This also means they have a very high data density.

Third, flash memories are performant, which means they can perform R/W operations fast enough for high performance applications.

Memory budget and memory choice

So, how do you calculate how much and which memory you need??

To determine how much memory we need, we have to carry out a memory budget analysis.?

The following are some critical considerations when doing a memory budget:

a) Consider the power consumption (current and voltage)

Most of these memories use 3V, 2,5V, and 1.8V, typical logic levels. Consider the image from the Winbond product manual below: (Link in the references section)

If your system is using a 3.3 voltage level, then select a memory that can handle 3.3V. It’s a no-brainer.?

b) Memory Endurance - Total number of memory R/W cycles over its lifetime

How much R/W operations can your memory handle over its lifetime? This can be mostly retrieved from the datasheet and/or product briefs.?

For the Winbond memories, the typical write cycles are 100,000.?

More industrial grade flash memory, like the FM25V10-G from Infineon, have a R/W endurance of a whopping 100 trillion cycles! , trading off for low memory size.

c) How much data are you going to be writing

Now this is crucial. Let's consider this scenario. You have a device that logs CO2 levels, temperature, battery voltage and let’s say the device state. Construct a table that show their respective data sizes:

Now assume that you are sampling every 10 seconds, which gives you a sampling rate of 0.1 Hz, if your device is on for 8 hrs per day, then the total size of data you collect in a day will be given by:

Data size = packet_size x total_samples

packet_size = 13 bytes?

total_samples =? 8hrs x 60 mins x 60 sec x 0.1 Hz = 2880 samples

Data size = 13 bytes x 2880 samples = 37 440 bytes?

Convert to KB

37440 / (1024) = 36 KB / day?

Operation time = 1 year (e.g before data dump)

Total memory needed = 365 x 36 KB = ~12.8 MB

So select a memory that can store your data for that period of time. Select a size like 16MB.

Decoding the memory label:

Take a W25Q family of memory devices from WINBOND. The product number for a? typical memory would be labeled as follows: W25Q128JVSIQ. The description below from Digikey shows this:

To find the size of this memory, divide the 128M-Bits as follows:

128M-Bit / 8 = 128 000 000 / 8 = 16 000 000 =? ~ 16MegaBytes (MB)

Circuit diagram?

Most flash memories use the SPI interface. The pinout diagram for a Winbond W25Q128JVSIQ is as follows ( again, from the datasheet)

A schematic diagram for this part with KICAD would look like below. Note the 100nf capacitor for high frequency noise filtering on the VCC line.

Using an ESP32, connect the pins above to the V_SPI channels follows:?

Code implementation

Now here’s another fun part. Accessing the memory functions over code. There are several code implementations used to access SPI Serial flash memory. The most common I found are listed below:

https://github.com/PaulStoffregen/SerialFlash

Another library used is the SparkFun Arduino library here - https://github.com/sparkfun/SparkFun_SPI_SerialFlash_Arduino_Library

Or this one - https://github.com/Marzogh/SPIMemory - very good too

The choice of these libraries depends on your use case.?

Since I want to access the memory using a file system, I chose the SerialFlash Library by PaulStroffregen lib.

I wrapped the file operations in their own class for modularity and ease of maintenance - you can find the complete code here - https://github.com/bytecod3/embedded-systems-articles/tree/main/using-external-flash-memory

Create an SPI Flash file object

/* data logging */
uint8_t cs_pin = 5;
uint8_t flash_led_pin = 4;
char filename[] = "log.bin";   
uint32_t FILE_SIZE_512K = 524288L;  // 512KB
uint32_t FILE_SIZE_1M  = 1048576L;  // 1MB
uint32_t FILE_SIZE_4M  = 4194304L;  // 4MB
SerialFlashFile file;
unsigned long long previous_log_time = 0, current_log_time = 0;
uint16_t log_sample_interval = 10; // log data to flash every 10 ms
        

Initialize the memory - check for hardware initialization

/**
 * @brief Initialize the flash memory
 * @return true on success and false on fail
 *
*/
bool DataLogger::loggerInit() {
    char filename[20];

    if (!SerialFlash.begin(this->_cs_pin)) {
        return false;
    } else {
        this->loggerEquals(); // prettify
        this->loggerInfo();

        // init flash LED
        pinMode(this->_led_pin, OUTPUT);
        digitalWrite(this->_led_pin, HIGH);        

        // return a list of files currently in the memory
        if(!SerialFlash.exists("dummy.txt")) {
            Serial.println(F("Flash doesn't appear to hold a file system - may need erasing first.")); // TODO: Log to system logger

            // format the memory
            this->loggerFormat();

        } else {
            Serial.println(F("File system found"));
            Serial.println(F("Files currently in flash:"));
            SerialFlash.opendir();

            while (1) {
                uint32_t filesize;
                if (SerialFlash.readdir(filename, sizeof(filename), filesize)) {
                    Serial.print(filename);
                    Serial.print(F("  "));
                    Serial.print(filesize);
                    Serial.print(F(" bytes"));
                    Serial.println();
                }
                else {
                    break; // no more files
                }
            }

            uint8_t file_create_status = SerialFlash.create(this->_filename, this->_file_size);

            // create logging file with the provided filename
            if (!file_create_status) {
                Serial.println(F("Failed to create file"));
                return false;

            } else {
                // open the created file
                Serial.println(F("Created log bin file")); // TODO: LOG TO SYSTEM LOGGER
                this->_file = SerialFlash.open(this->_filename);
            }

        }
       
        this->loggerEquals();

        return true;
    }
}        

Read sensor data

Here, you can read the data and package it however you want. I prefer reading the data into structs.?

/* example data */
typedef struct {
    uint8_t device_state;
    float co2_level;
    float temperature;
    float battery_voltage;

} sensor_data_type_t;
sensor_data_type_t sensor_data;


=======================================================
// READ SENSORS - somewhere in the superloop or a task in RTOS
void readSensors() {
    sensor_data.device_state = getState();
    sensor_data.co2_level = readCO2Level();
    sensor_data.temperature = readTemperature();
    sensor_data.battery_voltage = readVoltage();
   
}        

Log data to memory?

You can log the data to memory using the logic below:

 current_log_time = millis();
// is it time to record?
        if(current_log_time - previous_log_time > log_sample_interval) {
            previous_log_time = current_log_time;
            data_logger.loggerWrite(sensor_data);
        }        

? ? ? ?

The function to log data to memory is listed below:

/**
 * @brief write the provided data to the file created
 * @param data this is a struct pointer to the struct that contains the data that needs to
 * be written to the memory
 *
 *
*/
void DataLogger::loggerWrite(sensor_type_t packet){
 
    // write the packet to memory
    this->_file.write((uint8_t*)&packet, sizeof(packet));

    // TODO: maybe return the size of memory written

}        

The next article will discuss how to retrieve and dump this data from memory.?

Thank you for reading this far. Happy making!

References

1. Winbond flash memories - https://www.winbond.com/hq/product/code-storage-flash-memory/serial-nor-flash/?__locale=en&partNo=W25Q32JV
2. Infineon industrial grade memory - https://www.infineon.com/cms/en/product/memories/f-ram-ferroelectric-ram/fm25v10-g/
3. Digikey product brief - https://www.digikey.com/en/products/detail/winbond-electronics/W25Q128JVSIQ/5803943
4. Example logging code - https://github.com/bytecod3/embedded-systems-articles/tree/main/using-external-flash-memory