Exporting large data for your users

Hello, I hope you are doing great!

Have you ever worked with applications where you have to export large files?

Nowadays with more rules protecting users around the world, it is more and more common and necessary to give your users the ability to download the data your platform has about them, usually through an export mechanism.

If you have ever built a Social Network, or any application where users own lots of images or videos, the data file start to become bigger and bigger, it can grow to gigabytes in size. This means it is not wise to do an immediate export, and instead it is recommended to the request to a queue, process the request, then, once the process is finished, create a link to the file and send the link to the user's email.

Usually you do most of the job using a Background Service, however, there are some special considerations, first is you should create a compressed file, second is you must not have the whole file in memory, if you do you will for sure get issues such as OutOfMemoryException, and no, adding more memory is not the solution, you would still get the error when you add more users. The solution to the memory problem is to write the file directly to a stream, and in batches when possible.

Let's see an example from FairPlayTube Data Export.

As you can see in the previous image, we are using Azure Storage, we get a reference to the blob, then we open the blob's write stream, after that, we create a new ZipArchive, using the write stream we opened before.

Then we create new entries as we need like in the image below.

Since the write stream was initialized from the Azure Storage Blob Client, when we write to the archive, it will write to the remote stream instead of having a huge file in memory, preventing us from using too much memory and getting the OutOfMemoryException.

Images are from FairPlayTube, you can check the code here:

https://www.dhirubhai.net/smart-links/AQEckNXfO3LC9A

Remember you can help the development of these Open-Source projects by becoming a GitHub Sponsor:

https://www.dhirubhai.net/smart-links/AQGPNPu5KGUPlw

Hire My Services: https://www.dhirubhai.net/smart-links/AQEEZMHSFFnX1Q