Media Uploads to S3 via Python Lambda: Form-data / Binary file

Greetings, cloud enthusiasts!

Today I'm going to look at two effective ways to upload media files to an Amazon S3 bucket using Python Lambda functions: via form-data and binary payloads. Both methods have their use cases, but we'll break down the differences with step-by-step explanations and examples.

Note: I'm using images for the code because LinkedIn is terrible at formatting code, but you can find the complete code here. This article contains only the essentials to help you understand the differences between the methods for uploading files using Lambda.

Uploading Media Files using Form-Data

This is a common method when uploading media through HTML forms or tools like Postman. With form-data, files are uploaded as part of a multi-part request, typically including additional metadata.

However, when Lambda receives this form-data, it arrives in an encoded format that we need to decode. Here's why we use Python's cgi.FieldStorage for this task: it’s specifically designed to handle multi-part form data and extract the files from the request body.

Let’s break down the critical functions that make this work.

In this function:

• We decode the Base64-encoded body of the request to get the raw data.
• We create an environment that simulates an HTTP POST request (necessary for cgi.FieldStorage to function properly).
• FieldStorage is responsible for parsing the form-data and returning the file part of the request.

By using cgi.FieldStorage, we take advantage of its built-in capabilities for parsing complex form-data structures, making it easier to work with multi-part requests.

Lambda Function to Handle Form-Data

Next, in the lambda_handler, we validate the request, call our get_file_from_request_body function, and then upload the file to S3:

Uploading Media Files using Binary Method

The binary method is often simpler and more direct than form-data, especially when your client can send the file as raw binary data.

The primary task here is to decode the Base64-encoded file content and upload it to S3.

Decoding Binary File Content

Instead of parsing multi-part data like in the form-data method, we simply decode the binary file content:

The decode_file_content function takes the Base64-encoded body from the request and converts it back to binary format. This method is straightforward as it doesn’t need to deal with form-data structures.

Lambda Handler for Binary Uploads

The lambda_handler for binary data looks quite similar to the form-data one, but instead of using FieldStorage, we directly decode the file content:

Pre-signed URLs: the better way to upload files to S3

While both methods discussed above work, there is an even more efficient and secure way to handle file uploads to S3—using pre-signed URLs.

With a pre-signed URL, the client uploads the file directly to S3 without your Lambda function acting as an intermediary. This minimizes the load on your backend and keeps your Lambda functions fast and lean.

Here’s how you can generate a pre-signed URL:

The client can use this pre-signed URL to upload a file directly to S3 via a simple HTTP PUT request. This method improves performance and keeps your API secure.

These examples demonstrate how to upload files via a Lambda function and handle the data correctly. However, if your goal is specifically to upload files to S3, use pre-signed URLs. They provide a more secure, scalable, and efficient solution for most production use cases.

For the complete code, check here.

Happy coding!