Add External Python Libraries to AWS Lambda using Lambda Layers

If you are a Cloud Engineer or Developer who loves to code in python like myself, 7 out of 10 times, you might find yourself using AWS lambda. For Instance, you might be a Data Engineer trying to use AWS lambda to preprocess your dataset and you need to import pandas or number library. You will have a section of your code where you need to import the packages as illustrated below

import pandas as pd
import numpy as np        

If you run this line in your plain lambda function, you will get the response below from lambda

Unable to import module 'lambda_function': No module named 'pandas'        

How do we resolve this? We need to create our own libraries as custom package, then find a way to attach the package to lambda.

There are different ways to go about this but my favorite is using Lambda layers. According to AWS, ?"Lambda layers?provide a convenient way to package libraries and other dependencies that you can use with your Lambda functions."

We will divide this lab into 4 Sections:

1. Create a lambda function in AWS and test
2. Install your packages on your local machine and zip
3. Create lambda layer and upload packages
4. Attach layer to lambda and test

Create a lambda function in AWS and test

Login to your AWS and Navigate to lambda service. Click on "Create lambda" and fill the details below. Name the lambda function "external-package-lambda" . Choose Python3.7 as your runtime version.

Click Create. Wait till lambda creates. Scroll down and edit the code to look similar to what I have below. All I added was 2 lines extra lines of code to import 2 libraries (pandas and numpy). Try to type them in yourself and make sure your code looks exactly like this.

Go ahead and click "Deploy" to save the function. Next, click "Test" and you should see a pop up that prompts you to name the test event. Enter Event name as "test" and leave everything else as default. See below

Scroll down and click "Create". You should return back to main lambda page. Click "Test" again and you should get similar response below.

Unable to import module 'lambda_function': No module named 'pandas'        

Install your packages on aws cloudshell and zip

AWS Cloudshell is a newly created service in AWS where developers can programmatically interact with AWS resources without spinning up any server. This will be perfect for our case. We will be installing our packages from cloudshell, zipping it and then uploading to s3.

Quickly open the s3 console and create an s3 bucket and give it a unique name. This is where you will store the zip package created. Save the name somewhere

Open aws cloudshell. It can can be found at the top right of the console. Looks like the image below

Wait for the the terminal to load. Once it's loaded, create a directory named "packages", then change into the directory by running the commands below one after the other

mkdir packages
cd packages        

Next, create a virtual environment named venv and activate the virtual environment by running the commands below one after the other.

python3 -m venv venv
source venv/bin/activate        

Your virtual environment should be activated now. Make sure you remain in the same directory. What you will do next is create a folder named "python" and change into the python directory by running the commands below one after the other.

mkdir python
cd python        

Its important you create a directory named python. This is mandatory when working with lambda layers. Now that you are in python directory, install pandas and then numpy by running the commands below one after the other.

pip install pandas -t .
pip install numpy -t .        

We are using pip to install the packages specifying the current directory (python) as the target. Wait until installation is done.

Try to list what is in the current directory by typing "ls" in terminal and it should look similar to the below image.

Those are the packages we will be using for lambda. We will ned to zip the python folder where they live but before we do that, let's save space and delete objects with ".dis-info" extension from the folder. They are not needed. The best way to this is running the command below

rm -rf *dist-info        

After running the above, we should be left with only the relevant packages needed.

Get out of the the python directory by going a directory backward using the command below

cd ..        

Now, we will zip the python directory and give it a name called "my-first-lambda-package.zip". Run the command below to achieve this.

zip -r my-first-lambda-package.zip python        

Once done, you should have a zip file named my-first-lambda-package.zip in the current directory.

Next, upload the packaged zip into the s3 bucket you created earlier. My command looks like the below

aws s3 cp my-first-lambda-package.zip s3://your-s3-bucket-name/        

Visit the s3 bucket you created to confirm the zip file was uploaded successfully. Make sure you copy the object URL when you visit s3 to confirm by clicking the checkbox beside the object and clicking Copy URL above . See below (Save the URL somewhere)

Create lambda layer and upload packages

Go back to your AWS lambda console and select layers. See below

Click on "Create layer" and fill the needed configuration similar to mine. See below image. I selected the "upload a file from Amazon S3" option and I pasted the URL we copied earlier in the text box below. Make sure you select python3.7 runtime and click create

Attach layer to lambda and test

In this section, we will attach the lambda layer we created above to our lambda function.

Open the lambda function we created at the beginning of the project named "external-package-lambda" and scroll all the way down. You will see an option to add a layer on the downward right. See mine below

Click "Add a layer" and a page should appear for configuration. Select "Custom layers" and from the dropdown, select "my-first-layer" in previous section. If it asks for version, select version 1.

Click "Add".

Go back to the lambda function (Make sure update is complete). Then click on "test" again and your lambda should be successful this time.

Now you can go ahead and do whatever you want to do with those packages.

I hope this was helpful. Drop comments and feedbacks to help me improve and do more!

Don't forget to share with friends and connections.

Cheers!