Chop Chop #6
If someone else has done it better, use it
Welcome to ChopChop#6! This week we are going to be talking Python and running it in the cloud within a Lambda function. More specifically, we are going to look at adding an external library using a Lambda layer, but we will get to that, lets start of simple with the Lambda function itself.
A Lambda function is a piece of code, in our case Python code, that does what it is programmed to do, on the cloud. It is exactly the same as running the code on your own laptop, except you don't need to make sure you always leave your laptop on, because it's in the cloud and that is the whole point of putting it there.
Most Lambda functions are run when an event happens, which is known as a trigger. It might be a certain time of day, or the result of another piece of code, or even simply when someone clicks a button. There are so many things that can be triggers, and once that event happens, our Lambda function runs and does its job.
Now if our Lambda function is pretty simple, then no stress; the function runs, does its job, and then stops and waits for the next time that the trigger happens. Then the process repeats itself. But what happens when you want to do something a little bit fancy, such as using someone else's code? In Python someone else's code is known as a library, and your own code can use that code to do its own job better. Or faster. Or in less code. That is the whole point of programming, you build on what others have done and the collective is better for it. Once you use someone else's code, you create a dependency on that library, but that is a chat for another day.
Now to use a library on your laptop you simply download the code and then use the import statement in Python and it is there. Nice and easy. But in a Lambda function, it is a little bit more challenging as you cannot just download the code. This is where a layer comes in. A Lambda layer is where the code that you want your function to have access to is stored. You create a zip file that has everything inside and put it on the cloud and tell your Lambda function to use that layer.
领英推荐
Of course we know where the code we want is, so this should be pretty straightforward right? One would hope, but there are some technical issues that even I don't fully understand, but the short version is that Lambda runs on linux, which is an operating system like MacOS or Windows. If you zip up your code on a different operating system, Lambda throws a fit and doesn't accept that as valid code to use, so it just won't. This causes some issues as it is hard to know if your layer will be ok until you actually deploy it to the cloud and test it.
I ran into this issue in another project and my solution was to create an instance in the cloud that was the exact same one that my Lambda function would use, create my layer there, and then upload to the cloud. This is super manual, hard to replicate, and required more effort than I wanted to put in. This time, I turned to Docker. I won't go into the details of Docker here (let's save that for another post), but it is a containerisation software that helps, and I recommend everyone get comfortable using Docker.
So I created my Lambda layer with Docker, and then used our trusty infrastructure as code to deploy it, same as any other piece of cloud infrastructure in Chop. I now have a new repository called chop-infrastructure-layers that contains all the code to deploy the layer(s) I need, and it is easy to manage those dependencies moving forward. There was some quite useful blogs out there that helped to get all this set up, but really it boiled down to two lines of code:
docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.9" /bin/sh -c "pip install -r requirements.txt -t python; exit
zip -r requests_layer.zip python > /dev/null
These do look a little scary, the first line runs Docker and uses the AWS image that replicates what a Lambda function uses to download the correct code that I want to use. The second zips it up and makes it ready to be deployed into the cloud. All that was left was to build my CDK code that pointed at that zip file. This sounds simple but there was quite the trial and error to get this to work.
And what did I need to use the code for, I wanted to use the requests library to post data through the API of another microservice. This could have been easier to write to the database directly, but to make our microservices truly independent, everything has to go through the front door, and not take a shortcut around the back. The issues of coding things properly the first time.
That's all for now, chop chop.