Lambda in AWS
We asked our DevOps Engineer Karel Vesely to tell us more more about Lambda that is commonly used by our clients who use AWS and how it could potentially benefit you if you are not currently using this in your services -
"Lambda is an AWS service that can run ‘functions’ (essentially small portions of code) without requiring you to launch instances beforehand. The instances that the code runs on are managed by the Lambda service itself. This style of compute is known as “Serverless”, making it a very popular choice, as compute charges are only incurred for the execution time and provisioned memory of the function, which is basically negligible compared to the cost of an instance running constantly. This is aided by a large free tier allowance of 1 million requests a month and deep integration with many AWS services.
A lambda function is intended to be a short duration task, currently limited up to 900s execution and 10GB RAM. It supports most popular programming languages including Python, NodeJS and Go. The code is uploaded to AWS (as a zip file or container image) with a defined ‘Lambda handler’ that gets invoked by Lambda. This lambda handler must be defined as shown in the image below, accepting an event and context parameter. The event object contains details about how the function was invoked, for example an API Gateway would have a different structure than an S3 event notification or an SNS topic. The context object provides details on the function’s configuration such as the timeout and memory limit. The response format from the Lambda is less strict, but most commonly an HTTP request style is used, allowing these responses to be proxied through API Gateway.
Lambda has native integration with Step Functions, which are an alternative to Airflow DAGs and can be used to create Data Pipelines. They are defined in JSON instead of Python code.
A simple data pipeline could include processing a file uploaded to S3 and inserting records into a database, providing an email notification on success. This pattern is frequently implemented on client site with Airflow. The advantage of Step Functions is that the pipelines can be triggered based on an event rather than on a schedule. It is also serverless and more cost-effective than a cluster of Airflow instances in many scenarios.
More complicated workflows can be easily constructed, allowing retries and parallel jobs. Step functions can also start Glue jobs, which allow long-running data processing jobs to run with higher memory and vCPU. Pricing is based on the compute duration, making it a suitable alternative to expensive EMR clusters in many scenarios (which happens to be the largest AWS cost for one of our clients)."
In conclusion Lambda is an excellent, performant and cost efficient way to run your repeatable "Processes" in a serverless environment and is something we would recommend for clients who are not currently using this in their stack.
领英推荐
Inov8 provide a free service assessment to help you get hte most out of your cloud platform. Get in touch to find out more -
Find our details on Linktree
+44 (0) 141 328 2939
+44 (0) 131 261 7870