AWS Lambda Environment Variable Challenges
Today, we wanted to deploy a change in our AWS Lambda, which is packaged as a container image (having moved away from fat jars due to size limitations). The Lambda runs in a concurrent setup for scaling, processing multiple messages in parallel.
During the deployment, we introduced a change that included renaming an environment variable. Shortly after, we noticed several errors in our logs.?
As a precaution, we rolled back to the previous version. But, interestingly, we received another batch of errors — this time, solely during the deploy. So, we investigated.
It turns out that the execution environment is shared among Lambda instances, and changes to environment variables also apply to Lambdas that haven’t been upgraded yet during the rolling deploy. As a result, the function attempted to read a variable that no longer existed, causing errors.?
Luckily, our retry mechanisms saved us from any major headaches.
This behavior might seem counterintuitive, especially considering the expectation of isolation in containerized environments.?
A valuable lesson learned for us — and perhaps for you too!