AWS Lambda: Accessing private VPC resources and internet without NAT gateway

There is a commonly known design decision of AWS to launch lambda in a separate VPC that belongs to AWS itself. This spares the networking costs, as AWS pays for internet connection of lambda, however, this comes with problem of exposing customer VPC internal resources to that lambda. So, if we, for example, require to use lambda for querying an internet API and storing data in RDS DB/Redis inside a VPC, we are in trouble, since we can not connect directly and also don't want to expose internal resources to internet.

What common solution proposes: put lambda in (private) subnet that has routing to the internal resource, and add NAT gateway for connectivity. We just use this lambda configuration parameter (VPCconfig)

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-function.html#cfn-lambda-function-vpcconfig

that will allow us to specify VPC subnets and security group the lambda must be in. However, this comes at a price. NAT gateway costs around 0,045+ USD per Hour even if it is not used, and also traffic is billed. https://aws.amazon.com/vpc/pricing/?nc1=h_ls. That's a whooping 394 USD a year (without traffic!). So we don't want a NAT gateway for a casual use case.

My solution - exploit the fact that lambda API of lambda in private subnet of customer VPC is globally accessible. There, you can make a lambda "proxy" for another lambda to retrieve/store data. This can be achieved using lambda invoke API https://docs.aws.amazon.com/lambda/latest/dg/example_lambda_Invoke_section.html

With this, lambda can have access to internal VPC resource using another lambda as proxy. In our case, we just split one lambda in two. One with VPCconfig set and one without it.

Tuning concurrency

The biggest downside of this solution is added concurrency. When proxy is invoked too much this will incur costs and throttles. However, there are some good measures that help save costs:

1. Batch requests together. Process request in batches, keep in mind that lambda API supports up to 6 mb of payload. In my case around 80-100 set/delete requests were batched together.
2. If you don't care about event order or result (like you don't issue 'get' command), don't wait for the execution result (use InvocationType 'Event' instead of 'RequestResponse')
3. Make proxy lambda small. 128 mb of memory and a max timeout of 3-5 seconds should suffice

A very simple JS pseudo code example of redis proxy can be found bellow.

const Redis = require('ioredis');

function createClient() {
    return new Promise((resolve, reject) => {
    const client = new Redis.Cluster(...params from env/ssm); // new Redis(... params from env/ssm);
    client.on('connect', () => resolve(client))
    client.on('error', reject);
    }
}

let cache;
exports.proxyToRedis = async (event) => {
    if(! cache) {
      cache = await createClient();
    }
    // We assume that internet-connected lambda will send array of commands in this field.
    // { "bundle": [{ "command": "get", "args": [] }, { "command": "set", "args": ["foo","bar"]}]}
    const { bundle } = event; 
    const results = [];
    for(let command of bundle) {
        results.push(await cache[command.command](...command.args));
    }
    // Send array of results to requester lambda
    return results;
}         

#aws #lambda #subnet #natgateway