Access S3 & AWS Secret Manager over the Private Link

In the last article on multi-tier VPC, We discussed public and private components of a network and learned that the backend (the processing engine) of any application is recommended to be in a private network, not the public, to make it secure.

Despite being in the private layer, the backend requires a lot of back and forth communication over the network with other components. For example, communicating to the DB layer to get data, sending requests over the internet to download some software patches or update a package, and connecting to a data layer like Amazon S3.

In this article, we are going to talk about connecting to Amazon S3 and Secret Manager from backend components of the application i.e private section of the network. By default, all your network calls (API requests) for S3 and Secret Manager go over the internet link which is not very secure. But VPC helps us make it secure using the concept of VPC Endpoints. A VPC endpoint enables you to connect with particular AWS services that are outside your VPC network through a private link.

Creating a VPC Endpoint for S3 and Secret Manager Using the AWS Console

Follow the steps below to create a VPC endpoint for S3 using AWS Console

1. Log in to the AWS console and go to the VPC Endpoint portal
2. Make sure your region is selected in the top right corner of the screen, see the screenshot.

3. Type a name for your Endpoint. You can edit the name anytime. Select 'AWS Services' in the Service category.

4. To create an Endpoint for S3 look for 's3' in the Services search bar and search 'secret' for Secret Manager Endpoint. See 1st screenshot for S3 and 2nd for Secret Manager

5. Select your VPC setting.

Note: Route table entries decide the flow of traffic.

6. AWS allows creating a custom Endpoint policy to control which services and components have access to the Endpoint. You can choose between ‘Full access’ and ‘Custom’. If you go for custom then either you can start writing your policy in the textbox given below or you can use the policy builder tool.

7.? Add tags if required and click the button Create endpoint

8. You should see an Endpoint like this in your console.

Creating a VPC Endpoint for S3 and Secret Manager using Terraform

The following Terraform snippet can be followed to create a VPC Endpoint.

resource "aws_vpc_endpoint" "s3" {
  vpc_id       = "<vpc-id>"
  service_name = "com.amazonaws.<aws_region>.s3"

  tags = {
    tagKey = "tagValue"
  }
}        

Using VPC Endpoint

With AWS Boto3 SDK

Create an SDK client using the following code snippet and all your network traffic will go through the Private link instead of S3.

import boto

s3_client = boto3.client(

????service_name='s3',

????endpoint_url='https://<vpc-endpoint>'

)