Hybrid Cloud Storage with AWS Volume Gateway : store your data on-premise and/or in the cloud

There are various scenarios where your current on-premise infrastructure might need a boost in terms of storage capacity. Imagine you are running an application where users upload thousands of videos and photos daily. With an average 4k-video of an hour length being around 15GB to 30 GB, you might be right to anticipate Petabytes (1,000,000 GB) of storage space for these videos.

A common solution to this is to physically expand your current on-premise storage infrastructure. But this option is expensive and requires significant planning. You have to estimate your current and future storage needs, for this application and other applications you currently have. You have to purchase hardware, find space and make sure everything is configured properly. As you can tell, this solution is expensive in terms of time, resources (for planning and installation), and upfront capital spent.?

Another solution is to use Hybrid Cloud Storage by joining your current infrastructure with public cloud providers services like Amazon Web Services (AWS). This approach allows you to keep using your application as you currently are, while benefiting from virtually unlimited storage space provided by Amazon S3.

AWS has a special service dedicated to expanding your on-premise storage capacity using Amazon Simple Storage Service (S3) behind the scenes : AWS Storage Gateway Volume Gateway.

AWS Storage Gateway - Volume Gateway

AWS Storage Gateway - Volume Gateway provides cloud-backed storage volumes, which means that your data is stored both locally and in AWS Cloud (through Amazon S3). These storage volumes can be accessed by your on-premises application servers using iSCSI, a standard protocol for connecting storage devices. Essentially, it allows your servers to treat the cloud-backed storage as if it were local, which means easy integration without major changes to your existing systems.

When your applications send data, AWS Volume Gateway first saves it locally on your on-premise storage systems. Then the data is asynchronously uploaded to Amazon S3 in the background. This allows for fast, local storage while still benefiting from AWS Cloud’s unlimited storage capacity.

You also benefit from Point-in-Time snapshots of your volumes. In other words, your application server's data is frequently backed up in the cloud via Amazon EBS. These snapshots are managed through AWS Backup, ensuring that your data is consistently protected and can be restored when needed.

There are two types of AWS Storage Gateway - Volume Gateway :?

• Cached Volumes

• Store Volumes

AWS Volume Gateway Cached Volumes: Hybrid Storage for Frequently Accessed Data

When using AWS Volume Gateway Cached Volumes, a portion of your "hot" or frequently accessed data is stored locally, while a full copy of your data is securely stored in Amazon S3. This option provides low latency when accessing hot data because your applications will first try to read data from on-premise storage before making requests to the cloud.?

AWS Volume Gateway Cached Volumes also use an upload buffer. It is an on-premise staging area where data is kept before being securely uploaded to Amazon S3 via SSL-encrypted connections.?

With each cached volume supporting up to 32 terabytes of data and up to 32 volumes per gateway (totaling 1 petabyte), you can scale as your data grows. AWS recommends using about 20% of your local storage for cache to ensure optimal performance.

Key benefits of Cached Volume:

• Frequently accessed data is stored locally for fast, low-latency access.
• All data is durably stored in Amazon S3 for scalable, long-term storage.
• You can recover your data locally or in the cloud via EBS volumes.
• Reduced need for on-premise storage, leading to lower hardware costs.

AWS Volume Gateway Stored Volumes: Low Latency with Full Local Copies

With AWS Volume Gateway Stored Volumes, all your data is kept entirely on-site, which means anytime your applications need to access data, they pull it directly from your local storage. This guarantees low-latency access to your data because everything is stored on-premise. In parallel, a backup copy of your data is uploaded asynchronously to Amazon S3, providing you a cost-effective backup solution that reduces the need for expensive on-premise storage infrastructure.

Stored volumes can scale from 1GB to 16TB per volume, with up to 32 volumes per gateway, totaling 0.5 petabytes of storage per gateway.

Advantages of Stored Volume:

• All reads are performed locally, guaranteeing low-latency for mission-critical applications.
• Backup copies in Amazon S3 free up valuable on-premise storage space.
• Data can be recovered either locally or in AWS through EBS volumes, providing rapid recovery in disaster situations.
• Cost-effective cloud storage for your data without needing to over-invest in on-premise backup solutions.

Conclusion

AWS Volume Gateway gives you flexible options to manage and scale your storage needs with a hybrid approach. Whether you choose Cached Volume for a balanced solution or Stored Volume for low-latency, fully on-premise data storage, AWS ensures you get the best of cloud scalability and local performance, without the bank on hardware expansion or backup solutions.