re:Invent 2021: Where Data Lives

re:Invent 2021 brought the vision for the next chapter of AWS, with a new CEO, and a push for higher level solutions, Cloud services and abstractions. I went through these on my last article that you can read here: re:Invent 2021 in 4 keynotes.

While these new Cloud services and solutions truly enable more "pathfinders" to move even faster, the truth is that infrastructure matters more than ever. AWS continues to innovate in this space at an accelerated pace.

Innovating at the Foundation: Spindles and Flash

Foundations matter and Storage is the most critical foundation in the Cloud.

Data needs to be stored, accessed, and processed to generate business value, otherwise, it becomes costs and even a liability. Persisting information across its lifecycle remains one of the most challenging information technology problems to solve, and the Data Storage market has been continuously innovating in improving performance, efficiency, availability, and durability.

From the Mainframe to the Cloud, storage starts with the capabilities of the media.

Peter DeSantis' infrastructure keynote went deep on Hard Drives (HDD) and Solid State Drives (SSD)'s characteristics, advantages, limitations, and what AWS is doing to obtain the maximum efficiency of these technologies.

Starting with the HDD, if you are curious to learn how AWS makes Amazon S3 scalable, available, durable, and secure at scale, read the recent Amazon Science publication "Using lightweight formal methods to validate a key-value storage node in Amazon S3".

But it's Flash that powers data access at high performance and AWS is innovating in this space as well, with AWS Nitro SSD.

Designed to avoid latency spikes and deliver great I/O performance on real-world workloads, AWS' own benchmarks show instances that use the AWS Nitro SSDs deliver 75% lower latency variability than I3 instances, giving you more consistent performance.

Innovating on Cloud Storage Services

AWS infrastructure teams' approach to storage technologies enables further innovation on the AWS Storage Services portfolio. Block, Object and File Storage services, as well as data protection, data management and data movement services, brought significant new capabilities to help customers migrate and optimize their Cloud environments.

Here is a summary of all the announcements at re:Invent.

AWS announced new capabilities for Elastic Block Storage (EBS) around data protection and recovery. The new Recycle Bin for EBS Snapshots lets you set up rules to retain deleted snapshots so that you can recover them after accidental deletion. Amazon EBS Snapshots Archive provides a low-cost storage tier to archive full, point-in-time copies of EBS Snapshots that you must retain for 90 days or more, achieving up to 75% lower storage costs.

AWS also announced a new Offline Tape Migration using AWS Snowball Edge service that enables customers to seamlessly move tape media storage to the Cloud. By embedding AWS Tape Gateway service into AWS Snowball Edge migration services, it makes it easier for organizations to eliminate the risks and costs of aging and expensive media management solutions.

Amazon FSx for OpenZFS, a new managed file service in the Cloud extending the FSx family, allows customers to create ZFS shares on-demand, without having to deal with hardware provisioning, software configuration, patching, backups, etc.

Two significant AWS Backup announcements: AWS Backup support for VMware and VMware Cloud on AWS, a new capability that lets you centralize and automate data protection of VMware virtual machines in the Cloud and on-premises. In preview mode, AWS Backup support for Amazon S3 will enable customers to protect primary data stored in Amazon S3 with a fully managed data protection service.

Also announced in preview mode, Amazon FSx for NetApp ONTAP integration with VMware Cloud on AWS will offer high-performance?file storage with compression, deduplication, snapshots, clones, and storage-based replication across a Hybrid Cloud environment.

Innovating in Cloud Storage Optimization

Amazon Simple Storage Service (S3) was the first AWS Cloud service. After 15 years, S3 stores more than 100 trillion objects and serves as the foundation for over 10,000 Data Lakes in the Cloud.

Over the years, the Amazon S3 service team introduced lower costs storage classes like S3 Glacier, S3 Infrequent Access, and S3 Glacier Deep Archive. Lifecycle management capabilities enabled customers to move objects between different classes, and realize savings over time. On re:Invent 2018, AWS launched S3 Intelligent-Tiering (S3-INT), a new S3 class that offered managed and transparent data movement between storage tiers. Since then, S3-INT has been broadening its use cases by incorporating more intelligence in data optimization.

On re:Invent 2021, Amazon S3 took a giant step with the launch of a new storage class, S3 Glacier Instant Retrieval (S3-GIR). This storage class can be used directly, ideal for data that is rarely accessed but needs to be immediately available when needed. For instance, a mission-critical backup image, readily available to support a restore operation if necessary, or a video file that is stored at low-cost Storage but needs to be online once a user requests it for a workflow.

This same technology is also available as a third-level storage tier inside Intelligent Tiering, the S3-INT Archive Instant Access (S3-AIA) tier. Now, in addition to S3-INT Frequent Access and Infrequent Access tiers, data that isn't accessed in 90 days will automatically migrate to S3-AIA and further reduce overall storage costs with S3-INT. Customers using S3-INT today will notice a price reduction on the next billing cycle as data migrates to S3-AIA.

AWS announced 2 additional capabilities for S3. First, a new Amazon S3 Object Ownership setting that lets you disable access control lists and the Amazon S3 console policy editor which now reports security warnings, errors, and suggestions powered by IAM Access Analyzer. This simplifies data access management in consolidation use cases like Data Lakes.

Finally, AWS also announced new capabilities of Amazon FSx for Lustre and Amazon S3 integration, now featuring a full bi-directional synchronization of file systems with multiple S3 buckets or prefixes, making the convergence of Analytics, AI and ML with HPC easier in the Cloud.

The Big Picture: Where Data Lives

In addition to the speed and agility that the on-demand model allows, the Cloud also made it financially viable for businesses to operate at scale, and Storage is a critical component to this strategy.

Data has gravity. Where Data lives, applications will live as well. Like a Star with planets orbiting it, the most efficient Storage environment will likely be the preferred environment for applications to be deployed.

Watch the AWS Storage leadership

The AWS Storage leadership session is now available. Watch here:

Have a question about how AWS can help you? Don't hesitate to reach out. Hit me here on LinkedIn or on?Twitter.

Disclaimer:?These comments represent my opinion and not of my employer.