Everything is everything - AWS re:Invent Recap

Everything is everything. This year's re:Invent was focused on the modern builder. In his keynote, Andy Jassy referenced the song “Everything is Everything,” by Lauryn Hill, explaining that “everything is everything” applies to technology because the choice of platform/provider is incredibly important, and builders shouldn’t have to settle for less than everything. Well with its rate of innovation and slew of announcements last week, AWS has more than any other provider and is providing “everything” modern cloud builders need/want to develop impactful solutions.

Jassy mentioned that the pace of innovation is also continuing to expand, with an expected 1,300+ service announcements over the course of 2017. Wow.

Here are some highlights of new cloud services announced:

Containers

Amazon Elastic Container Service for Kubernetes (EKS), a managed Kubernetes service running on top of AWS. EKS has a number of features that Jassy outlined in his keynote:

• Hybrid cloud compatible
• Highly available (masters deployed across multiple AZs, for example)
• Automated upgrades and patches

This gives AWS two different managed container offerings: ECS and EKS. However, Jassy said that containers want more—they want to run containers without having to manage servers and clusters. This led to an announcement of AWS Fargate, which allows customers to run containers without managing servers, clusters, or instances (think serverless containers). Just package your application into a container, upload it to Fargate, and AWS takes care of the rest. Fargate will support ECS immediately, and will support EKS in 2018.

Compute

New range of compute instance types (such as the new M5, H1, and I3m [bare metal] instances).

Cloud9 IDE

The AWS Cloud9 IDE is now GA, providing builders with a cloud-based tool for writing, running, and debugging code.

Serverless

Functions as a Service, or FaaS. While AWS Lambda has already gathered hundreds of thousands of customers, FaaS gives you more than just code execution; you also get event-driven services (like Lambda and Step Functions), lots of event sources (all the various triggers from AWS services), and the ability to execute functions at the edge as well as in the cloud (like Lambda@Edge and Greengrass). The new AWS Serverless Application Repository lets cloud builders share and reuse serverless functions.

Lamda is also being expanded and now includes support for AWS API Gateway VPC integration. Lamda also now support the .Net Core 2.0 and Go programming languages.

Databases

In his keynote, the “freedom” discussion lead Jassy to a discussion about databases, and a number of not-very-subtle attacks against Oracle. Customers want open database engines, and this demand is what led AWS to create Amazon Aurora. Aurora is MySQL- and PostgreSQL-compatible but offers the scale and performance that users demand from commercial databases. Per Jassy, Aurora is the fastest-growing service in the history of AWS.

Jassy announced a preview of Aurora Multi-Master, which supports multiple instances of Aurora for both read/write support across multiple AZs (with multi-region support coming in 2018). The preview for single region/multi-master is available now.

Next, Jassy announced Aurora Serverless—on-demand, auto-scaling Amazon Aurora. This service eliminates the need to provision instances, automatically scales up/down, and starts up and shuts down automatically.

However, relational databases are not the only solution out there; sometimes a different type of solution is needed. Sometimes a key-value datastore is a better solution, leading Jassy to talk about DynamoDB and ElastiCache (which currently supports Redis and Memcached). To expand the functionality and utility of DynamoDB, Jassy announced DynamoDB Global Tables. DynamoDB Global Tables is the first fully-managed, multi-master, multi-region database. DynamoDB Global Tables enables low-latency reads and writes to locally available tables. It’s now GA.

Jassy next announced DynamoDB Backup and Restore, to simplify the process of backing up and restoring data from/to DynamoDB databases. This new offering will enable customers to back up hundreds of terabytes of data with no performance interruption or performance impact. This offering is GA today, with point-in-time restore coming in 2018.

To better enable using data across multiple databases, Jassy announced the launch of Amazon Neptune, a fully-managed graph database. Neptune supports multiple graph models, is fast and scalable, enables greater reliability with multiple replicas across AZs, and is easy to use with support for multiple graph query languages.

Virtual Reality

AWS announced a preview of Sumerian, a cloud platform that can be used by novice or expert builders to create virtual and augmented reality environments.

Video Streaming

Kinesis video streams service is now GA and enables uses to securely ingest and store video, audio, and other time-coded data.

Analytics & Insights

S3 Select helps AWS customers perform analytics on the correct subset of data that might be stored in S3, With S3 Select, the ability to use standard SQL statements to “filter” out or select the correct subset of S3 data. In his keynote, Jassy shared some TPC-DS benchmarks on a Presto queries (8 seconds without S3 Select, 1.8 seconds [4.5x faster] with S3 Select).

Glacier Select allows AWS users to run queries on data stored at rest in Amazon Glacier.

Amazon Comprehend service is able to understand text and pull insights and analysis from that data.

Machine Learning

In his keynote, Jassy cited the desire of builders for machine learning to be easier to use and embrace than it is right now.

AWS views three layers of ML:

The bottom layer is for expert ML practitioners who deeply understand learning models and frameworks, and Jassy re-iterated AWS’ support for all the various major frameworks and interfaces customers want to use.

The middle layer is for everyday developers who aren’t experts in ML, but it’s still too complicated for most users. To help with the challenges in this layer, Jassy introduced Amazon SageMaker (leverages open source Jupyter project). SageMaker provides built-in, high performance algorithms, but doesn’t prevent users from bringing their own algorithms and frameworks. SageMaker also greatly simplifies training and tuning, and helps automate the deployment/operation of machine learning in production.

To further help get machine learning into the hands of developers, Jassy announced DeepLens, the world’s first HD video camera with built-in machine learning support. Jassy brought out Dr. Matt Wood to talk more about DeepLens and SageMaker. After talking for a few minutes, Wood did a demo of DeepLens performing album identification and facial expression recognition.

The top layer, according to Jassy, is a set of application services that leverage machine learning. Examples here are Lex, Polly, and Rekognition. Jassy announces Rekognition Video, which is real-time batch video analysis (like what Rekognition does for photos). To help get video/audio data into AWS, Jassy announced Amazon Kinesis Video Streams. Rekognition Video is deeply integrated with Kinesis Video Streams.

On the language side (as opposed to video), Jassy announced Amazon Transcribe to convert speech into accurate, grammatically correct text (initially available with English and Spanish). In the near future, Transcribe will support multiple speakers and custom dictionaries.

Jassy also announced Amazon Translate, which does real-time language translation as well as batch translation. It will support automatic language detection in the near future.

Jassy also announced Amazon Comprehend, a fully-managed natural language processing service. It analyzes information in text and identifies things like entities (people, places, things), key phrases, sentiment, and the language of the content. Comprehend can not only identify information in a single document, but can also be used to perform topic modeling across large numbers of documents.

To talk a bit about how the NFL is using Amazon and machine learning, Jassy brought out Michelle McKenna-Doyle, SVP and CIO of the NFL. McKenna-Doyle shared some details on Next Gen Stats (NGS), which spans AWS services like Lambda, CloudFront, DynamoDB, EC2, S3, EMR, and the Amazon API Gateway (among others). NGS generates 3TB of data for every week of NFL games. McKenna-Doyle also talks briefly about future plans for incorporating machine learning and artificial intelligence into the NFL’s NGS plans (to do things like formation detection, route detection, and key event identification).

Security

Amazon GuardDuty analyzes API calls made to running virtual resources in a customer's AWS account to help detect anomalous activity that could be indicative of a security risk.

IoT Device Defender enables organizations to manage fleets of IoT devices to help minimize potential security risks.