AWS Trusted Advisor Explorer: Everything you need to know

Cost optimization is a top initiative for every organization - we know it, and the cloud providers know it. COVID-19 has amplified the criticality of cost optimization for businesses.

Amazon offers a plethora of native cost optimization solutions to help its customers reduce costs. In 2019 alone, Amazon released several cost optimization tools, including EC2 Right Size Solution (released in November), Amazon EC2 Resource Optimization Recommendations (released in July), and last but not least, AWS Compute Optimizer, which was released during Re:Invent 2019.

If you wish to learn more about AWS Compute Optimizer and other cost solutions from AWS, check out this article I published last year right after Re:Invent.

On May 7, 2020, AWS introduced AWS Trusted Advisor Explorer - its latest cost optimization solution. The same day, Amazon announced that 5 new cost optimization checks were added to AWS Advisor, all new checks focused on reducing costs with reservations and AWS Savings Plans.

It was quite a busy week for AWS Trusted Advisor. On May 4th, AWS announced that AWS Systems Manager Explorer, AWS's operational dashboard, will support multi-account summary and aggregation of Trusted Advisor checks. I will cover the differences between this functionality and AWS Trusted Advisor Explorer below.

What is AWS Trusted Advisor Explorer?

The new solution, offered as a CloudFormation template, will provision infrastructure leveraging several AWS Services and products. The solution will collect and aggregate cost optimization recommendations from AWS Trusted Advisor and AWS Resource Group data editor originating from multiple AWS accounts into a single Data Lake.

Users can then use AWS QuickInsight to visualize the data by creating custom dashboards. AWS indicated that “Any visualization that can integrate with Amazon Athena and Amazon S3 can be used,” so solutions such as Grafana should work.

The solution also allows users to run queries against the data using Amazon’s interactive query service for S3-based data, Amazon Athena.

It is essential to note the solution does not include new cost optimization recommendations to AWS Trusted Advisor. AWS shared that the solution is designed to “enrich the data with resource tags that further enhance the discovery and filtering capabilities.”

Currently, the AWS Trusted Advisor Explorer solution supports nine Trusted Advisor cost optimization checks including Amazon RDS Idle DB Instances, Amazon EC2 Reserved Instance Lease Expiration, Amazon EC2 Reserved Instances Optimization, Low Utilization Amazon EC2 Instances, Idle Load Balancers, Underutilized Amazon EBS Volumes, Unassociated Elastic IP Addresses, Amazon RDS Idle DB Instances, Amazon Route 53 Latency Resource Record Sets, and Underutilized Amazon Redshift Clusters.

How does it work?

The below diagram from AWS shows the components that will be provisioned as part of the solution and their integration points:

Once the infrastructure is deployed, an Amazon CloudWatch Event rule (e.g., Scheduler Block) will be created to parse the user’s AWS Organization to obtain the account list using an AWS Lambda function (users can override this and provide a custom account list via CSV file, if needed.)

Another Scheduler Block will handle the extraction of cost recommendations from AWS Trusted Advisor cost as well as tag data from all accounts and place them in a dedicated Amazon S3 bucket. The solution will create two S3 buckets, one to store the above raw Trusted Advisor recommendations, and tags and the other for access logging.

Once the data is on the S3 bucket, two AWS Glue crawlers will kick in and create tables in an Amazon Athena database. For those who are not familiar with the service, AWS Glue is a managed service that extracts, transforms, and loads (ETL) data for analytics.

Then, once the AWS Glue crawlers are done, another Scheduler Block will kick in to create the required Amazon Athena views to allow queries against it.

It is important to note that since the solution leverages multiple AWS services, some might not be available on all regions right now - AWS recommends deploying it in the US-East (N. Virginia) region.

Lastly, the AWS Trusted Advisor cost optimization checks (and few others) are only available for organizations with a Business or Enterprise-level Support plan with AWS. To check if you have it, simply head to the AWS Trusted Console and click on the Cost Optimization section – for example, the below is from my AWS account that does not have active support:

AWS Trusted Advisor Explorer vs. AWS Systems Manager Explorer

As mentioned earlier, AWS announced on May 4 that AWS Systems Manager Explorer will provide a multi-account summary of Trusted Advisor checks. There are a few notable differences between the services:

• Systems Manager Explorer will aggregate all AWS Trusted Advisor checks, including performance, security, and reliability checks. AWS Advisor Explorer will only aggregate cost optimization recommendations.
• Systems Manager Explorer is available in multiple regions, while AWS Advisor Explorer is limited (AWS recommends using US-East region) since it relies on multiple services that are not available in all regions.
• AWS Trusted Advisor Explorer provides more flexibility since it is a CloudFormation template users can modify. It also allows data queries via AWS Athena (SQL style) and custom dashboards leveraging resource tags and external visualization tools. AWS Systems Manager Explorer supports queries via AWS API to create custom reports and custom dashboards are available with pre-set widgets.

Summary

The solution is designed to address a few existing limitations when using AWS Trusted Advisor in a large-scale deployment with multiple accounts. It allows large organizations to aggregate the cost recommendations from multiple accounts’ AWS Trusted Advisor into a single location. The addition of resource tags will help to enrich the data visualization and associate cost opportunities with specific resources. Lastly, another benefit is the historical view of cost optimizations over time.

Our customers told us that, when it comes to cost optimization, visualization is indeed useful - but cost optimization is only achieved by taking trustworthy actions.

Current cost optimization native solutions from AWS, Azure, and Google offer cost recommendations that require humans to review and decide if they are safe. This is not scalable for most organizations that are looking to automate actions they can trust. Otherwise, organizations will have to leverage their smartest and most skillful staff for cost optimization efforts instead of using them for more critical business initiatives.

Originally posted on Turbonomic's Blog site