Unveiling Our IT Cost Reduction: Reducing from $27.8k/month to $181.44 for AWS Billing

Introduction

Managing IT costs is a critical aspect of maximizing efficiency and profitability. As the Head of Engineering at Nas Company, I have spearheaded efforts to revise our usage of IT services and successfully reduce 70% of our monthly IT expenses.

One of the most significant savings happened from Amazon Web Services (AWS). In this article, we will explore how Nas Company achieves substantial cost savings through AWS while highlighting the importance of avoiding costly mistakes in tech usage.

Situation

I joined Nas Company on Oct 25th, 2022 as the Head of Engineering. Like many start-ups, we faced the challenge of balancing rapid growth with limited resources. The increasing demand for our services put a strain on our IT infrastructure, leading to higher operational costs. We recognized the need to optimize our technology stack and find innovative solutions to reduce expenses without compromising performance.

Complication

Among all the costs, AWS is one of the most costly services. Revising the usage of AWS service will be the most effective step to take.

The company started using AWS in Sep 2019. After 3 years, the whole infra become less efficient due to the rapid change of business and people. Without careful oversight and optimization, AWS usage can quickly lead to unnecessary expenses. Inefficient resource allocation, idle instances, overprovisioned storage, and suboptimal utilization of cost-saving features contribute to inflated bills. Understanding how the cost is generated and avoiding these common pitfalls is crucial to achieving cost efficiency.

Resolution

After several months of effort, we successfully reduce the AWS bill from 27.8k to an acceptable level.

Following are the major operations to save out cost.

Optimise performance and allocate proper resource

AWS provides great flexibility to scale up service vertically and horizontally. However, it’s easy to make mistakes when blindly scaling up service without figuring out the root cause of performance issues and optimizing the solution. Also, people may tend to create new services using existing instance configurations without reviewing and planning based on the new use case.

After improving the logic of our services and allocating resources based on CPU/Memory/Network usage, we are able to

1. Reducing the stress to service by applying index, cache, proper logging, etc.
2. Downgrade the type of instances to a lower specification
3. Scale down the number of instances and enable auto-scaling.
4. Downgrade the DB to a lower-performance type with the same storage.
5. Remove some unnecessary high-availability support for certain services, such as multi-available zone support, DB backup, log retention, etc.

Clean up idle resources

After running for 3 years, there are some services deprecated. However, they may not tear down completely.

1. We have a staging environment, which is not being used anymore.
2. Some businesses are not operating anymore. But the service is still running.
3. When shutting down some projects, part of the resources are not released completely.
4. Migrating to a new tech stack but the old one is not cleaned.

Share reusable resources

AWS provided a lot scaffold to set up your service so you don’t need to do everything from scratch. But if we don’t know what is exactly happened, there could be some reusable resource being created multiple times

1. Dedicate application load balancer for each project
2. Multiple NATs created for each subnet when applying some cloudFormation template
3. Creating new elastic IP for each beanstalk project
4. Unify tech stack - Beanstalk, EKS, Lightsail
5. When setting up services using multiple stacks, it will be easier to waste resources due to harder to customize among different solutions.

Choose more cost-effective solutions

AWS provides some more powerful and easier solutions. But if we are not 100% familiar with how it works internally and do optimization, it will be easier for you to make mistakes.

1. AWS Elemental MediaConvert - This is a quick solution for us to support streaming videos in different formats. However, the cost of this is significantly high. We changed the configuration to only use the basic tier at the moment. But it will be more efficient to set up your own converting flow.
2. Monitoring - We used LogicMonitor and Datadog as visibility solutions. However, the platforms could charge a lot if not configured properly. Besides, they will massively load data from Cloudwatch API and increase AWS bills. In the end, we set up our own Prometheus and Grafana, which is only costing 10 USD per month. Meanwhile, it provides the latest LGTM stack which may bring in more benefits to us.
3. S3 Tiers and Intelligent-Tiering - According to usage estimation, we could find a better way to store resources in S3. For the resources with unpredictable access, Intelligent Tiering could provide a simple optimization.
4. Be more careful when using new solutions, such as EKS and Lightsail if you are not familiar with them or have a strong reason.

Apply saving bundles

If your usage has been optimized and you foresee there won’t be big changes, you can consider committing to a long-term contract to save some cost.

1. AWS Computing Saving plan - Compared with reserved instances, it provides the flexibility to change instance type freely or switch to any other region. Also, the plan can cover multiple computing services such as EC2, EKS, DB instances, etc.
2. Data Transfer cost and CloudFrount savings bundle - AWS will charge you when traffic is moving from your cloud to the internet. By optimizing the traffic route, you may be able to avoid some unnecessary costs. You can save the CloudFront charges on your AWS bill when you make an upfront commitment.

Pay through service resellers

We didn’t move into this in the end. But there are some AWS resellers that can provide savings by changing your billing through them. From their side, they will own a resource pool with cheaper resources and allocate them to the vendors. I was approached by https://www.cloudkeeper.ai/ to know this info.

Enrol in the AWS activate program

After the above operations, the latest estimation of our monthly AWS cost is around 7k-8k. At this point, our investigator introduces the AWS Activate program to us. The program provides company credit to encourage startups to explore different tools and accelerate their journey. In the end, we are able to get enough credits to cover bills. The current 181.44 is from CloudFront Saving Bundle which is not covered by this program.

Conclusion

Cost saving is a continuous journey and there are still a lot of to-do items from our side. If you haven’t looked into a bill for a long time, maybe this is a good time to do so and get some instant effect.

In the next article, I will share more savings from other IT subscriptions and share more pitfalls.