Protecting against Cloud Bill Shock — Beware the Untagged Filth

In this article, I explore different ways various teams should be looking at protecting against bill shock and the ways you can clean up costs along the way.

Why did I write this article?

Over the years, I’ve seen numerous products hit the market discussing ways to analyse the costs and bills that have come about AFTER bill shock has occurred. It’s always an afterthought, and I never understood why, considering the ways you can happily track your cloud service usage.

I wanted to talk about the ways you can, and probably should be protecting yourself against the shock. Selfishly, I’ve created ways in hava.io to help provide views of environments to help cover what some of this architecture discusses.

Disclaimer: I founded and sell Hava.io, but there are other products that can help with this view as well).

Who cares about bill shock?

Obviously, the first answer that comes to mind is the CFO right? But is that all… no it’s not all, everyone should be concerned with the operational finances. Have we forgotten what FinOps is? Is where the finance teams and the operational teams work hand in hand. That is, just like when we saw the evolution of DevOps being getting the Dev people to talk to the Operations people so that things actually get done with full visibility of each other. So too, should this happen with FinOps, the finance teams, the developers, the architects and the Operations teams.

So, who cares? What are their titles and jobs? Or better yet… What are the companies that support the end customers and why should they care?

Here we go:

• Managed Service Providers
• Professional Services Companies
• Audit and Compliance Businesses
• DevOps Engineers
• Operations managers
• The whole dang finance team.
• Developers
• Architects <- This is important, they’re often charged with coming up with what the environment will cost, but then they don’t actually have responsibility for tracking it!! Shift Right on Architects.
• Auditors
• ITSM Peeps — You know, those change and knowledge management people.

And, whilst you should have a good MSP (Managed Service Provider) doing this for you, if not your central IT team… here’s what each of these peeps could be doing to protect against Bill Shock.

MSP to protect against Bill Shock Scenarios

My customer just told me that they had bill shock and they’re cracking it, how can I help? Whilst some tools may catch the increase in spend or particular culprit, others can help protect against this happening across the customers entire environment.

• Action — You create a custom view in your visual product of choice (I’d recommend Hava) for that particular TAG if you’re wanting to see how the application is changing over time and review major changes that impact cost. This is then a simple extract or link provided to the end customer showing them what you’ve reviewed as well as what they can further review.
• Action — You create a custom view that pulls back the environments which have the culprit resource type across all of the customer’s accounts. Then you produce a report illustrating what’s changed over time for that resource type and, using the export to list view, add up the cost in change and provide a report on the trending cost implications.
• Action — Setup Alerts so that when a customer starts using a new resource type which may impact cost, both the MSP and the customer are immediately alerted when that architectural change is made.

Professional Services to protect against Bill Shock Scenarios

Bill shock just occurred and a customer wants you to do a review. You give them a full view of all of their environments. You then also give them a list view of each environment indicating the TOP 5 expensive architectural components they are using in each environment. This can give the customer an immediate view or place to go exploring to cut costs.

• Action — You set up your tool of choice and ensure it keeps tracking so that after talking to the customer for the first 2 weeks, you can highlight the changes occurring in the architectures to them and highlight what of those may have an impact on cost.

• Action — You run an “Untagged Filth” check across their environments and pull back a report on things untagged which means, possibly unmanaged or orphaned. With some products like Hava (had to), it’s much easier to see the context of the overall environment to understand if it should or shouldn’t be running.

• Action — You run the “Pet Detector”. That is, look for any older cloud resource types. It’s likely that as you’ve modernised in newer environments, old environments may still be using old cloud resource types that are now expensive. You run a report to analyse that and see where the cost can be changed.
• Action — After you’ve got a good view of all of the environments you’re running, you create a custom view of the expensive resource types. You then build a report for your client to explain how they can modernise and move away from the expensive architectural approach being used if possible with a trade off between the professional services cost to change it and the cost to keep running the architecture that way. Basically, a tech debt report on older architecture.

What DevOps / Ops can do to protect against Bill Shock Scenarios

When doing a CI/CD pipeline run, call your tool of choice to get the list view of the architecture for that environment and pull back the 5 most expensive resources. List these out so that the developer/engineer can get a quick glimpse and ask “what”?

• Action — Set up a running documentation section in your wiki under FinOps that has the embedded custom views mentioned in Professional Services above. Enable the Operations Manager to have a view of the wiki page pinned and they can check it often to ensure they have a way to ask questions about what’s changing cost wise.

What ITSM can do to protect against Bill Shock Scenarios

When you’re running through change management processes and ITIL processes, it’s important to have ways to make sure you protect costs being introduced as a risk, not just security and change risks.

• Action — Setup a link between you <zendesk / servicenow / other> to tool of choice via API or list view csv extracts to enrich the CMBD with things like architectural cost and environment cost.
• Action — When a CAB/Change is requested, ensure that the team is able to show the change in cost in the non-prod environment so as to illustrate what the actual cost change is likely to be in production. This is not just a forecast, which other tools will provide, this is evidence that actual change in cost will occur.
• Action — Implement an “Architectural Decision Record” in your <confluence, sharepoint, doco, etc> and use Hava to provide an interactive diagram of the pinned resulting architectural change for an example environment.

That’s probably a big enough read right now for anyone still with me… but hopefully this gives you some good ideas on how to protect against bill shock, rather than waiting for it to happen.

Stay tuned, in my follow up posts, I’ll be giving more insight into real live examples that happened with many more videos.

If you’re keen on seeing where all the diagrams came from .. please don’t hesitate to trial hava by visiting?https://hava.io