Cloud FinOps Spend Forecasting

Introduction

Forecasting of cloud spend is one of the most challenging areas to master for any firm utilizing the big three cloud vendors AWS, Microsoft Azure, or Google Cloud Platform (GCP). Worldwide end-user spending on public cloud services is forecast to grow 20.4% to total?$675.4 billion in 2024, up from $561 billion in 2023, according to the latest forecast from Gartner, Inc. (1). ?Cloud Ops spend forecasting involves predicting the future cost of running cloud operations. ?Similarly, cloud budgeting involves estimating, allocating, and controlling the financial resources dedicated to cloud services. Cloud forecasting and budgeting helps firms optimize their cloud budgets, avoid overspending, and align costs with overall corporate business objectives. Given the dynamic nature of cloud environments, where resources scale automatically and billing models vary, cloud spend forecasting requires a combination of historical data analysis, real-time monitoring, and predictive modeling. (Please read my article on Different Approaches to Cloud FinOps Modeling Tools which complements the contents of this article)

Reasons for Cloud Forecasting Variances

Many companies trying to forecast cloud spending have variances in the 10% to 20% range, which can add up to hundreds of thousands of dollars a month. Variances from forecast arise because of the following reasons:

1. Complexity of cloud cost estimation
2. Distributed nature of cloud assets
3. Unanticipated technical and operational issues
4. Lack of pricing transparency related to reserved instances and savings plans.
5. Human factors

Companies with less advanced forecasting practices are lucky if they get their spend variances are under 20%.? The firms with the most advanced FinOps practices typically have forecasts that are routinely within 5% of actuals (2). Highly accurate cloud spend forecasting is a time-consuming process and requires advanced automation as well as a diverse team member with practical experience.

Cloud Forecasting Methodologies & Models

There are a variety of cloud forecasting methodologies and models utilized by firms to forecast spending with cloud hosting vendors. Some of these methodologies and models are quite simple in nature and others are infinitely more complex. The following are some of the more common methodologies utilized in order of complexity from simplest to more advanced (2):

Naive forecasting is a basic method that uses previous spend history and assumes that future cloud spend will be the same as your previous. This method of forecasting is too simplistic for most firms where cloud spending can change dramatically as compute or storage demand fluctuates.

A static forecast is generated once at the beginning of a set period, such as 12 months. It is not updated until the period ends, when a new forecast is generated for the next period. This type of forecasting is only useful for companies with very predictable cloud spending.

A rolling forecast is generated at the beginning of a set period, and then is regenerated frequently either weekly, monthly, quarterly, and annually. Each time it is regenerated.

Trend-based forecasting uses past cloud history to as a basis for the initial forecast. However, it also incorporates some form of statistical analysis such as linear or multi-factor regressions to evaluate trends of past forecasting to facilitate prediction of future period spending patterns.

Driver-based forecasting, also known as multivariate forecasting, utilizes other business metrics and KPIs to help fine tune cloud forecasts. Adding a single or multiple business driver to cloud spend forecast can increase the accuracy forecasts, but only if they are thoroughly back tested to ensure accuracy. Cloud workloads that are scaled based on specific business KPIs are forecasted by applying the KPI growth on actual spending. This method will not be able to forecast resource workloads that do not exist in the cloud yet but are planned to launch at a future date.

Time Series Forecasting uses machine learning models like ARIMA, Prophet, or LSTM (Long Short-Term Memory networks) and can help forecast cloud spend by analyzing time series data of resource usage and costs.

Generative AI forecasts are complex deep learning models used to develop cloud vendor spend forecasts. As companies increase their investments in these forecasts their ability to identify trends and patterns increases as does their ability to forecast in a more granular manner. These models can be very time-consuming to develop and require significant computational, analytical & personnel resources to develop, calibrate and maintain.

A composite forecast combines several of the previously mentioned methodologies and models to achieve a more accurate forecast. In some composite forecasts the average of each of the multiple methods is utilized. In other composite forecasts, each method is assigned a weighting, and the final forecast is the weighted average of the various methodologies. ?A simple method for combining is to average the results given from multiple models.

Use Native Cloud Provider Tools for Forecasting

The big three major cloud providers offer tools that help with forecasting cloud costs:

• AWS Cost Explorer: Offers cost reports, historical data, and forecasting based on past usage. It can provide forecasts for up to 12 months.
• Azure Cost Management + Billing: Provides trend analysis, cost tracking, and forecasts based on resource usage.
• Google Cloud Billing Reports: Includes cost breakdowns by project, service, and region, with forecasting capabilities based on trends.

Use Third-Party Tools for Detailed Forecasting

Native tools have limitations so it is better long-term to use one of several third-party tools that can enhance cloud spend forecasting such as:

• Cloudability: Focuses on cost optimization and offers tools for forecasting based on historical data and planned business changes.
• CloudHealth by VMware: Provides detailed cloud cost management, budgeting, and forecasting capabilities.
• CloudZero: This platform?provides granular cloud cost visibility across flexible, business-centric dimensions. It helps organize cloud cost into unit cost metrics like cost per customer, cost per feature, without requiring tagging
• IBM Turbonomic: Is a software platform that helps firms optimize the performance and cost of their IT infrastructure, including public, private and hybrid cloud environments.

Please see my other article on New Cloud - Cloud Fin-Ops Modeling Tools Options for a more in-depth discussions on the topic.

Cloud Forecasting Challenges

Generating more accurate forecasts is one of the biggest challenges that Cloud FinOps team face. ?As companies grow their cloud presence can expand rapidly. For companies utilizing multiple cloud providers forecasting can be especially difficult due to the diverse nature of these providers. ?Forecasting involves estimating future events or trends. Forecasting accurately requires gathering the right historical data to improve the likelihood of a correct prediction. Budgeting for cloud services is especially challenging due to constant, often dramatic fluctuations in spending caused by the dynamic nature of the cloud. Other key challenges include (3):

Poor Cloud Visibility

Poor cloud visibility can hamper your cloud forecasting efforts. Cloud bills are complex and challenging to understand due to the large amount of usage and cost information that they contain.

Cloud Provider Pricing Models

Cloud providers offer a multitude of pricing combinations depending on the location, type of cloud resources, usage volume and other factors. The more common pricing models are:

• On-Demand Pricing: Pay-as-you-go models that fluctuate based on consumption.
• Reserved Instances: Long-term commitments that lower costs.
• Spot Instances: Discounted prices for unused capacity that can be interrupted.
• Savings Plans: Discount programs offered by providers for sustained use.

Multi-Cloud & Multi-Tenant Complexity

Tracking cloud costs across multi-cloud hosting providers is an incredibly complicated task. Poor resource naming conventions, tagging processes or inconsistencies among tags on different providers can make consolidating and analyzing data difficult. Making sense of tags or labels from shared or multi-tenant environments is also challenging.

Forecast Frequency & Duration

How often do companies need to update or recalculate cloud spending forecasts? The more frequently that it is done the more resources that are required. ?What duration of time do you forecast out to? Is 12 months, 18 months, 24 months or 36 months sufficient to accurately manage cloud spending. The farther out a forecast extends the lower the accuracy in the tail end of the forecasts. The most advanced cloud ops forecast update forecasts monthly or quarterly and do not extend beyond 18 months due to the rapid decline accuracy the farther out you go.

Allocation Processes

Cloud cost allocation is a financial process that identifies, allocates, and assigns costs across customer, business units, cost center and departments.? On an account-by-account basis, business team leaders in charge of products or projects need to be given a granular view of the cloud resources assets that they are utilizing. This increased visibility into spending month over month helps companies to close the allocation gaps and to improve cost savings. So, it is imperative that companies develop solid cloud cost allocation strategies.?

Cost allocations are especially important for large companies with multi-cloud providers. There are three main types of cloud costs that are usually allocated:

1. Costs from resources / assets that need to be directly allocated to customers.
2. Costs from resources / assets that can be directly assigned to internal business units, projects, or cost center.
3. Shared costs are allocated in equal proportions to different projects or business units.

Estimating the costs of cloud solutions is one of the key pain points for every firm transitioning to the cloud, from both a technical and financial perspectives.

Manual vs. Automated Forecasts

Many companies start out with basic manual forecasting processes utilizing simple rolling-based forecast with cloud billing and usage datasets loaded into large spreadsheets. ?And as time progresses, these companies partially automate forecasting processes using statistical and BI software like Tableau or PowerBI to analyze the ever-growing billing and usage datasets.

Eventually companies decide to invest in fully automated forecasting processes. ?They develop in-house tools or purchase external third-party software packages (such as CloudZero, IBM Turbonomic, CloudHealth and Apptio Cloudability) that fully automate the data collection, organization, and analysis of of millions of rows of billing and usage data across multiple cloud platforms. These highly automated tools can generate driver and trend-based forecasts more frequently allowing for faster decision-making from an operational perspective.

Inaccuracies

The better cloud FinOps analytics tools and software packages can factor in the variability, volatility, and fluctuations into forecast so that you have higher accuracy. ?Some cloud FinOp tools and models are affected more by outliers in the data such as short, sudden spikes created by an anomaly which can contribute to forecast inaccuracies. ?Forecasting algorithms work better with a longer history of cost data, whereas others can work with limited data points. ?The best approach for Fin Ops teams is to incorporate multiple forecasting scenarios and methodologies to try to mitigate or isolate the variances and aberrations in historical data.

Granularity

Some companies utilize a top-down approach to forecasting cloud forecasting while others use a bottom-up approach.

• The top-down approach generates forecasts for total cloud spend using future operational information and historical spend trend data. This forecast then is allocated to various business units and cost centers using historical proportions matched with future project expectations. A top-down methodology is useful for firms with little variability in cloud spend across department. Forecasting total cloud spend at a macro firmwide level does not allow you to monitor the variability that individual business units, departments or cost centers can experience and thus contributes to inaccuracy of your forecasts.
• The bottom-up approach focuses on the most granular level of cloud spend creating individual forecasts and is a preferred method for cloud forecasting. This approach allows for significant input from cloud engineers, infrastructure, technology, and finance teams and usually leads to the most accurate total spend forecasts. However, creating forecasts at a very detailed level can be a very time-consuming process, especially when you have large numbers of business units with small amounts of cloud spending. Bottoms up forecasting is best done with as much automation as possible, which is best done with third-part tools and software.

Strategies to Improve Accuracy in Cloud Cost Forecasting

There are several strategies that firms can utilize to improve cloud spend forecasting including:

1. Cloud budgets need to be monitored closely and regularly to keep track of unnecessary costs, idle resources, etc. There are a variety of cloud monitoring tools that make this process less time consuming
2. Make sure you thoroughly understand your cloud bill. Cloud costs are operational expenses and keep on fluctuating based on resource usage and demand.
3. Accurate cloud cost forecasting requires participation from developers, system operators, DevOps, finance, and other stakeholders who affect cloud costs spend decisions.
4. One of the easiest ways to fine tune cloud cost forecasts are to add or subtract a certain percentage from your last budget and continue to recalibrate it as you get more accurate information.
5. Cloud forecasting is not a one-time process but a continuous improvement process so continually review and re-iterate.

Conclusion

Forecasting of cloud spend is not an exact science and requires the right balance of methodologies, model, and resources. Forecasting cloud spend is challenging, requiring continued iterations and improvement to reduce variances supporting cloud cost optimization.?

Developing highly accurate forecasting requires companies to start out simple and improve over time by incorporating more advanced techniques and further automation. A bottom-up approach over time will help FinOps teams produce more accurate forecasts farther out in time.