What Is Cloud Asset Inventory?
On March 14, 2006,?Amazon Web Services (AWS)?launched?S3, the first public cloud service. Since then, cloud consumption has increased each year.?Gartner estimates that worldwide spend on public cloud services will reach $600 billion in 2023, up from $410 billion in 2021. At annual growth rates of over 20%, the trillion-dollar market size is only a few years away.
The Evolution of Cloud Offerings
Cloud providers foster that growth by continuously launching new products. Market leader?AWS?now offers a portfolio of over two hundred fully-featured services.
Two hundred may not seem significant until you start looking at the?SKUs for each service. Consider?EC2,?AWS' computing flagship service which comes in over 475 instance types with various combinations of CPU, memory, storage, and networking capacity. "And we're not even close to being done," said?AWS CEO Adam Selipsky during the 2021?AWS?re:Invent keynote.
With the emergence of containers and serverless functions, services have only become even more abstract, and the size of the individual deliverables smaller. And since cloud providers often match each other's portfolios, these trends apply to?Azure,?Google Cloud Platform (GCP),?DigitalOcean, and others as well.
A New Type of Technical Debt
On the enterprise side, adoption of new cloud services is driving forward innovation.
The flip side of that innovation is that companies now have an inventory problem. A recent podcast episode of Software Engineering Daily framed it well: Cloud inventory is a new type of technical debt, where you lose track of the assets running in your infrastructure and how they relate to your business.
The following graph render depicts how vast the problem can be. It shows the?AWS?infrastructure snapshot of a 200-employee startup with over 400,000 cloud resources:
What Is an Asset?
An asset can be a resource such as a virtual machine, database, or storage bucket. An asset can also be a VPC, subnet, policy, security key, or IP address. Assets also include third-party cloud services, like an identity platform or even channels on?Slack?or?Discord. Cloud inventory consists of the many discrete assets that form your stack.
IT?used to control asset procurement. Now, developers have liberal permissions to spin up new cloud resources. Reintroducing the old, rigid process of submitting?IT?support tickets would kill productivity.
If anything, companies have adopted new processes to iterate more quickly. Continuous integration (CI) and infrastructure-as-code (IaC) have enabled development teams to deploy more, faster. A startup with a hundred employees can easily have hundreds of thousands of resources, and larger enterprises millions.
New Operational Challenges
Platform teams responsible for infrastructure today deal with three new challenges:
If left unmanaged, inventory debt results in incidents that slow down development. Sooner or later, resources hit quota limits and result in the failure of new deployments. Misconfigured resources expose critical data. Cloud bills pile up from orphaned resources.
Tackling Inventory Debt
Reducing inventory debt is a preventative measure to avoid or prevent incidents before they occur, with fewer wasted engineering hours and lower cloud bills.
The first step to tackling inventory debt is to create a complete inventory of all assets. This requires collecting data about the state and configuration of each resource in every cloud account.
XOps Tools vs. Cloud Asset Inventory
XOps (DevOps, SecOps, FinOps, etc.) tools create inventories tailored for solving specific problems, such as performance, security, compliance, or cloud spend.
XOps tools only offer a retrospective, often with a narrow lens on a subset of inventory data. Their goal is to diagnose infrastructure issues and make adjustments to individual assets. They support a corresponding team (Dev, Security, Finance, etc.) and help "operate" and manage a single aspect of infrastructure.
A cloud asset inventory is different. A cloud asset inventory takes periodical snapshots of inventory data, creating full representations of your infrastructure at specific points in time. An inventory is a strategic, forward-looking tool that aligns infrastructure with company strategy and helps a company grow.
A cloud asset inventory is a meta layer that allows platform teams to analyze their infrastructure and track asset changes. It's different because it offers search features while being generic and extensible. Rather than supporting a single core use case like XOps tools, a cloud asset inventory prioritizes flexibility and exploration.
Building a Cloud Asset Inventory
A cloud asset inventory should include two types of data:
(1) Resource data, including but not limited to:
(2) Dependency data?that represents the relationship between different resources.
Modern cloud asset inventories use a graph-based data model, as it's better suited to capture dependencies than a traditional relational model. Resources are always connected to each other in some way. For example:
Dependencies are often nested and can run many levels deep. A graph-based asset inventory is particularly strong at exposing the hidden relationships between resources.
Cloud Asset Inventory Accelerates Growth
With an inventory in place, all teams now benefit from exploring the graph, either ad-hoc or by continuously running queries, to support a variety of use cases:
Resource data also aids in planning and forecasting for holidays or events (Black Friday, Tax Day, etc.).
But most of all, a cloud asset inventory is a foundation for increasing development velocity. Platform teams can give developers liberal permissions without accumulating inventory debt. Inventory data gives platform teams both visibility and control.
How Resoto Increases Development Velocity
Cloud providers already offer native inventory tools that support their own resources.
But platform teams support a wide range of assets across?SaaS?tooling, legacy on-premises assets, and also multi-cloud infrastructure.
Resoto is an open-source cloud asset inventory and alternative to closed-source native tools. Resoto is extensible through?plugins, so anyone can?build or modify a collector?to suit their needs.
In addition, Resoto adds unique capabilities that solve the three major problems platform teams face when working with cloud inventory:
Consider a scenario where a platform team announces a new cost policy. The policy requires that any?AWS?compute instance older than 24 hours with more than 32 cores be cleaned up.
Finding the instances that violate the policy is easy with Resoto's?search:
> search is(aws_ec2_instance) and instance_cores > 32 and age > 24h
Then, adding a?job?to automate cleanup after each new inventory snapshot is as simple as:
> jobs add 32_core_instance_cleanup
--wait-for-event post_collect
'search is(aws_ec2_instance) and instance_cores > 32 and age > 24h | clean'
And just like that, the team has automated the work of enforcing the new cost policy.
Start Now
Resoto enables platform teams to create measurable?KPIs they can track over time, build fleet-wide monitoring, and automate tedious infrastructure work.
Resoto is?open source?and free to use.?Install Resoto?today!
------------------
Thanks to Doris Houng , Anja Freihube , Lukas L?sche , Matthias Veit , Nikita Melkozerov , Raffaele Picca and Timothy Chen who reviewed drafts of this posts.
The post was originally published at?https://resoto.com?on July 28, 2022.
Cloud Native & AI Security @ xTriage by CloudEdge ?? | Public Speaker ??
2 年Interesting! I like
world
2 年????