Kubernetes for Data
Kubernetes (K8s) is revolutionizing how applications are distributed, operated and scaled. Its proliferation in the enterprise causes longer container life cycles with more persistent state, which requires extending the K8s operating model from compute to data. Scaled stateful workloads and infrastructure churn drive a much wider dynamic range of requirements into data storage, and the lack of a storage resource provider that can keep promises across the full data life cycle is increasingly becoming a major problem for enterprise K8s.
Datera was designed from ground up with a K8s-like operating model in mind, and can seamlessly extend K8s to data. It is an application-driven data services and management platform (with block and object storage as initial services) that combines enterprise performance and features with a cloud-like operating model. Datera brings data policies to life with policy-driven live data mobility across a broad range of servers, availability zones (AZs), data centers, and private/public clouds, optimizing the performance, resilience and cost of data for every application.
With its declarative application policy (intents), Datera immunizes applications against data infrastructure churn. While this brings value to any compute ecosystem, its strong affinity to K8s and its ability to let K8s keep its promises make this combination particularly potent.
Deep runtime integration ? Rapid deployment
Datera is disaggregated scale-out storage over standard protocols (iSCSI, S3, NVMe-oF Alpha), combining compute platform/framework flexibility with rapid deployment velocity and access to data from anywhere. Datera gives K8s enterprise customers the peace of mind of a future-proof data services platform that is ready for diverse and demanding workloads - as K8s continues to proliferate, it's only a matter of time until it will containerize high-end workloads, too.
Datera’s CSI driver deeply integrates with the K8s runtime. It allows deploying entire stateful multi-site K8s clusters with a single K8s command, and pushing application-specific telemetry to the Datera policy engine, so that it can intelligently adapt the data fabric.
Datera gives IT a private/hybrid cloud data platform to consolidate both traditional enterprise and modern cloud-native workloads. IT operators gain the flexibility to plan, deploy and scale their compute resources independently from their Datera storage resources, while application owners can self-service and consume infrastructure as they go.
Customer Use Case #1
A Fortune 100 multinational financial services company re-platformed their infrastructure from Fibre Channel arrays to hyperconverged infrastructure (HCI), reflecting their agile business needs, but is running into typical HCI pitfalls by not being able to plan, operate or scale compute and storage independently. With a growing deployment, they are increasingly struggling with unpredictable storage services because of noisy compute neighbors, stranded capital in underutilized CPUs and/or storage media, and skyrocketing licensing cost.
With Datera, this Fortune 100 customer is moving its data center to a disaggregated, distributed, and independently scalable (virtualized and containerized) compute and storage architecture that continues to minimize its storage cost, while lowering its I/O latencies by more than 90%, multiplying its compute and hardware utilization, and reducing its total cost of ownership (TCO) by up to 75%.
Deep policy integration ? Day-0 zero ops
Datera seamlessly adds persistence to K8s stateful workloads through labels that let application owners define declarative policy (application intent) for persistent volumes (PVs) in K8s manifests, such as access protocol (iSCSI or S3), performance (throughput or IOPS), availability, data/media placement, snapshot and clone policy, data reduction techniques, lifecycle management, backup-aaS, and other service level objectives (SLOs). Fine-grained policy gives application owners control over their storage personalities and behavior (within their tenancy limits).
Datera organizes policy into a hierarchy of application classes (templates or blueprints) and application instances that can be exposed as custom resource definitions (CRDs) to K8s consumers, creating a class-/intent-based storage provisioning scheme that integrates seamlessly into K8s storage classes.
Declarative policies (intents) are invariant, making them portable, composable and scalable across workloads, tenants and organizations. Datera's application intents immunize applications against data infrastructure churn, let K8s keep its promises, and extend zero ops to day-2 and beyond.
Co-orchestration of compute and data ? Day-2 zero ops
Datera architected an adaptable data plane by implementing a mechanism that autonomically moves data across the Datera cluster, driven by current and future application intent.
Datera’s live data mobility brings policy to life - it lets application consumers and Datera storage operators independently adjust their needs as they go. Live policy can be applied via application classes at scale, or for individual application instances (“snowflakes”), and Datera executes these changes non-disruptively by adapting the data fabric (all of its active and passive elements).
Application intent gives Datera awareness of how its data is being used, so that it can autonomically operationalize it by applying governance constraints (like application anti/affinity rules, etc.) and data center state (like network topology, availability zones, hardware substrate changes, failures, etc.), allowing K8s to co-orchestrate compute and data across their entire lifecycle. For instance, Datera can transparently lift mission-critical K8s applications to carrier-grade resilience by intelligently replicating their data across AZs.
Application intents hold sway above data center churn, so K8s consumers can scale applications or adjust policy as they go, and IT operators can supply resources or adjust governance rules (e.g., tenant storage quotas) on the fly, all independently from each other, allowing zero ops across the entire application/data life cycle. Customers can guess their persistence requirements wrong - and Datera will always give them an operational out.
Customer Use Case #2
A tier-1 e-discovery services company delivers a tiered SaaS offering from Microsoft SQLserver database (DB) farms, running on VMware and bare metal, spread over four EMC product lines corresponding to the service tiers. Scaling their business, the exponential complexity of running a matrixed SaaS/storage setup, combined with the slow manual delivery and high OpEx/CapEx of arrays, became unsustainable.
With Datera, this customer consolidated four EMC product lines (VMAX, VNX, XtremIO and VPLEX) onto one mixed cluster. Their SaaS requirements fluctuated and were difficult to project, so Datera let their DB admins define service tiers as application intents, while their storage admins started with hybrid flash nodes, and can add all-flash and/or Intel Optane nodes on demand. This resulted in 98% less management time, tripled performance, and accelerated service and infrastructure delivery by more than 10x.
Mapping K8s key concepts to Datera
Datera was designed from ground up with the same operating principles as K8s, so it's no wonder that K8s concepts naturally map onto Datera concepts and seamlessly extend the K8s compute fabric to the Datera data fabric. Unlike other storage products for which policy-based operation like in K8s was an afterthought, Datera needs no “backwards patching” in its CSI driver to placate operational discontinuities.
Some key examples how Datera maps and extends K8s concepts:
The following two sections describe two of these concept mappings in more detail.
Namespace to tenancy mapping ? Governance for K8s data
As K8s matures from dev to ops and from stateless to stateful workloads, its demands on operational resilience increase significantly. Datera meets these demands by mapping K8s namespaces to its multi-tenancy, dynamic quota and node pool architecture, binding resources to namespaces, and applying a comprehensive governance concept to safely operationalize policies, while augmenting K8s pods with hard data segmentation and isolation.
Datera complements K8s with robust data governance, and effectively sandboxes storage resources to safely run modern cloud native workloads at massive scale, while maintaining 100% compatibility with traditional enterprise applications.
Live persistent volumes & claims ? Efficient K8s cluster setup & scaling
Datera provisions PVs “live”, and leaves them dormant through thin provisioning, until applications push the corresponding persistent volume claims (PVCs), at which point it scales the PVs as the applications consume them. This way, Datera can respond just-in-time to changing application requirements, without employing expensive resource pre-/overprovisioning and caching schemes (e.g., in client-side proxies) that conventional storage systems need in order to placate discontinuities in the operational model.
This makes Datera a true just-in-time persistence resource manager that complements and completes enterprise K8s solutions.
Intent-based multi-cloud control plane ? Universal policy plane
Intent-based provisioning gives Datera awareness of how data is being used, making data and its behavior portable, composable and scalable across diverse operating environments. It’s hyperscale control plane (fully symmetrically distributed, floating control processes) can then enforce a single policy universe across a wide range of distributed operating environments, ultimately including and equalizing private and public clouds.
This allows Datera to intelligently manage data at scale, based on its behavior, while hiding underlying infrastructure complexity and churn.
For instance, Datera brings game-changing operational sophistication to DPaaS. While conventional solutions can only backup passive bits, and need to restore them into similar hardware configurations to maintain their behavior, Datera retains data together with its behavior (intent), so it can intelligently re-hydrate it across a wide range of environments. This also allows implementing powerful cloud-integrated, intent-based DPaaS solutions.
K8s and Datera together can make global compute and data placement decisions that drive optimal economic outcomes.
Multi-site data plane ? “Frictionless” data surface
Datera’s data plane is based on a lockless transactional distributed coherence protocol that delivers enterprise-class low latencies and game-changing elasticity over standard Ethernet/IP backends. This creates a data surface with predictable enterprise performance, and consistent I/O service latencies of less than 40μs, that can linearly scale throughput with each additional storage node, spanning across one or more data centers.
To avoid segmenting this “frictionless” data surface with L2 network overlays, Datera seamlessly integrates into modern data centers with flat virtualized L3 networks (a single flat IP address space), by allowing each of its nodes to act as a BGP peer. This lets Datera streamline network configuration and management, and makes its data services continuously available by letting them float across a single flat data surface (behind service ports with fixed virtual IP addresses) with practically instant session failovers (via active/active multi-node multipathing). As a result, K8s and Datera can control compute/storage placement independently of network topology, which significantly simplifies compute/storage fabric connectivity.
Datera underpins K8s with a future-ready data foundation that can keep up with new technologies like flat high-speed networks (e.g. 100GbE), low-latency protocols (e.g. NVMeoF), and low-latency storage media (e.g. Intel Optane), and that can keep up with diverse and demanding workloads as K8s continues its landslide success, and may eventually absorb enterprise workloads, too.
Customer Use Case #3
The leading travel metasearch engine and #8 hyperscaler in the world had problems scaling their rapidly growing and seasonal SaaS business on rigid storage arrays that imposed severe operational discontinuities, such as prohibitive planning, deployment and management overhead, and accelerating obsolescence cycles with disruptive data forklifts. They needed a highly automated data services platform that could scale transactional workloads, too.
Re-platforming to K8s and Datera let this Fortune 100 customer drive eight million transactions per second across large, elastic NVMe flash server farms, lower I/O latencies by up to 90%, cut power draw by over 50%, double storage hardware utilization, and practically eliminate storage hardware maintenance. It now runs its SaaS portfolio in K8s pods that float on a single virtualized L3 data surface, giving it a single pane of glass (control, compute and data) across multiple giga data centers, and reducing its TCO by 75% for typical workloads.
Turnkey enterprise K8s ? Game-changing customer value
Datera combines enterprise performance and features with a cloud operating model, and it complements continuous consumption of storage as-a-service with continuous provisioning of storage hardware (HW-aaS/IaaS). K8s and Datera can converge a set of heterogeneous compute and storage servers into turnkey enterprise K8s pods that run a rich mix of applications, from modern cloud-native to traditional enterprise workloads.
Datera can federate K8s pods with a single data surface and a uniform policy plane that effectively equalizes the underlying environments, or intelligently extends its data services to a manifold of third-party environments, incl. private/public clouds. It can intelligently broker between these environments, autonomically exchange compute and data among them, and optionally create exponential business value with innovative services on top of them (e.g., a smart DPaaS that can broker data based on its intended behavior).
K8s and Datera together can build future-proof turnkey K8s solutions that continuously deliver compute, persistence and infrastructure as autonomic end-to-end services, and bring unique, game-changing value to enterprise customers.
Summary
Datera decouples storage consumption from provisioning, and continuously brokers between them. It transforms the storage operating model from homogeneous, rigid point-in-time systems to composable and scalable just-in-time data services that empower both application owners and storage operators to independently optimize their needs:
- Application owners: can self-service by defining composable and scalable data services as they go – and change their intents anytime; and complementarily
- Storage operators: can independently scale hardware resources by adding servers as they go – and select the best fit at any time.
K8s and Datera bring a rich private/hybrid cloud platform to the data center that automates the entire IT life cycle, from planning through obsolescence. By letting each stake holder move at their own pace, K8s and Datera can deliver unmatched simplicity, efficiency and business value to enterprise customers, reducing TCO up to 75% for many typical use cases:
Datera is proven in production at scale with industry leaders, and has a 100% technical win rate against large incumbents. Three customer use cases can further illustrate some of the unique business value that Datera brings to K8s and private/hybrid clouds:
