DevScience and beyond; the next frontier in DevOps

In my previous few articles, I discussed lean philosophy, dual-track agile, DevOps and related metrics. Many of these concepts now are in some form included in SAFe 5.0 agile at scale framework. While each track of the agile processes is KPI driven, now most innovation includes some form of machine learning and artificial intelligence. Interestingly, to support both these we need a unified data strategy. 

KPI creation, tracking, monitoring, and control across processes are well established, but they are isolated in functional silos with no common source of truth, showing true and integrated functional progress status.

The data lake is a concept generally associated with raw data storage used for data science and machine learning is a tool that can be used to create a data repository that holds data from all functions, business tools and products. a combination of both ETL and streaming architecture can be used to ensure the consolidation of enterprise data in a common location for analysis and unified dashboard creation. This not only allows centralization of data analysis tasks to achieve economy of scale but also pay the path for more advance machine learning-based data science work that can uncover hidden insights. DevOps data coupled with business drivers provide a relationship between business and operation KPIs hence ensuring focus on epic items of most business value. Advance machine learning models with real-time complex processing engines coupled with rule engines are in use in mission-critical deployments. 

Almost every product these days comes with dashboard and analytics capabilities. consumers are getting used to real-time notifications, alerts, self-healing systems, preventive maintenance, personalized services and more, which all are powered by data and data analytics at the core. Many advance machine learning systems also curate data for itself. An architecture that leverages common data lake, specialized product data sources, stream analytics, machine learning models and real-time feedback feeds included models are becoming common in new generation products.  

For lean, agile and DevOps to prove its promise of speed to value, speed to market and speed of innovation, data plays a pivotal role. When we talk about data, it is about a data platform that includes the entire ecosystem of a data lake, datamart, analytics, dashboarding, data science, machine learning development, deployment, monitoring, and control.