Four Great DataOps Articles
Christopher Bergh
CEO & Head Chef, DataKitchen: observe & automate every Data Journey so that data teams find problems fast and fix them forever! Author: DataOps Cookbook, DataOps Manifesto. Open Source Data Quality & Observability!
It’s been a busy week for DataOps in the news. We’ve see nfour great articles published about DataOps this week. BlueHill Research, the Eckerson Group, Booz-Allen, and Lidl have all publish works on DataOps.
They all have a similar set of takeaways:
Julian Ereth at Eckerson Recommendations
We see that, like for many other sectors, DevOps [DataOps] holds great potential for business intelligence and analytics. Especially the management of complex DW solutions can benefit from a holistic approach and a defined deployment pipeline. However, there are many more other applications, like advanced analytics, where DevOps [DataOps] can help to generally establish standards and assure quality.
In summary, DevOps [DataOps] should not be reduced to its promotion of automation or its defined processes. It rather should be seen as an approach to create an environment that is focused on efficiency, quality, interdisciplinary and continuous improvement.
Toph Whitmore at Blue Hill Recommendations
Treat DataOps — specifically, the management of data workflows in/through your enterprise — as a discrete function. Establish ownership, endpoints, metrics, and benchmarks.
Recognize that data workflows are business workflows.
Deliver services across the ENTIRE DataOps value chain (curate, integrate, prep, analyze, act).
Alexander Bauer, a Data Scientist at Lidl Recommendations
Data science projects rarely fail because of insufficient modeling skills
Focus on business value, deliver ?good enough“ models first
Deliver in small increments that already provide value end-to-end, present in Sprint Reviews to all stakeholders
Manage stakeholers using a clear product vision, a user story backlog and release plans
Deploy as early as possible to ensure user acceptance, declare as ?beta“ mode
Build an infrastructure that enables agile development
Kirk Bourne and Peter Guerra at Booz-Allen Recommendations
An agile DevOps methodology for data product development is critical — we call this DataOps. DataOps works on the same principles as DevOps: tight collaboration between product developers and the operational end users; clear and concise requirements gathering and analysis rounds; shorter iteration cycles on product releases (including successes and fast-fail opportunities); faster time to market; better definition of your MVP (Minimum Viable Product) for quick wins with lower product failure rates; and generally creating a dynamic, engaging team atmosphere across the organization. In addition to these general Agile characteristics, DataOps accelerates current data analytics capabilities, naturally exploits new fast data architectures (such as schema-on-read data lakes), and enables previously impossible analytics. With a sharpened focus on each MVP and the corresponding SCRUM sprints, DataOps minimizes team downtime from both lengthy review cycles and the costs of cognitive switching between different projects. Mature data science capability reaches its full potential in an agile DataOps environment.
Use Agile and leverage “DataOps” — DevOps for data product development and …celebrates a fast-fail collaborative culture.