What are DevOps and DataOps?

Most ML models that are deployed and go live need to be operated, maintained and supported.

In traditional IT, software solutions are developed by one team and are then handed over to another team to do the operations and keep the lights on. This sounds logical at first, because every team can specialize in a narrow skillset in order to do their tasks better.

However, the traditional IT operations model has several major drawbacks. The most obvious one is that the development team and operations team have different KPIs:

• The development team needs to deliver a solution that pleases the users and matches specifications as quickly and with as little resources as possible.
• The IT operations team needs to minimize the costs to run and support the software solution; keep failures, bugs and downtimes to a minimum; and have fast error resolution times.

As a result, the development team does not have many incentives to deliver a solution that is easy to operate, maintain and support, because their sole focus is on the user and time to market. And consequently, the operations team sets ever higher criteria for deployment, to protect themselves from badly implemented and documented software that leads to a lot of additional work for the development team.

Often, this leads to unnecessary bureaucratic procedures and mistrust. Handoff meetings between both sides can turn into formal, frosty events with little knowledge sharing and a lot of paperwork. This, in turn, leads to less collaboration and discussion, which makes it hard for the development team to understand how software can be improved in the future in order to prevent problems in operations. A vicious cycle.

The main idea of DevOps is to put both sides into one team: hence the same team that develops also runs the code afterwards. The benefits of this approach can be quite impressive:

• There is no need for mistrust, as everyone is now responsible for the process of the software lifecycle.
• Solutions can be delivered faster and in smaller chunks, since more pragmatic approaches to deployment are used as and there is a more efficient way of organization.
• Because developers have to fix bugs and provide support for their own software, they have a big incentive to program clean and well-documented code and automated tests.
• Providing support and removing bugs from your own code is much less time consuming than doing so with code that somebody else programmed. Even if another person in the team did the coding, it is easy to ask the actual developer questions, at least in the first critical phase after going live.

DevOps

DevOps is usually combined with agile methodologies in software development. It is particularly suitable when there is a need for frequent deployments, in contrast to a fixed half-yearly or quarterly release plan. Therefore, many DevOps teams work under the continuous integration and continuous deployment (CI/CD) logic, with the ability to deploy code to production at any time. In such situations, cloud infrastructures are of huge benefit, as hardware and middleware resources can be switched on and off very easily. Moreover, dependencies between software applications during releases with other IT infrastructure teams can be limited. There are four architectural prerequisites in order for software for DevOps to make sense and be effective. These are that the software can be:

• Deployed
• Modified (because it is likely to be changed)
• Tested against defined specifications
• Monitored to detect problems automatically and, if possible, before they even happen

Usually, with DevOps the team aims to automate the procedures to test software solutions as much as they can. To achieve a higher grade of automation, there are many new software solutions that support the DevOps way of working. For example, tools like Ansible and Terraform allow you to create, share and deploy infrastructure as code, which means that a new project does not need to rebuild the entire infrastructure manually but can reuse and adapt previously built infrastructures. This is mostly relevant for cloud infrastructure environments.

DataOps

So what is DataOps then? DataOps builds on DevOps, as ultimately all ML models we build are algorithms that are ultimately embedded into software code. While DevOps was developed with a clear focus on the software engineering world, DataOps adds further aspects that are specific to ML products, with a focus on ensuring that data and model quality are well monitored and maintained.

A key technique of DataOps is statistical process control, which is applied to constantly track the data analytics pipeline. In this way every step in the data flow can be automatically monitored to allow for the early detection of problems with the data quality and the model output quality. At certain points in the flow, people might need to check regularly whether the data input and model output look reasonable and match their experience.

A further core principle of DataOps is to automate manual data preparation done by data engineers and make these processes as robust as possible to anticipate and prevent errors coming from varying data input feeds and make them ready for higher data volumes.

Last but not least, DataOps includes the management of the data used following data governance management principles; for example, by establishing a well-maintained data dictionary which allows people to understand data faster when problems need to be resolved during operations.

Implementing DevOps and DataOps

DevOps and DataOps are a paradigm shift compared to traditional IT. It is therefore highly recommended that companies redefine roles, processes and technologies before starting to implement DevOps and DataOps. The implementation typically needs a lot of management attention and change management, as changes are quite substantial for those companies that have been around longer.

We hope you enjoyed reading this article covering one of the frequently asked questions in data and AI, there are 149 more in our new book "The Ultimate Data and AI Guide"! 

Please visit our book page to get more free content: www.data-ai-guide.com

Your data & AI experts,

Alexander Thamm, Michael Gramlich, Alexander Borek

"This book is the hitchhiker′s guide to the data galaxy." - Dr. Diego Villuendas, Global Head of Data & Analytics at SEAT