Introduction to Kubeflow

If we talk about Machine Learning, it is an application of Artificial Intelligence that provides the system to learn automatically and improve from the experience without being explicitly programmed. It is everywhere and is made up of different pieces. When building a model, we may not think about the infrastructure needed to train and serve it, there Kubeflow comes into the picture.

In this article, we will be going to talk about Kubeflow, its general introduction.

Now the main challenge which every machine learning engineer face is the maintenance of the machine learning systems. It is very challenging to manage all the application, platform and resource considerations and machine learning apps have a different footprint from the other web or mobile deployments. For example, a training phase is resource-intensive, while an inference phase is lightweight and speedy. For this we also need tools and frameworks to run all these pieces. Existing solutions that are great for building applications in other domains don’t capture the variation needed for machine learning.

So Kubeflow is all about creating a standard for deploying enterprise machine learning apps, not just development or production, but their entire lifecycle.

What is Kubeflow?

Kubeflow is an open-source project that is built on top of Kubernetes, a platform for running and orchestrating containers. It contains a curated set of compatible tools and frameworks specific for machine learning. The primary goal of Kubeflow is to make it easy for everyone to develop, deploy and manage portable, scalable machine learning. It is built in such a way that it runs consistently across different environments, and this is how Kubeflow operates.

?Three principles of Kubeflow

Kubeflow is built around three principles which are as follows:

1.??????Composability

2.??????Portability

3.??????Scalability

Composability: It means that we can choose what’s right for the project. This can be looked this way, people starting out in machine learning tend to be focused on the end result of the model, but there’s always more than just building the models. The building of the model itself has many stages, and each stage is made up of different building blocks that we have choose. It doesn’t matter whether we have special requirements for our machine learning workflow, we can even use different versions of TensorFlow for different parts of our project. The Kubeflow here does is, it extends the Kubernetes’ ability to run independent and configurable steps with machine learning specific frameworks and libraries.

Portability: It means that we can run all pieces of our machine learning project anywhere we are running Kubeflow. So, it handles the platform specifics, and we can just focus on our model. Just write the code once and let Kubeflow handle all the abstraction so that we can run into our systems and in the cloud.

Scalability: It means that our projects can access more resources when they are needed and release when they are not. Every environment can have different computing resources like CPUs, GPUs and TPUs, but we may also need to scale across teams and running experiments. It leverages the abilities into Kubernetes to maximize available resources and scale with as little manual effort as possible.

?SUMMARY

If we see Kubernetes, we can probably see how it enables Kubeflow to do all of these, but there is also lot of complexity in knowing how Kubernetes works, which can require expertise in many additional concepts. Kubeflow is designed to let Kubernetes do what its good at and allow us to focus on defining and running ML workflows, including managing data, running notebooks, training models, and serving them.?