Federated Analytics Changing The Paradigm Of Data Science

If you are someone who regularly deals with data you should be knowing in and out of the data collection process and also should be well aware of the mechanism how they are being processed, transmitted, analysed or used to make some auto predictions in the IOT devices. In this article I will be discussing about how Federated Analytics is taking over the traditional Machine Learning and kind of giving rebirth to Data science.

In today’s world, data is inherently distributed. With advent of IoT and increase in the usage of smartphones number of endpoints having data has increased exponentially. With IoT, data is generated and often stored close to sensors or observation points. In many cases, moving this data into a single location before it can be analysed can be a challenging proposition. And in some cases, data simply cannot be transmitted to a central location because of bandwidth constraints. In other cases, data movement is limited by governance, risk and compliance issues, along with restrictions imposed by security and privacy controls. So where does that leave us? If we can’t bring data together for analysis, we have to take the analysis to the data and that is where Federated Analytics Model comes into picture.

Before i discuss further on Federate Analytics I would like to set the context by walk you all through the traditional machine learning. The traditional machine learning approach consists of a central server to store data and train models. There are two ways then to use such trained models.

1. Building a data pipeline to make all data pass through a central server, which hosts the trained model used for making a prediction. The predicted results are then shared via a dashboard or used to initiate a service. The drawback of the approach is that all the inputs collected by sensors/devices present in the environment are to be send back to a central server and then processed results are to be send back. This bars the ability of a model to quickly learn and adapt according to the environment and provide real time results.

2. The second approach is to ship the trained models to devices directly interacting in the environment. A benefit of such approach is that prediction happens in the same environment from where the input is collected and is much faster. However, even in this model for continuous learning, the training data is to be collected at each device and then send back to the server where the model is retrained.

In a nutshell we can conclude that?the traditional approaches of machine learning are not equipped to deal with such vastly distributed data and train models on it.

Now provided we some gained some insight on the working mechanism of the traditional machine learning process, i will dig deep int Federated Analytics. Federated machine learning is an approach which enables us to get rid the complexities there were being faced by traditional machine learning by enabling the models to be trained at the device itself. These trained models are then send back to a central server where they are aggregated and then one consolidated model is send back to the devices. It utilises the concepts of distributed computing to maintain track of each of the models at the devices, aggregation and re-distribution of the models at each of the devices.

Federated analytics allows data scientists to generate analytical insight from the combined information in distributed datasets without requiring all the data to move to a central location, and while minimising the amount of data movement in the sharing of intermediate results. Under a federated analytics model, most data is analysed close to where it is generated. Through analytics, learning happens at the edge, in the fog/core, and in the cloud or enterprise data center, and collective, collaborative learning happens at a global level.To enable collaboration at scale, federated analytics allows the intermediate results of data analytics to be shared while the raw data remains in its locked-down location. When the shared results are combined and analysed, a higher order of learning happens, and the owners of the individual datasets have the opportunity to compare their local results against the results of analysing the combined pool of data.

Federated Machine Learning works In the following ways:

1. Ship predictive model to the device

2. Consume input and make a prediction. Observe action taken by user and store the differences as training data

3. Use this training data to improve the predictive models.

4. Send these re-trained models from multiple devices to a central server

5. Re-distribute and aggregate weights from all the different models to create one model

6. Ship back the re-trained models to all the devices

These steps are repeated in a loop to enable the process of continuous learning.

Benefits of Federated Machine Learning

1.?Data Security & Privacy: Since the training happens on the device and only models are transported back it gets rid of one of the major problems of storing large amounts of highly sensitive or personal data at a central location which is prone to attack by hackers.

2.?Real Time Prediction: Since the prediction happens on the device itself, this approach also gets rid of the time lag that occurs due to transmitting input back to a central server and then shipping results back to the device

3.?Offline Prediction: Since the models are present on the device, the predictions work even when there is no internet connection available. As long as a device is able to get an input the predictive models can be utilised to do their work.

4.?Minimal Infrastructure: This approach requires minimal hardware (hardware available in our mobile devices is more than enough) to run machine learning models and truly brings the power of machine learning to complete use.

I know it got a bit lengthy but I really hope, you would find this article useful.

Please do not forget to show you love by giving a like.

Would also like to have your valuable feedbacks through comments.

Regards

Subhadeep Pal