Using Data Without Using it! A New Era of Data Science with?PySyft

A few months ago, I found myself stuck in a frustrating situation. I was working on a critical project that required access to a large dataset held by a partner organization. This data was essential to my research, but the organization had legitimate concerns about sharing it due to privacy risks, legal implications, and potential misuse. No matter how promising my analysis could be, the reality was that I couldn’t move forward without this data. This roadblock felt insurmountable.

As I dug deeper into potential solutions, I stumbled upon a concept that seemed too good to be true — performing data science without actually accessing the data. That’s when I discovered PySyft, an open-source library that enables exactly this. With PySyft, I could finally see a way to move past the barriers that had been holding me back.

PySyft allows data scientists to perform deep analyses on private datasets without ever needing to see or obtain a copy of the data. This breakthrough meant that I could work with sensitive information securely, while the data owners retained full control over their data. Intrigued and excited by the possibilities, I decided to dive into this new way of doing data science. This article shares what I learned along the way and how PySyft can revolutionize the way we approach data-driven research.

The Challenge of Data Accessibility in Modern Data Science

Data is the lifeblood of modern science and innovation. Yet, data owners often face legitimate concerns about sharing their datasets. These concerns range from the risk of data breaches and privacy violations to potential legal ramifications if data is moved outside its original silo. Traditional methods of data sharing often involve compromising either the security or the utility of the data, leading to a significant bottleneck in the progress of scientific research.

Introducing PySyft: Data Science Without Data Access

PySyft addresses these challenges by introducing the concept of Datasites — a secure environment where data can be analyzed without being accessed. A Datasite functions similarly to a website but is specifically designed for data. It allows data scientists to run analyses on data stored on the Datasite without downloading or viewing the data itself.

This capability is made possible through PySyft’s integration with secure multi-party computation (SMPC), differential privacy, and other privacy-preserving technologies. With PySyft, you can perform any statistical analysis or machine learning task directly on the Datasite, using third-party Python libraries, all while ensuring that the data remains private and secure.

How PySyft Works: A Step-by-Step Guide

1. Connecting to a Datasite

To begin using PySyft, the first step is to connect to a Datasite. Think of a Datasite as a web server for data. When you connect to it, you’re logging into a secure environment where you can perform your analysis.

Here’s a simple example of how to log into a Datasite using PySyft:

import syft as sy

# login to data_site
client = sy.login_as_guest(url="20.51.219.43:80")        

In this scenario, you’ve logged into a Datasite as a guest, which means you can explore the datasets available but won’t have direct access to the data.

2. Exploring Available Data

Once logged in, you can explore the datasets hosted on the Datasite. However, as a guest or even as a registered user, you won’t have access to the actual data. Instead, you get pointers to the datasets and their assets, along with metadata that describes the data.

client.datasets        

This command returns a list of datasets available on the Datasite, including descriptions and details that help you understand the nature of the data.

3. Developing and Testing Your Code

One of the most powerful features of PySyft is the ability to develop and test your code using mock data — artificially generated data that mimics the structure and characteristics of the real dataset. This allows you to ensure that your code works as expected before running it on the actual data.

For instance, if you’re working with a breast cancer dataset, you can develop your analysis code using the mock data provided by PySyft:

def average_radius_of_nuclei(data, labels) -> tuple:

"""Calculate the mean of `radius` feature, for benign and malignant diagnosis"""
y = labels['Diagnosis'].values.ravel()

mean_benign = data[y == "B"]["radius3"].mean()

mean_malignant = data[y == "M"]["radius3"].mean()

return mean_benign, mean_malignant

average_radius_of_nuclei(features.mock, targets.mock)        

This code calculates the average radius of nuclei for benign and malignant diagnoses using the mock data.

4. Executing Code on Real Data

Once you’ve verified that your code runs correctly with the mock data, you can submit it to the Datasite for execution on the real, non-public data. PySyft allows you to wrap your function with a special decorator that specifies which datasets and assets it should access.

@sy.syft_function_single_use(data=features, labels=targets)

def average_radius_of_nuclei(data, labels) -> tuple:
# function code here        

This decorator ensures that your function only accesses the data it’s supposed to, maintaining strict control over data usage. After submitting your code, you can retrieve the results without ever seeing the raw data.

The Benefits of Using PySyft

1. Enhanced Privacy and Security: By enabling data scientists to work with data without accessing it, PySyft significantly reduces the risk of data breaches and privacy violations.

2. Legal and Ethical Compliance: PySyft’s framework helps organizations comply with legal regulations regarding data privacy and sharing, such as GDPR, by ensuring that sensitive data remains protected.

3. Wider Access to Valuable Data: Data owners are more likely to share their datasets if they can control how the data is used. PySyft opens the door to a wealth of data that was previously inaccessible, driving innovation and discovery across multiple fields.

Getting Started with PySyft

To start using PySyft, simply install it via pip:

pip install -U “syft[data_science]”        

You can run a development server directly in your Jupyter Notebook or deploy it using Docker or Kubernetes, depending on your setup. Detailed guides and tutorials are available on the PySyft documentation site to help you through the process.

Conclusion: The Future of Data Science with PySyft

PySyft represents a paradigm shift in how we approach data science. By decoupling data access from data analysis, it enables a new era of innovation where privacy and security are not obstacles but integral components of the process. As more organizations adopt PySyft, we can expect a significant expansion in the availability of data for scientific research, leading to faster and more impactful discoveries.

So, if you’re a data scientist looking to push the boundaries of what’s possible while maintaining the highest standards of privacy and security, PySyft is your new best friend. Dive in, explore its capabilities, and be part of the future of data science.