Buying and Selling Image Data, A Practical Solution for VNA, AI and Clinical Research

Every now and then I am asked about, or I read an article about, someone selling massive amounts of data to one of the big companies involved in AI and/or clinical research. And, when you have a lot of data the obvious thought is, I want some of that free money! 

As a thought-exercise lets look at some of the realities of moving more than a PetaByte of image data. A PetaByte is 1,024 TeraBytes or 1,048,576 GigaBytes. Many, but not all, VNA’s store data in a proprietary format that is like DICOM but is not a straight .dcm file. This means that to get data out you can’t simply copy the file but instead, have to do a DICOM transaction. Whether the data is stored as a DICOM file or not, both data transfer types require the additional step of de-identification, so although DICOM transactions add time, it is not the end of the world.

In my experience a single server tops out at somewhere around 15,000 studies per day, which is approximately 500GB. So, less than 5/100th of 1% of that PetaByte of data is moved. Doing the simple math, 10 servers dedicated to nothing but copying this data, ignoring a penalty for de-identification or additional compression will move 1 PB in 209 days. I submit that this is not practical and there is a better way.

First, we are looking at the problem from the wrong end. Whether for clinical research or training an artificial intelligence (AI) engine, it is likely that the buyer doesn’t want ALL data, but is instead looking for very specific use cases. In particular, what diagnosis are they trying to research or train? Instead of dumping billions of images on them and letting the buyer sort through it all, I propose that preparing a system that can provide a targeted data set, instead of a generic query like “send me all chest x-rays” will go a long way in cultivating a long-term relationship. To achieve this targeted approach, we must begin at the report level, not the images. 

To initiate this targeted method, we would build a database that holds all reports (not images) for the enterprise. Simply start with pulling an extract from the EMR for all existing reports, and then add the HL7 or FHIR connection to get all new reports. With the reports parsed into the database any future questions or required data parameters, can be answered with relative ease. This database can then be queried for the specific data set desired, the output of this query would be accession number, patient ID, date of service and procedure description. Obviously, there should be a 1-1 relationship between accession number on the report and the images in VNA, but the additional output data will help if there is an accession number mismatch.

Now, armed with this export, a savvy VNA team can, instead of drowning the buyer in millions of “chest x-rays”, provide them with the several thousand “non-smoker, males, between the ages of 15-30 with a lung cancer diagnosis” files they actually want. I am not a researcher, but I suspect that this type of fine-tuned data capture would be more beneficial to them, as well as much easier to service from the VNA, in effect a win-win for all involved.

To view all of my articles and posts visit me at kylehenson.org