Big Data Analytics in Healthcare: A Comprehensive Guide

Introduction

The healthcare industry is experiencing a data revolution. With the advent of Big Data Analytics, we're seeing a transformative shift in how healthcare providers use vast amounts of clinical data to enhance patient care.

In this article, I will try to cover the foundation of any big data analytics implementation project and go in-depth into the ETL process at the heart of any such implementation.

But first, a fun fact:

Did you know that hospitals produce an average of 50 petabytes of healthcare data each year, with 97% of that data remaining unused! The healthcare industry generates roughly 30% of the world's data.

What is Big Data Analytics?

Big Data Analytics examines large and varied data sets to uncover hidden patterns, unknown correlations, health quality trends, patient preferences, and other useful information.

In healthcare, this means analyzing complex and voluminous data collected from various sources, including electronic health records (EHRs), medical imaging, genomic sequencing, wearables, and more.

Okay, that sounds interesting, but I am (kind of) new to this. Can you please do an ELI5 for me?

Sure, let me try again! Let's start with 'What is data analytics?'

Imagine you have a giant toy box filled with all kinds of toys: cars, dolls, legos, puzzles, and so on.

Now, if I ask you to tell me what types of toys you have, how many of each type, and which ones are your favorites, you would look through your toy box, sort them out, and then tell me, right?

That's a bit like data analytics. It's the process of looking at a lot of information (or data) - like your toy collection - and making sense of it by sorting it, understanding it, and then making decisions based on what you find.

For example, a hospital network might examine its clinical data to determine whether the flu season has started, whether it is well into its peak, and whether it is about to end. Similarly, a toy company may examine its data about what toys people are buying more to decide which products to make more of.

The Importance of Big Data Analytics in Healthcare

Enhanced Patient Care

Big Data Analytics enables healthcare providers to offer personalized care. By analyzing patient data, medical professionals can identify the best treatment plans for individual patients, improving health outcomes. Or, they can focus on a segment of patients that need that extra push (a phone call, reminders, transportation help, social services visit, nurse visit, etc.) to stay healthy and stay on top of their preventative care.

Predictive Analytics

Healthcare systems use Big Data to predict and improve health and disease management and determine potential health crises before they become widespread (from pandemics to critical and chronic care management).

Cost Reduction

Analyzing healthcare data helps identify inefficiencies and over-utilization of resources, leading to cost reductions and better resource allocation.

Extract, Transform, Load

Data analytics software oftentimes aggregates data first before performing its magic! Bringing those data in, cleaning, organizing, and performing analysis requires ETL.

ETL, you said! What's ETL?

Sure, let's talk about ETL, which stands for Extract, Transform, and Load. This is a bit like organizing your toys.

• First, you 'Extract' all your toys from the toy box.
• Then, you 'Transform' them - maybe you put all the cars together, all the dolls together, and so on, and maybe even fix a few broken ones.
• Lastly, you 'Load' them back into the toy box, but now they're organized so you can find them easily.

In the world of data, ETL is a way of taking data from different places (like taking toys out of the box), changing it so that it's easier to understand and use (like sorting and fixing toys), and then putting it somewhere safe and easy to use, like in a database (like putting toys back in the box, but in an organized way).

TLDR ELI5

Data analytics is like understanding and learning from your collection of toys, and ETL is like the process of taking them out, organizing them, and putting them back in a way that makes them easy to use and understand.

Understanding the ETL for Project Implementation

The process of turning unstructured data into structured data (or at least into a format that can be analyzed) is a key part of many ETL processes.

The ETL process involves extracting useful information from unstructured sources and transforming it into a structured format, such as populating a database or data warehouse for analysis.

• Extract: This is where data is collected from various sources. The data at this stage is raw and unprocessed.
• Transform: This is the stage where several processes clean and organize the data. Scrubbing is one of these processes. Others steps include sorting, filtering, aggregating (combining data in a summary form), and more. Data scrubbing involves cleaning the data, fixing errors, removing duplicates, etc. is a crucial part of the transformation process.
• Load: Finally, the cleaned and transformed data is loaded into a database, data warehouse, or any other destination where it will be used.

Note: The "Transform" step is one of the most complex and time-consuming when implementing a new analytics solution for a large and diverse clinical dataset.

A Bit More On Scrubbers

We'll stick with our toy box analogy to explain scrubbers in the context of ETL (Extract, Transform, Load).

Imagine you're sorting through your toy box again. But this time, you find some toys that are broken, some that are really dirty, and maybe a few things that aren't toys at all, like a sock or a piece of candy.

You wouldn't want to keep these things with your good toys, right? So, you take them out, clean the dirty ones, fix the broken ones if you can, and throw away what doesn't belong.

In ETL, a "scrubber" works a bit like that. When data is 'Extracted' from its original source, it often isn't ready to be used right away. There might be mistakes in the data, duplicates, or irrelevant information - kind of like the broken toys, dirt, or things that aren't toys at all.

A scrubber is a tool that helps clean up this data. It goes through the data and fixes errors, removes duplicates, and gets rid of anything that shouldn't be there. This makes the data much cleaner and more useful, just like how cleaning up your toys makes your toy collection better.

So, scrubbers in ETL are important for making sure the data is accurate and in good shape before it's 'Transformed' (organized and modified to be more useful) and 'Loaded' into a new place, like a database, where it can be easily used and understood.

Transforming the Raw Data into Structured Data

As discussed above, the ETL process involves several essential sub-processes to prepare the data for analysis.

Now that we have scrubbed the dirty data and made it ready for transformation, there are several key sub-processes that happen next during the "Transform" step. Let's look at them one by one.

• Data Cleaning: This is where data is scrubbed and quality is improved. It involves correcting errors, filling in missing values, and removing duplicates. For example, if you have a list of patient addresses, this step would involve correcting misspelled street names or removing entries that are not valid addresses.
• Data Standardization: This process ensures that all data follows the same format and meets certain standards. For example, dates might be standardized to a format like YYYY-MM-DD, or names might be formatted to have the first letter capitalized.
• Data Normalization: This is about organizing data in a way that reduces redundancy and dependency. It's imperative in databases to ensure efficient storage and easy retrieval. For example, instead of storing a customer's full address with every order record, you might store an address ID that refers to a separate address table.
• Data Integration: Often, data comes from different sources and needs to be combined. This could mean merging datasets, aligning columns from different tables, or aggregating data. For instance, clinical data from different EHR/EMR systems might be integrated into a single dataset with a standard format.
• Data Enrichment: This involves adding additional relevant information to the dataset. For example, if you have a dataset of billed amounts for claims, you might enrich it with external data like claims paid vs. rejected by the Payer (Insurance companies). This type of dataset combination and enrichment can provide more context.
• Data Aggregation: This is the process of summarizing or grouping data. For instance, health trends per region or distributed by zip codes or number of hospital visits.
• Data Sorting: Organizing data in a certain order, such as chronological, alphabetical, or based on values like billed amount per patient or number of hospital visits per patient. This can make further analysis and querying more efficient.
• Data Transformation or Conversion: This involves changing the format or structure of data. For instance, converting a column of text dates into a date format that a database can recognize and use for time-based analyses.

Each of these sub-processes plays a crucial role in transforming raw, unorganized data into a structured, clean format that is ready for analysis and decision-making.

Load the Data & Run Analytics

The "Load"is the last step in the ETL process. It involves transferring the processed data to a storage system, which could be a database or, more likely, a data warehouse.

These are different types of storage systems, each suited for various needs and data types. Let's explore what each one is and how they differ:

• Database: A database is a structured collection of data. It's usually used to store and manage smaller datasets. Databases are great for quick, real-time queries and updates. They use a structured query language (SQL) and are highly organized, typically using tables and schemas to keep data consistent and maintain integrity.
• Data Warehouse: A data warehouse is a large storage system designed specifically for data analysis and reporting. It's optimized for reading, aggregating, and querying large amounts of historical data. Unlike databases, which are designed for real-time, faster processing, data warehouses are built to handle large volumes of data from various sources and are used for making strategic decisions.

Sources of Healthcare Data

• Electronic Health Records (EHRs): Digital versions of patients' paper charts, including medical history, diagnoses, medications, treatment plans, etc.
• Pharmacy Prescriptions: Information regarding medication prescribed to patients.
• Medical Imaging: Includes data from X-rays, MRIs, and CT scans.
• Wearable Technologies: Devices like fitness trackers and smartwatches that provide data on a patient's daily activity and health status.
• Genomic Sequencing: Analysis of genetic data to understand how diseases affect different individuals.

Where is the Dirty Data Extracted and Stored?

Good question. The short answer is Data Lake. Unstructured data is often housed in file storage systems like data lakes. Handling and processing unstructured data usually require more sophisticated techniques like natural language processing for text, image recognition for multimedia, and big data technologies for large-scale unstructured datasets.

A data lake is a storage repository that can hold a vast amount of raw data in its native format until it's needed.

Unlike a structured database or a data warehouse, a data lake can store unstructured, semi-structured, or structured data. This flexibility makes it ideal for storing everything from text files and images to traditional database records. Data lakes are often used for big data processing, analytics, and machine learning, where large amounts of varied data are needed.

Is Healthcare (Clinical) Data Dirtier?

Yes, healthcare (clinical) data does tend to be more complex and often 'dirtier' compared to other sectors such as finance or retail. There are several reasons for this:

• Variety and Complexity: Healthcare data encompasses a wide variety of data types, including clinical notes, lab results, imaging data, patient histories, and insurance information. This variety, combined with complex medical terminologies and coding systems, adds to the challenge.
• Unstructured Data: A significant portion of healthcare data is unstructured. Clinical notes, for instance, are often written in a free-text format, including abbreviations, jargon, and narrative text that are not easily categorized or analyzed.
• Data Entry Errors: Healthcare data entry often occurs in high-pressure environments (like emergency rooms) where speed is crucial, increasing the likelihood of errors. Additionally, different healthcare providers may use different systems and formats, leading to inconsistency and errors when the data is combined.
• Compliance and Privacy Regulations: The healthcare industry is heavily regulated to protect patient privacy (like HIPAA in the U.S.). These regulations can complicate data sharing and consolidation, often leading to fragmented data sets.
• Outdated Systems: Many healthcare systems still rely on older technology or have fragmented IT systems due to the slow pace of digital transformation in healthcare, which can further complicate data integration and cleanliness.

In contrast, sectors like finance and retail often deal with more structured data. Financial transactions, for instance, are usually recorded in standardized formats. Similarly, retail data (like sales, inventory, and customer information) tends to be more uniform and structured, making it easier to manage and analyze.

Challenges in Big Data Analytics for Healthcare Data

Data Privacy, Security, and Regulatory Compliance: Ensuring the confidentiality and security of patient data is paramount. Adhering to healthcare regulations like HIPAA throughout the process.

Data Quality and Integration: Collecting high-quality data from various sources and integrating them for analysis is more complicated than it sounds. Oftentimes, a lot of analysis and time is required to get all data sources identified, extracted, and transferred.

You may wonder, if an EHR/EMR system is a "structured" record of data, then where does the "unstructured" data infiltrate the clinical systems? How does clinical and/or healthcare data get dirtier, broken, and unstructured?

Valid point as to how structured data can become 'dirty' or 'unclean' over time. Let's explore this problem statement in depth.

Sources of Unstructured Data

Unstructured data often comes from sources that are not as neatly organized as databases. Here are some of the common sources:

• Text Files and Documents: These include Word documents, PDFs, emails, and text files, which contain a lot of valuable information but are not organized in a structured way like a database.
• Multimedia: Images, audio files, and videos are examples of unstructured data. They contain a wealth of information, but not in a format that can be easily read and processed like rows and columns in a database.
• Web Pages: Information from websites is often unstructured. This can include text content, user comments, links, and multimedia elements.
• Sensors and IoT Devices: These devices generate vast amounts of unstructured data, like readings, logs, or event alerts, often not in a format immediately suitable for analysis.

How Structured Data Becomes 'Dirty' or 'Unclean'?

Even data from structured sources like databases can become 'dirty' for various reasons:

• Human Error: Data entry mistakes, like typos or incorrect data input, can make data dirty.
• System Errors: Errors in data processing systems or transfer processes can corrupt data or introduce inaccuracies.
• Inconsistent Data Entry: Different standards for entering data (like using different date formats or address formats) can lead to inconsistencies.
• Incomplete Data: Missing values or partially recorded data can make a dataset incomplete.
• Duplication: Data can be duplicated due to errors in data entry or merging datasets from multiple sources.
• Outdated Information: Over time, information can become outdated but still remain in the system, leading to inaccuracies.
• Integration of Multiple Data Sources: When combining data from different sources, inconsistencies and discrepancies can arise, leading to unclean data.

Conclusion

Big Data Analytics in healthcare holds immense potential in transforming patient care, advancing medical research, and optimizing operational efficiencies.

As technology continues to evolve, integrating Big Data in healthcare will become increasingly vital. This will lead to more informed decisions, better health outcomes, and a more efficient healthcare system.

I hope this comprehensive guide gave you a foundational understanding of how Big Data Analytics is changing the healthcare industry landscape. All the best for your next implementation.

If you have any questions, please let me know in the comments. Thanks for reading.