Data is the fuel for Machine Learning (ML) models. But how much data do you need and how good should it be? In this article, you will learn about the trade-off between data quality and quantity, and how to balance it for your ML projects.
Data quality refers to how accurate, consistent, relevant, and complete your data is for your ML task. Poor data quality can lead to errors, biases, and unreliable results. To improve data quality, you need to perform data cleaning, validation, and transformation. This involves removing outliers, missing values, duplicates, and noise, as well as checking for data types, formats, and ranges. You also need to ensure that your data is representative of the problem domain and the target population.
When working in healthcare, it's important to put data governance front and center. This is just as much about the people and the process as the data itself. Taxonomy is key, for example from the American Health Information Management Association (AHIMA) defines data governance as "The overall administration, through clearly defined procedures, plans, that assures availability, integrity, security, and usability of the structured and unstructured data available to an organization (AHIMA, 2020) Data as a lifecycle must be managed through capture, process, use, store, and disposal. Further, they define data quality and Integrity as "the extent to which healthcare data are complete, accurate, consistent, and timely throughout its lifecycle. "
Data quantity refers to how much data you have and how diverse it is for your ML task. More data can help you train more complex and robust models, and avoid overfitting and underfitting. However, more data also means more time, cost, and resources to collect, store, process, and analyze. You also need to consider the diminishing returns of adding more data, as it may not improve your model performance significantly beyond a certain point.
Determining the optimal balance between data quality and quantity for your ML models is not a straightforward process, as it depends on various factors such as your ML goal, data sources, model complexity, evaluation metrics, and constraints. However, there are some general principles to guide you in making better decisions. Firstly, prioritize data quality over quantity since it is easier and cheaper to fix data quality issues than to collect more data. Additionally, use data augmentation techniques such as cropping, flipping, rotating or adding noise to increase the quantity and diversity of your data without compromising its quality. Furthermore, use data sampling techniques such as stratified sampling, bootstrapping or synthetic data generation to reduce the quantity and complexity of your data without losing quality. Finally, employ cross-validation, regularization and feature selection techniques to prevent overfitting and underfitting and optimize your model performance with the available data.
In case of imbalanced data, i.e., where the instances of one class are few relative to the other, oversampling can be a very robust and effective technique. The instances pertaining to the minority class can be considerably increased by various oversampling techniques, such as the Synthetic Minority Oversampling Technique (SMOTE) and the Adaptive Synthetic (ADASYN) sampling. This helps the model focus more on the patterns in the minority class by generating similar synthetic samples using interpolation and neighborhood extrapolation. The 'imblearn' library provides a number of useful oversampling and undersampling algorithms that could greatly enhance the quality of the data and in turn, the model.
Using validation data while training can also aid in understanding the direction in which the data acquisition should be moved to. It will provide more insights in terms of overfitting and underfitting and based on that evaluation correct measures could be taken into account.
When measuring the quality of your data, you can use different types of data quality metrics, depending on the nature and purpose of your data. Completeness is the percentage of missing or null values in your data, accuracy is the degree of conformity to true or expected values, consistency is the degree of agreement across different sources, formats, or standards, relevance is the degree of alignment to your ML goal and target population, and timeliness is the degree of currency or freshness.
Measuring the quantity of your data requires understanding the complexity and diversity of the data. Common data quantity metrics include volume, which is the size or amount of your data in terms of bytes, records, or dimensions; variety, which is the number or types of your data sources, formats, or features; velocity, which is the speed or frequency of your data generation, collection, or processing; veracity, which is the degree of uncertainty or variability of your data due to noise, outliers, or errors; and value, which is the degree of usefulness or impact of your data for your ML goal.
To improve the quality and quantity of your data, you can use different tools and frameworks, depending on the stage and scope of your ML project. Common data quality and quantity tools include data quality tools for data cleaning, validation, and transformation (e.g. Pandas, Scikit-learn, TensorFlow Data Validation, or PySpark), data augmentation tools for generating new or modified data (e.g. Keras, PyTorch, or OpenCV), data sampling tools for selecting a subset or a synthetic version of your data (e.g. NumPy, SciPy, or SMOTE), and data analysis tools for exploring, visualizing, and evaluating your data (e.g. Matplotlib, Seaborn, or TensorFlow Data Analysis).
Data cleaning is especially important for projects revolving around image data examination. If there's too noise, image size inconsistency, or even different color modes (RGB vs. CYMK), your data will lead to the rise of biases that would hinder the data analysis process.