What are the best practices for assessing data quality before cleaning it for Machine Learning?

In my years leading data science teams, a recurring challenge I've observed is the ambiguity around "good enough" data quality. The ambiguity often stems from failing to differentiate between "absolute data quality" and "relative data quality." Absolute data quality pertains to meeting global standards, while relative pertains to the adequacy of data for a specific model or analysis. For instance, while 95% accuracy in a dataset might be excellent for a general trend analysis, it could be catastrophic for a financial fraud detection system. Always ensure that data quality criteria align with the end application.

In my experience of working with data, the quality is of utmost importance. Before beginning to explore data, the analyst must make sure the data is relevant, up-to-date and in accordance with your company standards, keeping in mind the end goal of analysis. Take a broad spectrum overview of data before diving into it. Collect information about metadata and understand the data fields, there should not be any ambiguity. It's always best to be honest and discuss data quality issues with your team.

One of the biggest criteria for me is the sparseness of the data itself - is it just an n*n matrix with a super sparse values ? and the second one is reputation: does it come with reputable sources? How was the data obtained? And of course, the more of it the better - A bigger collection leads to better models later :)

When I work with data, I always prefer to look at metadata, it tells whether your data comes from first party, second party or third party. Metadata helps in understanding data format and its type. It also helps in identifying bias in data and helps in making sure if the sample is collected randomly considering whole population not just a specific part of the population.

To better understand the relationships between variables in the data. There are underlying relationships that can improve model performance can be learnt by using logarithmic, squared, nth-rooth or other transformations. These transformations should be used without changing the underlying information the data represents.

Using the missingno python package is a great first step to identifying missingness in data . It can help to identify systematic errors in data collection processes to improve data completeness.

A few things come to mind when considering how to build ethical ML algorithms: 1. Consider the objective of the algorithm and how it will live in the world. If your algorithm results in someone or some groups being treated unfavourably, be doubly careful about fighting bias in the data set. E.g. if you’re building a CV screening algorithm, make sure you strip out protected characteristics like age, gender, ethnicity out of the training features. 2. Understand the inherent bias in the data. E.g. if you find all top candidates went to a few specific schools or lived in a few specific postcodes, you might want to strip those variables out as well. 3. Representative sample- you want the sample to reflect the population it needs to work on.

For effective machine learning, evaluating data quality before cleaning is essential. Start by looking for missing data, extreme cases, and discrepancies. Review the balance and distribution of the data. In order to spot unrealistic values, one must comprehend the domain and context. To improve model performance, consider the usefulness and redundantness of features. Visualise data to find irregularities and trends. Verify the data using expected bounds or restrictions. Utilise statistical analysis to find anomalies. Make sure the data is impartial and representative. Effective data quality assessment streamlines pre-processing and ensures models learn from reliable information, leading to better ML outcomes.

How easy is it to make fake data for testing purposes ? How easy is it to deploy ML stuff that we are building? I mean, for real world applications, we need to be able to deploy our models at scale - this is a point that is always overlooked