How can you balance data cleaning and other ML tasks effectively?

It is also important to check that the data is skewed properly. One thing to remember is data collected overtime can change and thus the relationships data represents also changes. It's always good to visualize and test different hypothesis on your data.

While there are several tasks involved in assessing data quality, a good way to start is by thinking about top 5 things to assess that would give you the big picture of the data. This is necessary as assessing data quality can take a lot of time and effort. Getting the big picture will help target the focus. A good set of questions can be: 1. What proportion of data is complete? 2. What proportion of data would need to be discarded? 3. How is the data distribution? 4. Is the usable data sufficient for good results? 5. Is the data (collection/reporting) prone to biases (social/ethical)? With these questions, it is easy to select the method and tools to assess data quality and dig deeper if it is useful for the subsequent ML tasks.

Automating data cleaning can be great way to have reproducable and cleaner data. It can reduce efforts however any automated job should be monitored on a regular basis to make sure that the quality of data is consistent. Some validation checks can be introduced that run on your cleaned data to make sure it's staying consistent over time.

Iterating on your data cleaning can sometimes feel like having to start all over. A few things that might help you maintain it in a more low effort way: - Have a notebooks just dedicated to data exploration. It can be a notebook full of summaries and plots. These can live outside of your regular ML pipeline. - In your ML pipeline you can include a configuration file in the form of a yaml, json or ini file with all data preparation parameters. When you have to change something you only have to touch this file in stead of re-doing and re-testing your code. - Work with data contracts. The data can only change if that data contract between you the consumer and the the producer (data engineer) has been altered. This prevents unexpected changes.

Data cleaning has an impact in the long term of the project; it not only produces a clean dataset to work with but also helps select future technical methods and gives a good understand of the domain beyond ML technicalities. This produces a better ML solution that is technically, socially and ethically sound. For example, if you discover that your data contains information of only specific types of people, then scaling the solution to diverse population may make the solution biased. Or if you find that a significant text data starts with Hello followed by few more characters, they can be labelled as 'Greeting' without feeding these to a classification model. Spending too little time on data cleaning might not give these useful insights.