What are some of the best practices for data preprocessing and augmentation for deep learning?

It is worth noting that data preprocessing and data augmentation when done correctly are benefitial to all machine learning algorithms and not just deep learning. The objective of data preprocessing is to improve quality of data so we can optimise knowledge extract. The objective of data augmentation is usually preventing overfitting so we can improve model performance. Data preprocessing takes about 50% to 80% of total efforts whereas challenge with data augmentation is to not create data that is irrelevant. Data preprocessing techniques include: data cleaning, data normalisation, data transformation & data reduction. Data augmentation include: symbolic augmentation, rule based augmentation, neural augmentation & geometric transformation.

Standardizing data is useful when the distribution is in Normal or Gaussian, though it isn't necessarily true. It transforms the data to be centered around the mean with unit standard deviation.StandardScaler() is the sklearn function to perform the task. Normalization method transforms the data between 0,1 or -1,1 based on the min, max values. Normalization is used when the features are in different scaling and not in normal distribution. Normalization is sensitive to outliers.MinMaxScaler() is the sklearn function for this. Choosing between normalization and standardization is based on the model, data.Sometimes, it's better to train the model in raw, normalized, standardized data separately and compare the results that are optimal.

This is where the understanding of business helps provide context. There is no single method that works best across industries and domains. Understanding why data was missing in the first place is key because it helps you determine how to address the missing values. It is a good practice to check whether there’s certain segment or group that is more likely to have missing data before processing. For instance, if certain underrepresented ethnic groups are more likely to have missing data, deleting missing data could cause severe issues in the models later on.

Most algorithms still don't work on missing data. This is especially true in real-life scenarios, where missing and messy information is the norm. Some of the most common solutions involve imputation, which means filling the missing information with a specific approach. This can be mean imputation or zero imputation, which are common and simple approaches but tend to perform poorly in most scenarios. Imputation leveraging Machine Learning, such as KNN Imputation, tends to lead to the best results, filling missing information with data similar to similar samples.

In a longitudinal dataset, we can use Last observation carried forward method to impute the missing values with last valid observation. eg: pandas code: df['systolic']=df['systolic'].fillna('ffill'). In timeseries, interpolating the values before and after the time stamp can be done. Eg: pandas code : df['temperature']=df['temperature'].interpolate(method='linear', limit_direction='forward', axis=0). we can also try to use models which can accept missing data such as random forest. If the values have to be imputed using mean/ median, then median can be better and insensitive to outlier.

Tip: some effects may not be frequent in your training/eval data but very important for your application (e.g. fairness & sensitive issues, offensive content, etc) Create “slices” — issue-specific subsets — to help you to pay attention to them during training and evaluation.

For any organization that is considering attracting the top data science and data engineering talent, consider which data-related processes you can put into place to ease the friction and tension. Bringing in top talent only to have them frustrated by spending a majority of their time wrangling the data rather than solving innovative problems is bad for business. Tips to consider: - have a clearly articulated data strategy - ensure data priorities are aligned to top business priorities and business problems - have a network of business SMEs available to support data projects - have clear ownership of data identified - ensure an established data use & data reuse policy exists - have governance of use cases and data identified Good luck!

Remember, the choice of preprocessing and augmentation techniques depends on the nature of the data, the problem at hand, and the specific deep learning model being used. It's essential to understand the data and experiment with different approaches to find the best preprocessing and augmentation strategy for your specific scenario.