What are the best methods to identify outliers in transportation management data?

1 Visual inspection

One of the simplest and most intuitive methods to identify outliers is to use visual tools, such as charts, graphs, or plots, to inspect the data. Visual inspection can help to spot obvious outliers, such as extreme values, clusters, or gaps, that do not fit the general pattern or trend of the data. For example, you can use a boxplot to show the range, median, and quartiles of a dataset, and mark the outliers as dots or asterisks outside the whiskers. You can also use a scatterplot to show the relationship between two variables, and look for points that are far away from the main cluster or the regression line.

2 Statistical tests

Another method to identify outliers is to use statistical tests, such as z-scores, t-tests, or Grubbs' test, to measure how much a data point differs from the mean or the standard deviation of a dataset. Statistical tests can help to quantify the degree of deviation and assign a probability or a significance level to each data point. For example, you can use a z-score to calculate how many standard deviations a data point is away from the mean, and set a threshold (such as 3 or -3) to flag outliers. You can also use a t-test to compare the means of two groups of data, and detect outliers that cause a significant difference between the groups.

3 Machine learning

A more advanced method to identify outliers is to use machine learning techniques, such as clustering, classification, or anomaly detection, to learn from the data and find patterns or rules that define normal and abnormal behavior. Machine learning can help to handle complex and high-dimensional data, and discover outliers that are not easily detected by visual or statistical methods. For example, you can use clustering to group similar data points together, and identify outliers as those that belong to small or isolated clusters. You can also use classification to train a model that can predict the class or category of a data point, and identify outliers as those that have a low probability or a high error rate.

4 Domain knowledge

The last method to identify outliers is to use domain knowledge, such as business rules, industry standards, or historical trends, to validate and interpret the data. Domain knowledge can help to understand the context and the meaning of the data, and explain why some outliers occur or how they affect the transportation management objectives. For example, you can use business rules to define acceptable ranges or limits for certain variables, such as delivery time, fuel consumption, or customer satisfaction, and identify outliers as those that violate the rules. You can also use industry standards to compare your data with benchmarks or best practices, and identify outliers as those that deviate significantly from the norms.

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?