How do you validate and calibrate choice models to ensure their reliability and accuracy?

1 What is validation and calibration?

Validation and calibration are two related but distinct processes that aim to improve the quality and performance of choice models. Validation is the process of checking whether the choice model can reproduce the observed behavior of the data used to estimate it, or a different but similar data set. Calibration is the process of adjusting the parameters or specifications of the choice model to improve its fit and predictive power, either by using additional data or by applying statistical techniques. Both validation and calibration are essential to ensure that the choice model is robust, consistent, and realistic.

2 How to validate choice models?

Validation of choice models varies depending on the type, complexity, and purpose of the model. Common methods for validating choice models include comparing predicted and observed market shares or choice probabilities with mean absolute error (MAE), root mean square error (RMSE), or percentage error. Additionally, you can compare the predicted and observed elasticities or sensitivities of choice probabilities to changes in attributes, such as price, travel time, or service quality with mean absolute percentage error (MAPE) or correlation coefficient. Furthermore, you can compare the estimated and theoretical values of the parameters, such as utility coefficients or scale factors, with tests such as t-test, F-test, or likelihood ratio test. Lastly, you can compare the choice model with alternative models, such as nested logit, mixed logit, or latent class models with measures such as Akaike information criterion (AIC), Bayesian information criterion (BIC), or McFadden's pseudo R-squared.

3 How to calibrate choice models?

When calibrating choice models, there are various methods and criteria to consider, based on the type, complexity, and purpose of the model. For example, you can update parameters or specifications with new data, such as revealed preference (RP) or stated preference (SP) surveys. Statistical techniques can also be used for weighting, scaling, bias correction, or regularization to account for errors or uncertainties. Additionally, numerical techniques like gradient descent, Newton-Raphson, or genetic algorithms can be used to optimize parameters and maximize a certain objective function.

4 What are the benefits and challenges of validation and calibration?

Validation and calibration can provide several benefits for choice models, such as improving the accuracy and precision of the model and its predictions, as well as identifying the strengths and weaknesses of the model. It can also explore the sensitivity and robustness of the model and its parameters. However, validation and calibration can also pose some challenges, such as requiring sufficient data and information to perform the process, which may not always be available. Additionally, there may be trade-offs between different criteria and objectives, depending on the context and purpose of the choice model. Lastly, there are uncertainties and risks associated with the choice model that may not account for all possible factors, behaviors, and outcomes.

5 How to improve your skills in validation and calibration?

Validation and calibration are not one-time or static tasks, but rather ongoing and dynamic processes that require constant learning and improvement. To enhance your skills in this area, you can review literature and best practices for different types of choice models and applications, use appropriate software and tools such as R, Python, Biogeme, Nlogit, or Larch to perform validation and calibration, apply it to your own choice models or case studies, and compare and contrast your results with others. Additionally, you can seek feedback from experts and peers on validation and calibration, as well as participate in forums, workshops, or courses on the topic.

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?