How do you check data quality after modeling?

I agree, I also think it is critical to note exactly what processes and constants model outputs incorporate and what they leave out, and then consider how much of an influence those variables can have on an output and take that into consideration when using model generated outputs. As for most models it’s near impossible to incorporate every single factor for modelling (in my experience, hydrodynamic and/or morphodynamic modelling) so by noting what is and isn’t used, you can apply error margins/ estimations as to how skewed/varied the outputted data can be.

Verify the accuracy, completeness, and consistency of the data. Look for missing values/outliers/anomalies. Examine the features of the model for relevancy, meaningfulness, if they are properly transformed to align with the solution to the problem. Analyze the distribution of the target variable to check if it is balanced and representative of the real-world scenario. Evaluate the model's performance using appropriate metrics like accuracy, precision, recall, F1-score, AUC-ROC.

Having a clear idea crucial prior to any advance analytics (data mining or modern AI). Maybe sound like "old school". i.e. Telco mobile "churn" predictive modeling definition or clarity of 1. customer segment (business or individual) 2. Churn (voluntary vs involuntary) 3. Product/service types (Post-paid vs Pre-paid) The above selected has its own sets of business nature (post-paid with contract & credit limits, pre-paid with expiring period & reload) & subscribers behaviors that affecting the churn model development & results. Likewise the definition approach can be applied to most advance analytics.

I find printing a financial model on paper is one of the best ways to check data. Yes, old school. Grab a pen and actually look at the numbers. I will nearly always find something that jumps out as "a little odd" or needs a "second look" before publishing.

A good example here, when dealing with large datasets - missing values and zero. Missing values have big impact on the results which have to dealt with appropriately.

Create test cases around completeness and reconciliation of data. Reports or scripts could be written in SQL to ensure completeness. Example - check completeness against totals of customers or subscribers in Salesforce, provisioning and billing systems within a time period depending upon kpi being calculated.

Perform k-fold cross-validation to assess model stability and reduce the risk of overfitting.Check for bias in the model's predictions across different groups. Ensure fairness and ethical considerations are met.Examine misclassified instances or instances with high prediction errors. This can provide insights into areas where the model might be struggling.

Sanity checks are extremely valuable! When I finish the architecture of a model, I print the outputs and compare my forecast to Wall Street consensus. If I'm not in the ballpark, it's time to check for errors (or invest!). Are there any spikes in growth rates or margins? For DCF analysis - I check what portion of the final value is attributable to the forecast period vs. the terminal value. If TV isn't 65% or higher, time to check the mechanics. I also cross check both DCF methods against each other. For PGR, I calculate the implied EBITDA multiple. Does it line up with my comps? For exit EBITDA, does my implied g make sense? (Usually never outside 2-5%). Print your work and check it with a calculator. Low tech, high value :)