What are the most effective strategies for building a regression model?

Before creating a regression model, it's crucial to clearly define the problem you're trying to solve and your main goal. This means understanding what you want to predict or explain, like predicting house prices based on certain factors, and why it's important, perhaps to help homebuyers make informed decisions. Defining your objective helps you focus on what matters most and guides the entire modeling process, from selecting the right data and variables to choosing the appropriate regression technique, making it more likely that your model will effectively address the specific issue you're aiming to tackle.

Like any other aspect of statistical analysis and modeling, regression is not exempt from the "context dependency". Your regression model will depend on the nature of your data and task. For instance, though tree based ensemble and gradient boosting models are popular for a good reason, they are incapable of inducing full functional approximations and extrapolate outside training dataset.

Also check what data you have: how far back they go, how "clean" they are, how sparse, how reliable is the process that generates them, how accessible they are (how many gates and gate keepers you have to go through to get to them), how much prep work they require. Also spend some time thinking about the data you do not have, e.g. you may be missing direct measurements and have only derivative metrics, or you may be missing several cause variables and have only effect variables.

When data scientists start a new classification project, logistic regression is often the first model to be tried to get a feel for the most important features and the direction of dependence. Later, we may use a less interpretable classifier such as gradient boosted trees or random forests if we want to gain performance, or we might stick with logistic regression if it is important to be able to reason about the model.

Selecting the type of regression model depends on the nature of your data and the kind of problem you want to solve. If you're dealing with numerical data and aiming to predict a continuous outcome, linear regression can be a good choice. For categorical outcomes or classification tasks, logistic regression is often used. In cases where relationships between variables are more complex, other types like polynomial regression, ridge regression, or decision trees may be appropriate. The key is to pick the regression method that aligns with your data and research goals, ensuring the most effective model for your specific analysis.

First, look at the data. Analyse outliers. Try to find relationships visually. Discuss and understand with domain experts. i.i.d. does not mean normally distributed. In the real world I have never found normally distributed data.

It's very easy to include too many variables in a regression model, and this can sometimes be okay, though it's important to watch out for when this might be a problem. When forecasting, for example, too many variables won't really be an issue, since ultimately what you're looking for is an accurate prediction. If instead, however, you're looking to examine the strength of the relation between a specific independent variable with what is being predicted, extra variables may bias the accuracy of traditional tests of significance.

Stepwise selection is horrible. I use genetic algorithms to find an ensemble of models, and apply concepts of Bayesian ensembles. Quadratic and cross terms can easily be selected automatically, but I can never find any half decent implementation in R or Python.

What do you do once you have found significant correlations? Manipularte the data? Create a multivariate model? Discuss with domain experts? Correlations totally screw up any attempts at decision optimisation. Also, what about confonding and distinguishing decision variables?

You should mention criteria such as AIC, BIC, AICc, BICc and also discuss effective sample size, particularly for time series data.

It may also be important to re-estimate your coefficients as more data is added, or construct a different model altogether if later data shows a different relationship between the dependent and independent variables.

Linear models rarely fit clinical data assumptions, also there is the mega problem of repetitive measurements, ex: GEE and GLMM are robust statistical methods designed for complex data structures. GEE is particularly useful for obtaining population-average estimates and focuses on a specified working correlation structure for the residuals. It's ideal for estimating the average effect across a population while accounting for correlations within clusters or subjects. On the other hand, GLMM provides subject-specific estimates by incorporating random effects to capture individual or cluster-specific variability. Unlike GEE, GLMM requires the specification of a random effects structure, allowing for more nuanced, individual-level comparisons.

The majority of business prediction problems with tabular data are nailed with simple linear regression models. If you decide to use a few meaningful and powerful variables, you might not always reach the best validation metric but you end up with a more robust and intelligible model - which you are more likely to use as a decision tool.

However, you will/may get negative predictions with the linear regression model which will require a “hand-wave” approach of converting them to zero. Calibration of the model will be useless under such conversion