How can you model nonlinear and complex relationships without linear regression?

Polynomial will cause overfit if not handled with great caution and will make the model useless. Not a fan of polynomial unless business has great justification behind it!!

I've successfully used polynomial models to represent results from lab tests in civil engineering. Importantly this was used because the literature based on the physical realities of material properties said that polynomials would be expected. Also, I've always used data visualization tools to show the engineers the fit. That has been tremendously helpful to overcome overfitting, which can be a real issue for higher order polynomials.

Choosing the right degree for the polynomial is as well crucial. A model with too many terms may fit the training data well but perform poorly on new data due to overfitting. Conversely, a model with too few terms may not capture the underlying relationship adequately, leading to underfitting. Techniques such as cross-validation can be used to select the optimal degree for the polynomial. Furthermore, Polynomial regression can be sensitive to the scale of the features. Large values can lead to extremely high or low predictions. Therefore, it’s often a good practice to standardize or normalize the features before fitting a polynomial regression model.

Anything more than a quadratic model is an exercise in an over abundance of hope and optimism. I have rarely seen a high order polynomial fit bring any value to the exercise. That is not to say it is impossible. Just improbable.

Overfitting is probably the most common newbie mistake. The model should not just fit the data, but make sense; a polynomial regression more often than not makes sense only if you can justify it theoretically. It's very easy to add higher order terms, fit the data very well and have a model that can't be generalized, from which you can't extrapolate and that is generally useless.

Multinomial regression can be used if the dependent variable has more than 2 categories, though the coefficients are somehow difficult to interpret. Most common newbie mistake with categorical variables, also as covariates: make sure that the reference category, the one all other categories are compared against, is one with a high relative frequency and meaningful (i.e. not "other").

Three more points: Odds Ratio: Logistic regression provides the odds ratio for the predictors, which is the ratio of the odds of the event happening to its not happening. This can be particularly useful in understanding the relationship between the predictors and the outcome. Multinomial Logistic Regression: While basic logistic regression is used for binary classification problems, multinomial logistic regression can be used when the outcome variable has more than two categories. Model Evaluation: It’s important to evaluate the performance of a logistic regression model using appropriate metrics, such as the confusion matrix, accuracy, precision, recall, F1 score, and ROC curve.

Logistic Regression is a powerful modeling technique used when the relationship between variables is not linear, particularly in binary or multi-class classification problems. Unlike linear regression, logistic regression doesn't predict continuous outcomes but estimates the probability of an event occurring. By employing the logistic function, which maps any real-valued number into a range between 0 and 1, logistic regression models can effectively handle non-linear relationships. It's widely utilized in various fields, including healthcare and finance, for predicting outcomes such as whether a customer will purchase a product or a patient will develop a certain condition.

With nonparametric models one trades parsimony (and often explainability) for model fit. Moreover, parametric models are, in my experience, less computationally intensive. Extrapolating from nonparametric models is often problematic. Having said that, the flexibility of nonparametric models is far greater than of parametric models. Whether this flexibility is a must has to be considered carefully.

Would recommend trying model non-linear and complex relationships with Generalized Additive Models. However, it’s always important to plot the data before trying to model the relationship.

Non-parametric regression is a versatile modeling approach that doesn't rely on explicit assumptions about the functional form of the relationship between variables. Unlike parametric methods, non-parametric regression doesn't assume a specific equation shape, allowing it to flexibly capture complex and non-linear patterns in the data. Techniques like kernel regression or spline smoothing are common in non-parametric regression, enabling the model to adapt to the inherent complexity of real-world data. This makes non-parametric regression particularly useful when dealing with relationships that cannot be adequately represented by predefined mathematical functions.

Cubist regression models combine regression trees with linear regressions at the leaf level. They can make sense when you are confident that your variables have a direct effect on the dependent variable.

Handling of Outliers: Tree-based methods are less sensitive to outliers in the data, as they split the data based on conditions, rather than values. Scalability: Tree-based methods, especially those based on ensemble techniques like random forests and gradient boosting, can handle large datasets and high-dimensional feature spaces. Overfitting: While tree-based methods can capture complex patterns, they can also overfit the data, especially if the trees are allowed to grow too deep. Techniques like pruning, setting a maximum depth for the tree, and using ensemble methods can help mitigate this risk.

Tree-based methods, such as decision trees, random forests, and gradient boosting, offer robust solutions for modeling complex relationships without the linear assumptions of traditional regression techniques. These methods recursively split the data into subsets based on key features, creating a hierarchical tree structure. Decision trees are interpretable and able to capture non-linear relationships, but they can be prone to overfitting. Random forests aggregate multiple decision trees to enhance predictive accuracy and mitigate overfitting. Gradient boosting builds trees sequentially, with each tree compensating for the errors of its predecessor.

A useful perspective to consider when discussing neural networks is their versatility in solving various real-world problems. Neural networks have gained popularity due to their ability to tackle complex and nonlinear patterns in data. Their adaptability to different tasks, including regression and classification, makes them a valuable tool in fields like image recognition, natural language processing, and autonomous vehicles. Despite their challenges, such as data and computation requirements, overfitting, and interpretability issues, the capacity to learn intricate relationships within data sets often outweighs these drawbacks when applied appropriately in problem-solving contexts.

Few more points: Variety of Architectures: There are various types of neural networks, each with its own strengths and use cases. These include Convolutional Neural Networks (CNNs) for image processing, Recurrent Neural Networks (RNNs) for sequential data, and many others. Transfer Learning: Neural networks can leverage pre-trained models through a process called transfer learning. This can significantly reduce the amount of data and training time required. Computational Requirements: Training neural networks, especially deep ones, can be computationally intensive and may require specialized hardware, like GPUs.

Neural networks, inspired by the structure of the human brain, are powerful models for capturing complex and non-linear relationships in data. Comprising layers of interconnected nodes (neurons), neural networks learn patterns and representations through training on large datasets. They excel in tasks such as image recognition, natural language processing, and complex function approximation. With their ability to adapt to diverse data structures and hierarchical features, neural networks offer unparalleled flexibility. However, their complexity and the need for substantial computational resources can make them challenging to train and interpret.

Consider breaking a non-linear process into multiple operating regions and find independent linear relationships or linear models for each of those operating zones. This will keep things simple and easy to program in an industrial automation setting. A sliding scale type gain for each operating zone is another way to simplify the models. The golden rule is TSTF. Try simple things first. Do not over complicate things unless absolutely necessary. Always keep in mind the long term sustainment of what you are building with people changes and technology platform upgrades.

Step one for any fit of a model to data, is to understand the data. It might very well be that linear regression, polynomials, or even more advanced tree-based models do not provide good results. There might be some cut-off value where the data goes from one pattern to another (like a clear dent in a curve consisting of two straight lines). Sure, that behaviour *could* be modelled by a neural network, but maybe a domain expert could just as easily tell you that that's the way the world works. Before you consider which model to use, consider who you could talk to first.