An Introduction to Multiple Regression

Business Applications of Regression

Linear Regression is a very powerful statistical technique that has a variety of applications in today's business. Some of the most common applications where regression is being applied are:

• Generate insights about consumer behaviour to analyse the marketing effectiveness, pricing and promotions of sales of a product. For instance, if company XYZ, wants to know if the funds that they have invested in marketing a particular brand has given them substantial return on investment, they can use linear regression. The beauty of linear regression is that it enables us to capture the isolated impacts of each of the marketing campaigns along with controlling the factors that could influence the sales. In real life scenarios there are multiple advertising campaigns that run during the same time period. Supposing two campaigns are run on TV and Radio in parallel, a linear regression can capture the isolated as well as the combined impact of running this ads together. Once the regression function is learnt from previous data, the effectiveness of future marketing promotions can be predicted. 
• Understanding business and factors influencing profitability. Linear regression can be also used to asses risk in financial services or insurance domain. For example, a car insurance company might conduct a linear regression to come up with a suggested premium table. The risk can be assessed based on the attributes of the car, driver information or demographics. The results of such an analysis might guide important business decisions. In the credit card industry, a financial company might be interested in minimising the risk portfolio and would like to understand the top five factors that cause a customer to default. The Data Scientist in this case can analyse the default data and create a regression function. Based on the results the company could implement specific EMI options so as to minimise default among risky customers.

Normal Linear Regression

Linear regression is a parametric method  in that the regression function is defined in terms of a finite number of unknown parameters that are estimated from the data..

In standard linear regression, we have a training set Y = f(X)+ ε, where X ε R^p, Y ε R and ε = N(0,σ^2). The objective is to predict Y from X by f(X) so that the expected loss function E(L(Y,f(X))) is minimised. 

Let there be n equations y = f(x) , where βare coefficients of the independent variable x and εis the noise. The function might be a non-linear function of x, but it should be linear with respect to the β parameters. It is assumed that noise is distributed as iid and follows a normal Gaussian distribution i.e. ε = N(0,σ^2)

Y = β+ βX + βX2 + … + βXn + ε

Y = β+ βX + βX2 + … + βXn + ε

…………………………………………

Y = β+ βX + βX2 + … + βXn + ε

These linear equations can be written in matrix form as Y = X + ε .

Or ε = Y – Xβ

ε^2   = ε'ε

We want to find elements of β such that ε2 is minimized.

ε^2   = ε'ε = (Y – Xβ)'(Y – Xβ)                                                

Minimising ε^2 w.r.t. β and solving, we get 

β*= Inv(X'X).(X'Y) ; where Inv function means matrix inverse function.

Ridge Regression

Ridge regression is often used to overcome the overfitting phenomenon and it involves adding a penalty term to the loss function in order to discourage the coefficients from reaching high values i.e. tuning to the random noise. The simplest such penalty terms involves taking sum of squares of the coefficients β, leading to a modified error function of the form:

which is equivalent to minimisation of the following equation subject to constraining the sum of the squared coefficients.

Thus ridge regression puts more constraints on the parameters by adding the penalty term. This penalty term is λ (a pre-chosen constant) times the squared norm of the β vector. This means that if the βj's take on large values, the optimization function is penalized. We would prefer to take smaller β's, or βj's that are close to zero to drive the penalty term small.

Code Libraries

I have implemented the above mentioned methodology to learn the regression function from data using Python in this  git repository. This shows the technique to implement regression from scratch without relying on any machine learning library.