Types of Statistical Regression Models

Inspired by this and this

Types of Regression:
Attributes of a regression model:
There are 4 key attributes:

1. No. of Independent variables
2. Type of Dependent variables
3. Shape/Curve of the Regression line
4. No. of Dependent variables

Disclaimer: This is by no means exhaustive. Just an attempt to write around key techniques

  1. Linear —

* – Shape of regression==> straight line.
* – (no. of variables is assumed 1 independent vs 1 dependent)
* – Type of output/dependent variable(Numerical)
* – Type of input/independent variable(Numerical)
* – More variables can be used by collapsing them into one(some formula like weighted sum)
* – Or just by adding more variables to the line eqn(and modifying the convergence algorithm): Y = a + b * x1 + c * x2 + …. + xn

2. Logarithmic —

* – Shape of regression ==> exponential/logarithmic curve
* – (no. of variables is assumed 1 independent vs 1 dependent)
* – Type of output/dependent variable(Numerical)
* – Type of input/independent variable(Numerical)
* – More variables can be used by collapsing them into one(some formula like weighted sum)

3. Logistic —

* – Shape ==> S-shaped logistic curve
* – (no. of variables is assumed 1 independent vs 1 dependent)
* – Type of output/dependent variable(Binary-aka 2 categories)
* – Type of input/independent variable(Numerical)
* – More categories in output variable can be used by collapsing them into one(some formula dividing the output curve)

4. Polynomial —

* – Shape ==> Some complex polynomial function of order >= 2
* – (no. of variables is assumed 1 independent vs 1 dependent)
* – Type of output/dependent variable(Numerical)
* – Type of input/independent variable(Numerical)

5. Stepwise —

* – no. of variables is assumed multiple(n)- independent vs 1 dependent
* – Multiple steps, with each of them selecting specific variables based on variance, R-square, t-tests etc..
* – Forward selection , selects predictors/independent variables
* – Backward elimination , starts with all input variables and eliminates them by above mentioned methods.

6. Ridge

* – no. of variables is assumed multiple(n)- independent vs 1 dependent
* – adds a shrinkage (lambda) parameter (to regression estimates) to solve multicollinearity between independent variables
* – also called as l2 regularization.. it’s a regularization method

7. Lasso

* – Least Absolute Shrinkage and Selection Operator
* – also called as l1 regularization.. it’s a regularization method.
* – It shrinks coefficients to zero (exactly zero), which certainly helps in feature selection.
* – If group of predictors are highly correlated, lasso picks only one of them and shrinks the others to zero.

8. ElasticNet

* – Hybrid of Ridge and Lasso method, uses l1 and l2 prior as regularizer
* – It encourages group effect in case of highly correlated variables
* – There are no limitations on the number of selected variables
* – It can suffer with double shrinkage

9. Multi-variate —

* – no. of variables is assumed multiple(n)- independent vs 1 dependent

  1. Multi-variable —

* – no. of variables is assumed multiple(n)- independent vs multiple(n) dependent

Update: These are hardly exhaustive. A closer read and reflection will point out that adding multiple predictors, changing the function, or shape of the relationship, or changing the algorithm to find the right(aka converged) coefficients are easy ways to add combinatoric explosions.

Update2:  I made a jupyter notebook, demonstrating the differences between some of these it can be found here. Also there’s a completely different approach seen here. 

Update3:  Here is a link detailing how to check how good the model worked. Of course there’s the goodness of fit test I wrote some time back.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s