In data science, specifically in predictive use cases, I have seen application of Linear Regression plays bigger role. Most of the predictive problems undergo this method knowingly or unknowingly and then based on the results the statisticians/ data scientists progresses towards the solution. My question is about model fitting, when we fit a model on particular dataset, how do we get to know in practical scenarios that model is over fitted. What parameters should be tuned to remove overfitting? I hope there will be many ways, I would like to know as many as possible.
Your response is appreciated in advance, Thank You!
Do you mean over-fitting as in too many variables as too many variables? Or too many interactions among these variables? Or too many pieces (if you mean piecewise linear)? With any of these in ideal circumstances follow Jochen's advice. If you can't collect more data in the same way, then there are methods that try to mimic that, like cross-validation. Besides worry about over-fitting, some people prefer simpler solution because of the bet on sparsity principle. This has made more impact in statistics where (generally) there is more interest in understanding the model than in machine learning where (generally) there is more interest in prediction.
Use data splitting and x-fold cross validation to fit models, including variable selection steps followed by testing on "validation" set. Selecting "important" variables using cross validation approaches generally results in smaller more parsimonious models. That said, unless candidate models are developed in a logical way that eliminates potential spurious (non-causative) factors resulting models will be valid only for prediction--As Daniel notes- the models may not provide for accurate interpretatoins, despite being highly reliable as predictors.
Developing causative evidence requires testing of carefully devised models. For example to see that smoking causes lung cancer, one would expect to see evidence pointing in the same direction for various mutually exclusive groups of subjects...If cancer rates are correlated with smoking in men but not in women then it would be difficult to conclude causation...Similarly if smoking is causative, one should see cancer rates higher in smokers than non-smokers, but also the duration of smoking should be positively associated with cancer rates.
Careful causative inference from observational studies (where over fitting is a problem) is strongest when many mutually independent causative hypotheses can be tested.
Dear Mayur,
Over-fitting can be avoided by considering the principle of parsimony in linear regression modelling, i.e. choosing the simplest model with the maximum explanatory power. Adjusted R2 is a potential way of measuring explanatory power. This could be measured against the number of independent variables or cross terms in the model. As more variables and cross terms are added progressively less value is added to the model thus a suitable cut-off point can be determined.
Except Peter's good answer i think that it's some misunderstanding here. The rectilinear model can not be over-adjusted / over-fitted, because it's not a curve line that can be "learned", or "bended". It may be an over-correlation of variables ("too big collinearity") or a misinterpreted variance, but there in the normal linear regresion model there nothing more that straight line and model equation result is just a straight line.
Similarly to Peter's answer above, you can add terms in a progressive/hierachical manner and check if there is any significant improvement to the model using ANOVA/F-test. For a non-linear model, you can take a similar approach but use AIC or BIC as your measure of improvement.
- Badges
- Science topic