In every statistical model, the aim is to predict the dependent variable, after obtaining the predicted values we compare it to the actual values, let's assume we are examining a non-linear relationship and we are having the following cases:
Of course the squared errors in the first two cases will be so high, and in third case we'll have small squared errors.
Could the three cases respectively be an indicator for overfitting, underfitting and good fitness.
In Regression modeling, over or under fitting is addressed in terms of portion of total variance about mean w.r.t. the number of degrees of freedom taken up to do so, the model is called overfitted when too many d.f are used up to account for a certain portion of total variation about mean, underfitting occurs when the variation accounted for is small for the present no. of d.f used and increasing d.f in this case results in a model with better fit.
In situations 1 and 2 your model shows significantly large bias, positive and negative respectively, but it does not tell you here how many d.f are involved in the model space that produces the prediction vector, if the residual space is having a large d.f then the error sum of squares, which may be high in situation 1 and 2 can be associated with low mean residual sum of squares in this situation, which contradicts the assumption of under fitting/ over fitting in this situation.
In case 3 also, d.f of residual space will play a role in determining the extent of over/under fitting of the model.
So, is it possible to got over fitting even when the model predictions are close and at some points are identical to the actual values? that is my main concern.
I would suggest you compute and compare the three statistics : R2, R2adjusted and Q2 .
quality of fitting depends in general on the closeness in fitting over the entire experimental space, not at isolated subset of experimental points, the above 3 statistics together address the quality of fitting issue. You can find the associated and relevant literature in internet and books and journals.
One such very good text in this regard is:
Applied Regression Analysis: Draper and Smith.
I will add a few suggestions here if they are useful to you. Try splitting the data into test and training sets to get an idea of how much is the difference in prediction error in the training and test sets. This can also be a function of the number of predictors and flexibility of the model (e.g. polynomial functions of the predictors). You can repeat this procedure multiple times by taking random test and training samples, similar to a cross validation framework. Your other options are performing posterior predictive checks (see here for different examples: https://laplacebayes.wordpress.com/category/model-checking/).
Fitting equations to data can find supposedly acceptable models for explaining the data; sometimes lasting for decades. However, unless the parameters have independently measurable values, the fitted model may fit well but be nonsense mechanistically. The preferred approach is to have a model based on a proposed mechanism of action, with independent measurements of each parameter value, and then predict the outcome from your model. That way, your model has made a testable quantitative prediction, with no room for parameter fudging, and will be shown to be a best candidate for reality (the way your modeled system works) or it is incorrect and needs to be changed (and then repeat the cycle) or discarded . The precision estimates for each of our parameter values (and equipartition of variance) will determine the value of your model.
To add to the above, I suspect that how you measure overfitting might also depend on the specific method that you use to build your model. For example, in classification, Umar Niazi 's suggestion to check the difference in prediction error between train and test set would not be very informative in the case of kNN, as it memorizes the training data, but it would work fine for SVM, as it summarizes the data by maximizing the margin along the separating hyperplane. However, I second the idea of cross validation or validation via boot strapping - do it multiple times and check the variance in the prediction error. You can even do leave-one-out cross-validation, it will help you determine in which subspace your model fails.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling