Anmol -

I suggest that you forget R-square and p-value. They are often misleading. You could take a closer look at graphics.

A graphical residual analysis with predicted y on the x-axis, and estimated residuals on the y-axis can show you a lot about model fit and heteroscedasticity. Cross-validation, where you hold out some data and then see how well you would have "predicted" it can help avoid overfitting to your particular sample.

Also, you could consider what seems reasonable from a subject matter perspective.

Cheers - Jim

PS - You might try more than one model's results on the same scatterplot using graphical residual analysis, to make a better comparison. But try not to overfit. You can squiggle a curve close to points and have lower residuals, but just be fitting to that sample.

Anmol -

I suggest that you forget R-square and p-value. They are often misleading. You could take a closer look at graphics.

A graphical residual analysis with predicted y on the x-axis, and estimated residuals on the y-axis can show you a lot about model fit and heteroscedasticity. Cross-validation, where you hold out some data and then see how well you would have "predicted" it can help avoid overfitting to your particular sample.

Also, you could consider what seems reasonable from a subject matter perspective.

Cheers - Jim

PS - You might try more than one model's results on the same scatterplot using graphical residual analysis, to make a better comparison. But try not to overfit. You can squiggle a curve close to points and have lower residuals, but just be fitting to that sample.

James R Knaub is right. Also, one can check if there is any physical model in the literature that explains your variables' relationship which will help you to decide the model.

Would be great if you could show us a scatterplot. I think cross-validaton, in accordance with what James R Knaub suggested, is a good approach here. But in order to perform cross-validation and compare alternative models correctly, you need to optimize your point predictions in accordance with the loss function used to compare models. Generally, you need to use median of forecast density if you want to compare models using out-of-sample MAE-based metrics and mean if you want to compare models using MSE-based metrics. If you use transformations (i.e., logs) to stabilize variance and then return forecasts to the original scale, you'll get forecasts optimal for the linear loss (and not for the quadratic loss).

You can try the leave-one-out cross-validation (LOOCV). Once you obtain out-of-sample forecasts, you can use our framework to compare alternative models (originally developed for time series, but can be adapted for regression):

Conference Paper Data Formats and Visual Tools for Forecast Evaluation in Cyb...

Leave-one-out (at a time) cross-validation could be good, and especially practical when you do not have a lot of data, but if you do, it would be good to have a whole other sample to check. Then, if you overfit to the first sample, it should be obvious, though it depends on the samples.

Also, rather than use logs if there is heteroscedasticity, which may not be a large enough measure to take, or may be too much, I prefer to model with the heteroscedasticity included (weighted least squares regression). There are examples in the following, which includes one for a quadratic linear regression:

https://www.researchgate.net/publication/333642828_Estimating_the_Coefficient_of_Heteroscedasticity,

and here is a tool for doing this with your own data:

https://www.researchgate.net/publication/333659087_Tool_for_estimating_coefficient_of_heteroscedasticityxlsx

I assume you are doing prediction (regression), but not forecasting, and probably do not have to consider autocorrelation.

James R Knaub Thank you for providing useful references. What do you think: which one is better to use to compare alternative regression models: PRESS or AICc? Thanks.

Andrey -

I'll use one number, such as an estimated relative standard error with regard to an estimated or predicted total in official statistics, though additional metadata may be helpful, but I'd rather not just use one number (statistic) to describe the performance of a model. It does not seem adequate. I've heard of PRESS though, and it seemed OK. I heard BIC can encourage a more parsimonious model (avoid overfitting). Does AIC do that to any extent? I'm not familiar. But I think if you want to use any of these methods of comparison, that you have to consider that your results will differ with a different sample selection from your population of interest. Graphs differ too, but if you just look at one number, you don't want to think of it as absolute. For graphs you can get a better overall picture. "A picture is worth a thousand words."

All that said, I can't really answer your question. I don't know. However, I suppose PRESS might be a little simpler to envision. Perhaps you can give us the pros and cons between PRESS and AIC.

Cheers - Jim

James R Knaub Thanks a lot for your answer. It was interesting for me to know your opinion. It seems the question of how to compare alternative models is not an easy one. Many people are now recommending this work (suggesting scoring rules assess the quality of probabilistic forecasts):

Article Making and Evaluating Point Forecasts

As for PRESS vs AICc, I think some simulation experiments are needed to understand their pros and cons. I agree that PRESS is very easy-to-interpret.

Regards,

Andrey

The Penn State Stat501 notes on Information Criteria and PRESS might be helpful.

• https://online.stat.psu.edu/stat501/lesson/10/10.5

Looking at the abstract of the Tilmann Gneiting article, Andrey, it reminds me of how in the 1980s and maybe 1990s, academicians seemed to me to go to heroic lengths to discuss the null hypothesis and p-values when I found that to be of little to no practical value. Here, when all is said, we are still talking about a single number to describe a complex and varied performance. I don't want to 'sell [that article] short,' but from my point of view, sorry, it sounds like it might be interesting to many academics, but I'm not sure how useful it might be. But thank you for pointing it out.

I am glad to see a contribution from Penn State, Bruce, as I have so often referred people to their material. I do not always completely agree with them, but they do a great service with the course material they have, freely available to all, and I have told them so. Here, I see that the "+2p" part of AIC does provide some protection from overfitting. Anyway, they provide a lot of clear, introductory material to many topics which people may ask about on ResearchGate.

......

Anmol, et.al. -

It occurs to me that you might find something of interest here:

https://www.researchgate.net/publication/313925533_Comparing_Model_Performances_Graphically

Cheers - Jim