Since you have different dependent variables, there is no meaningful way to compare them statistically.

However, these models might serve some purpose that could be evaluated externally. So you could use this external evaluation and compare the model performances for this purpose. This is usually quite straight foreward for logistic regression models, where the aim is to classify an outcome, what can easily be evaluated using an independent reference data set (and make a confusion matrix etc.). For linear models such a "natural" approach does not exist, but you may get some idea at least, in which direction you could think.

Prof Wilhelm pretty much covered the area but the attached paper gives references to some of those methods and as such may be useful to you. Best wishes David Booth

As rightly said by Prof Wilhelm unless and until you don't have the same dependent variable (which may be expressed in terms of different independent variable) there is no meaningful method to compare the models.

The only deciding question is: what is the theoretical relationship you expect between your variables? Then you can test if this model works, by fitting it to the measurements. Not the other way around. If you give us more details about the variables & expected connecting formula, I might be able to help you further.

Assume that you consider a set of candidate methods and each of them represents a valid relationship between the quantities involved. Then you can do residual analysis. The difference between the output of a candidate method and that of the reference method is the residual. Ideally, the residuals should be approximately normally distributed around zero. If the residuals are not centered at zero, there is a bias in the candidate methods with respect to reference method. In addition, the uncertainty (or variance) of the residuals measures the precision of the candidate method. A preferred method should be the one that has the smallest bias and uncertainty (or minimum RMSE) among a set of candidate methods.

Hening Huang Looking at the residuals is interesting, but even if they show a pattern, it doesn't mean the model function is completely wrong; there could just be another hidden variable playing a role. The best model is one that is explainend by a physical theory.

Also, looking at extrapolation of the model can be useful. If that gives crazy results, the model is definitely wrong, even is it fits nicely through the data points.

Koen Van de Moortel I totally agree with you that "the best model is one that is explained by a physical theory." Physics should always come first and statistical analysis comes second. Therefore, we do need the assumption that each of the candidate methods under consideration represents a valid relationship between the quantities involved. I also agree with you that if residuals show a pattern, there could be another hidden variable that is not accounted for by the model (or method). Therefore, residual analysis helps to determine whether the model takes into account all important variables.

If you have multiple models with different response variables, this isn't a statistics question; it is a subject matter question. Choose the model that makes the most sense--or use several models together. There is no statistical reason not to do that.

Dear all

From your comments/suggestions, it can be summarized as follows:

- A direct comparison of the regression lines can not be made as I draw them using different response variables (new methods);

- I checked the precision of the regression lines individually by looking at the R squared (R2) and standard error (SE);

- Then I looked at the RMSE of the residuals (calculated as Observed value - Predicted value of each new method). The model that showed the lowest RMSE was considered as of highest accuracy among all the methods tested.

I am still looking forward to further comments, if you think relevant.

Thank you very much.

Tho Nguyen -

You have "A direct comparison of the regression lines can not be made as I draw them using different response variables," but that is not true. A "graphical residual analysis" can be used to compare two or more different regression models on the same scatterplot, for the same sample. "Cross-validation" is used because fitting a model too tightly to one sample may mean it will not fit well for other data in the intended population or subpopulation. But you can try other samples on other scatterplots. Each graphical residual analysis scatterplot will be used to compare different models for a given sample, if you plot the results for each model for that sample on the same scatterplot, one scatterplot per sample.

The "graphical residual analysis" (you can research that term in quotes) will show the model fit, including heteroscedasticity, a naturally occurring feature which can be modeled by including regression weights. Typically variance of residuals becomes greater with larger predicted-y-values. This can be seen as the density in the y-direction on the scatterplot becomes less, indicating more variance in the y-direction, regardless of range, as you move to larger predicted-y-values. See https://www.researchgate.net/project/OLS-Regression-Should-Not-Be-a-Default-for-WLS-Regression, and the various updates and references there.

If you research "graphical residual analysis," you need to remember that you can put the results of more than one model for the same sample on the same scatterplot.

Best wishes - Jim Knaub

PS - Actually, my comments above are consistent with Hening Huang's statement that "...you can do residual analysis."

Tho Nguyen - Wait, you said "...(same independent variable, different dependent variables)...." If you do not have those reversed, then I don't know what you are asking. You might look into "multivariate regression," not multiple regression. If you wrote that down correctly, then I do not understand what you are asking. Sorry.