1. I think R-square is a better judgment tool than AIC.

2- Maybe instead of using Prism, Minitab software is a better way to identify outliers data.

3- Minitab can fit the best model for you by dividing the outliers data into Large residual and Unusual X categories and deleting them.

4. Which nonlinear models do you use in Prism? Dose-Response?

Joao -

You say, "Essentially, a point could be an outlier in model X but not model Y...." I assume "X" and "Y" are just names like "1" and "2," not a reference to variables. Anyway, how can a data point, (x,y), be valid for one model and not another? Are you saying that it is not a matter of an outlier which is a mistake because a datum or data do not belong to the population(s) - either the x or the y or both - but instead it is a matter of data quality where one model is robust enough to use the data point, but another is not? Perhaps one model only "thinks" a data point is in error or of low data quality because that model does not account for heteroscedasticity, but for another model which does account for heteroscedasticity, the data point is shown to not be suspicious at all? (Also remember that some data which may not appear to be out of place may actually be outliers, but you can only look at 'apparent outliers,' which may or may not actually be inappropriate.)

So do you really want to use data in some models, but say they are outliers for others? Generally, one should not remove data just because they seem inconvenient. You need a good reason. Say a respondent discovers that they sent the wrong data, or an experiment is found to have been done incorrectly. Perhaps a number was reported in the wrong units. But throwing out a number because it is more extreme than you expected is generally not a good practice. And in the extreme situation that you might try to justify such an action, it would undermine your argument to then say that you still want to use the datum or data in another model.

You might want to reconsider.

Cheers - Jim

Sorry, I haven't checked the responses in quite some time.

Abolfazl Ghoodjani - I am using a variety of models: exponential, power, polynomial, etc...

Using a different program to detect outliers is not going to answer my initial questions though. Also, minitab's website itself says that R2 is not valid for non-linear regression, which is what I am trying to do (https://blog.minitab.com/en/adventures-in-statistics-2/why-is-there-no-r-squared-for-nonlinear-regression)

James R Knaub - You are correct that I meant model "1" and model "2" and instead of model "x" and "y"

There are multiple reasons, related to data collection and analysis as to why one point could be an outlier or not so that is covered. Also, I am not removing any outliers by hand, they are being removed through the rout algorithm, so I am not being purposely biased towards one model or the other.

My drawing skills are terrible, but, for example, in the attached image, you can use all points to trace some sort of bell shaped curve, but, if one of the points is removed (assuming it is an outlier for that model), the best model is probably an horizontal line.

The questions then is, how do I know which of these models is better?

I would not suggest a model which tries too hard to "connect the dots." That means being very careful not to overdo it with a loess regression.

So are you deciding whether a point is an outlier and choosing a model, and sticking with this? It appeared to me that you were planning to sometimes use a data point in some models and sometimes not in others, and use all of those results for different purposes. But if you are really making a choice and sticking with it, that's different.

I still say you should be extremely reluctant to throw out data just because it is inconvenient. If you find that a point is reported in error (i.e., by mistake), that is one thing, but you should be reluctant to drop data otherwise. It may be telling you something important, an example being large numbers with large estimated residuals which just indicate substantial heteroscedasticity. But your graphical example indicates another problem: reacting too strongly to "noise." If a model reacts to a single data point by changing direction to come close to it and then changes direction again to get back to the other points, then I'd say that is not a good idea. The idea of a model is to accommodate all the points as a whole, without getting too hung up on any one of them. That may leave a few unsatisfactory looking cases where you might suspect they are outliers, but you may still want to include them. You might see what happens if you leave such points out and include that option in an appendix as a sensitivity analysis.

If a suspected but unproven possible 'outlier' is just one point, and it has a very big residual, then you might want results with and without it in your main report, and discuss the difference there.

PS - You have that "...I am not removing any outliers by hand, they are being removed through the rout algorithm...." So there is an algorithm which tosses points which have high estimated residual under a given model? That amounts to the same thing. The point is that it is always possible to be in the tails of a distribution. If you start chopping those out, where does it end? I had a guy working for me once that did that, and then other points 'stuck out.' And then he did it again. You can keep this up until you might get something pretty, but life is messy, and you may lose a lot of good information that way. It became obvious that he was overdoing it. I was surprised that he tried that. Desperation can make you do things you might regret. :-)

James R Knaub as far as the example in the image goes, it was just a poorly drawn paint example to make a point.

As far as outlier removal, it should only be run once, so if only one point gets removed, that is it, you cannot rerun the outlier detection algorithm in that dataset, so you would not just "keep removing" points that stick out.

However, I feel we are getting too hung up on the exact nature of what an outlier is and how to remove it instead of addressing the main questions. In my crude drawing we have a point that is not an outlier in model 1 but is in model 2, Is there a way to mathematically compare these two models and determine which one is a better fit to the data?

It does seem to me that there is no concrete answer. I was already planning on reporting both full data set models and outlier removed models (and hopefully show there isn't much difference in total numbers of model changes in my large dataset) but i was hoping i would be able to do more than that

Model 1 and model 2 look like the same loess model format with and without the suspected outlier. ???

If you have a general model form (quadratic, splines, loess, nonlinear, whatever), that does not have any points your algorithm wants to throw out, that would be good.

"...total numbers of model changes...." ???

Have you tried graphical residual analysis with predicted y on the x-axis and estimated residuals on the y-axis? That may help you see if anything really looks that bad.

How about cross-validation? Can you obtain any more data? Whatever model holds up best to new data may be best.

James R Knaub

despite the lack of drawing ability, the models look different to me, one is more "gaussian-like" and the other is more of a horizontal line, these models would both have different equations. As far I understand, that makes them different models, whatever the algorithm that lead to its creation, correct?

As i mentioned in the initial questions, I am looking a several thousands of correlations at the same time, so looking at each individual fit is not an option, I need to do some sort of selection first and then look at the more problematic fits individually.

Obtaining more data might help in some cases but not in others, so that is also not a good solution.

What I mean by "total number of model changes" is to calculate the percentage of correlations, out of the several thousand that I have, whose best fit model would change where outliers to be removed. If this number is high, it indicates that the initial modeling was probably not very good. However, once again I would need to find a way to compare the goodness of fit of model A (with all datapoints) with model B (with outliers removed) and that is the questions at hand.

model parsimony figures of merit such as BIC and/or AICc are key in ensuring that a model is not over-fitting the data with an excess of parameters ... very easy to increase overall likelihood by adding additional parameters

Q: "I would need to find a way to compare the goodness of fit of model A (with all datapoints) with model B (with outliers removed) and that is the questions at hand."

Apples and oranges. A single measure, such as AIC, is in my opinion, not going to always be adequate to make the comparisons wanted, and when you throw in the fact that a potential outlier will be removed, that is a huge unknown factor. If you throw out inconvenient data, you will certainly have a better model fit to what is left. That is the problem I noted above.

So you need a philosophy for this problem. I certainly know about dealing with many sets of data and on a frequent basis too. But the modeling was simple and we had ways to check on the potential problem data. Here is a philosophy that you might consider: If you don't have good reason to throw out data - and remember that 1% of the data should be in the 1% tail of a distribution, and a few percent might be reasonable - just leave in suspicious data, but simplify your model so such a data point has little impact. Sure you can throw out data, use a complex model for the rest, and overfit to the remaining data. That would give you a great fit, but it would not be very useful. So, unless you have a reason to throw out data, like an experiment that was known to be done incorrectly, or survey data that a respondent tells you was submitted in the wrong units, or whatever makes sense given whatever you are doing, don't just throw out data. Depending on your application I might make very rare exceptions to that. If you have 100 data points and one is in the 0.0001% tail of a distribution then maybe it would be better to make a leap and assume it is actually an outlier. If you had two or more such points, might it be better to assume your distribution is not what you thought it to be?

Generally best if you try to obtain good data, don't throw any out, and don't overfit your model. It sounds like you have software which automatically identifies potential outliers using complex modeling. (Loess??) I suggest you see what happens if you use less complex modeling and don't drop any data.

You could use the complex modeling, remove a lot of data, and get great model fits. But results will likely be misleading. Similarly, you could use very complex modeling to wiggle through all the data with very small estimated residuals, and that would be an overfitting, also likely misleading.

So, I suggest seeing what happens if you keep all data, and simplify modeling. Yes the fit won't be as good. But it could be more reasonable. This is where cross-validation comes in. Yes I know you have a lot of cases. But you could do some general checking. Perhaps you could take out some data and see how well you predict for them. No it isn't like comparing two numbers, and saying this is bigger so it's better, but that is the thing about statistics. You aren't always comparing two statistics. Sometimes you need an overall picture.

Best wishes - Jim

I'd say AIC are somewhat analogous to p-values. One alone tells you little to nothing. (See https://towardsdatascience.com/introduction-to-aic-akaike-information-criterion-9c9ba1c96ced.) But you can compare them. However, it has to be on the same data set. The penalty for more independent variables does not make up for a thorough cross-validation, but helps. But a bigger problem here is not comparing models on the same data.

James R Knaub

We are in agreement and I think you are finally understanding my questions here.

I undestand the reluctance in removing outliers, and I am not doing it because its inconvenient. I am using graphpad's robust non linear regression model to fit the data and detect outliers (it uses the rout algorithm - more info here:

https://www.graphpad.com/guides/prism/latest/curve-fitting/reg_rout_method_outliers.htm)

The models I am using are simple (along the lines of A*X^B+C and so on) as i do not want to overfit.

as to the philosophy of outlier removal, my thought process is that in the course of adquiring data, especially biological data, such as the one that I am analysing, there are thousands of reasons as to why one datapoint could be an outlier. Maybe this piece of tissue was collected inproperly, maybe someone handled the tube with a dirty glove. Biology is messy and things happen.

This is no excuse to remove outliers because "they look wrong". that Is why I am runing statistical tests (ROUT in this case) to determine what can be considered an outlier. Sometimes this could even lead to a worse fit. If we have statistical sound reason to believe there is an outlier in the data reporting it is as much of a disservice to science as we will be making conclusions based on incorrect data.

Charles Recchia

I am using AIC to distinguish which model fits for each dataset, as noted in my initial question. The issue here is that AIC (nor BIC) are valid measurements to distinguish removal of an outlier provides a better fit for the same data set (minus the outlier). Hence me coming here to try and figure it out.

From the discussion here it seems that this is not possible, or at least, not trivial and I will have to try and figure out a new way to tackle this problem.

Regardless, I want to thank you all for the discussion