Why did you plot residuals by ANOVA cells? A total box plot of residuals will suffice. Also a qq-plot of residuals will be worth looking.

Try drawing scatter plots of residuals vs predicted or vs predictors and see if any signs of heteroscedasticity and outliers exist.

Thanks Mehmet. The guide I'm using suggested that I need to plot the residuals for each cell separately. Will try the approach you suggest.

Yes, it makes sense in these analyses to look at the residuals, perhaps studentized residuals per observation. You might also look at Cook's distance per observation. The idea is that you want to take into account all the factors together, as the analysis is. This is sometimes called a multivariate outlier.

But also, if you are looking at means from Likert items, how much of an outlier can someone be? Scoring a 10 when everyone else like them is scoring a 6? This may play some havoc meeting the assumptions of your test, but it would be difficult to consider the observation illicit without other evidence.

Thanks Salvatore - yes, I think it is a bit odd on a 5-point scale having outliers. that's why I'm struggling! If I look at a box-plot for the raw mean scores for the whole sample then only two scores appear to be outliers (and only one is an 'extreme outlier'). I've also found this article that suggests for small sample sizes outliers are quite likely and not necessarily indicative of deviation from the underlying population norm. http://ww2.amstat.org/publications/jse/v19n2/dawson.pdf

So I'm wondering if it makes sense to acknowledge that there were outliers but to cite that article as justification for keeping them in, given that I couldn't find any evidence of experimenter or participant error?

Sarah, first we need to make clear what you do.

You are

i) investigating outliers of your dependent variable (not the residuals)

ii) doing this in each ANOVA cell

iii) by investigating boxplots (with some rule)

Are these correct?

Can you make it clear also that how do you find means and sums of Likert items produce different outliers?

Hi Mehmet, thanks for following up.

My IVs are Depression (depressed/not depressed) and Stress (high stress/low stress).

My DVs are Negative Affect (NA) and Positive Affect (NA) overnight.

I measured NA and PA in the evening and in the morning using the PANAS, which consists of two ten-item sub-scales measuring each construct.

I have calculated means for each construct (NA and PA) at each time point (evening and morning). I have then subtracted the mean evening score from the mean morning score for each variable, to arrive at the mean change.

In terms of the analysis:

- I have been looking at the outliers of the residuals of each ANOVA cell - that's what it says to do in the guide I'm using...

- Then I'm looking at boxplots. Anythhing more than 2 SDs from the boxplot edge is considered an outlier and anything more than 3 SDs from the edge is an extreme outlier.

I have tried analysing the data with the outliers excluded and the results are very similar - the only difference is that if I take them out then the main effect of stress on NA change becomes significant.

I'll answer one piece of a larger question: Should you keep the outliers in your data set (and maybe note their existence in the results?

Yes.

I don't think you need to cite anything to argue that you should keep outliers. On the contrary, I think you would have to present pretty compelling evidence of why you were justified in discarding outliers if you did so.

And I'd probably advise throwing away whatever document recommends discarding outliers due the result of a mechanical method.

What could justify excluding outliers?

1. If the values are impossible or known to be wrong. If it was recorded that someone scored 200 on a test that they could only score 100 on, or if it was recorded that it rained 10 inches on a day when you know it just didn't on that day. But these cases may call into question your whole system of measuring these things. (Though sometimes rare errors occur.)

2. I suppose if you had some observation that was correct but was so far out of the norm that it just isn't representative of your population, you'd have a tough decision to make. I mean if you were measuring the income of people in the local deli, and by some freak occurrence Bill Gates happened to be there, you'd have a dilemma about how to handle that data point.

But otherwise, try to think of the logic of discarding outliers applied to real life. If I spend, say, $5 - $10 / day on incidentals: stopping for coffee, a soft drink, a beer after work, but once in a great while, say every 356 days, spend $5,000 on booze and gambling, can I tell my wife that that's just an outlier and can be ignored? I don't think so. I think you have to find a way to account for that in the model. (Luckily my wife doesn't know about robust statistics.)

You don't want to change the data to meet your expectations. You want your model to explain the data that exists.

That being said, your data still needs to meet the assumptions of the analysis. If you are using a one-way anova, be sure to understand the assumptions of the analysis. It's possible that these outlying values will play havoc with these assumptions. But there are usually robust or nonparametric tests that can be employed in these cases.

Thanks Salvatore - I agree that it seems mad to throw out data just because it doesn't fit the test. The only issue for me is that I don't know of a non-parametric alternative to the 2-way ANOVA... so that's why I tried running it without the outliers to see if they were skewing the results. It seems to me that they don't change the overall trend of the results (eg interaction graphs look v similar)... but it does move the significance levels a bit, such that one IV becomes significant without.

I'm wondering if it is worth doing a set of non-parametric paired tests (eg Wicoxon Signed Rank) for each cell of the ANOVA and for the main IVs instead? This way I would compare between subjects mood scores (evening and morning) for each category - depressed-low stress; depressed-high stress etc... ? I have tried running these and I get quite a few more significant results...

Thanks Sara.

I can say that having normally distributed residuals at each cell of ANOVA is a very restrictive assumption that is hardly satisfied. Also as pointed by you, when sample sizes are small (as ~10 observations in a cell in your case) it is highly likely to call an arbitrary observation outlier and hence the distribution not normal, especially with a graphical method like boxplot. So this makes the above-mentioned restrictive assumption even more restrictive. I think your case is a good example of this and what you find as outliers are actually not. Instead, you can inspect a boxplot/qq-plot of all residuals as I described above.

If you need, I can recommend you considering some diagnostics (I partly mentioned in a previous post) which are described in regression chapters of statistics books.

Among them, you can use Cook's distance and DFBeta values to find outliers and more importantly influential observations. An outlier may not necessarily be influential and it is not a good practice to delete them.

Sarah, I now see your reply to Salvatore and want to add that I don't think you need a nonparametric test for your data.

A couple of t/wilcoxon/mann whitney tests are not substitutes of a 2way ANOVA, so avoid them. If you need to compare categories of your IVs, compare lsmeans. SPSS has EMMEANS keyword (also some point-click functionality) that does this.

Ok, thanks Mehmet, that's really helpful advice on the non-parametric tests. (It's a shame because the non-parametric Mann-Whitney U test gives me a significant result for stress whereas the ANOVA doesn't). It sounds like I'm being over-stringent in my outlier tests. When I look at the Q-Q plot and boxplot for all residuals there does still seem to be an extreme outlier... I think that I will mention this in my analysis (the outlier) but keep it in. I am running out of time on this so I'm not sure if I should go down the rabbit hole doing more complex outlier analysis.

Thanks again!

Sarah

Hi Salvatore S. Mangiafico and Mehmet Sinan Iyisoy - to give you an update, I took your advice and plotted the Cook's differences for the residuals for the whole sample and there is definitely one data point that is a huge outlier. So I'm going to include it but mention that I have one outlier and this may have influenced the results.