A good intro to WIndsorizing (and other robust methods) is: https://www.amazon.com/Fundamentals-Modern-Statistical-Methods-Substantially/dp/1441955240/ref=sr_1_3?crid=2COCT1SXBB82Q&dchild=1&keywords=rand+wilcox&qid=1597898555&sprefix=rand+wil%2Caps%2C253&sr=8-3

I think you may have some more basic issues, concerning statistical analyses. A paired sample t-test is identical to a one-sample t-test against 0, with the difference scores of your within variables. Therefore, it is not surprising that outliers on you "raw" variables are not necessarily found in your difference variable (which would be internally used for a within t-test). For example, if you have the three pairs of data (1,2), (3,2) and (100, 101), on the raw values, the last one would look like outliers, but the differences, i.e. -1, 1 and -1 would not indicate any outliers. Therefore, all assumptions for within t-tests refer to the difference scores, not the raw values. Additionally, I am afraid your Winsorizing was not done as intended.

Please have a look at the book proposed by Daniel Wright it is really easy to understand and shows several methods to detect outliers and how to apply different robust analyses methods, e.g. trimming, winsorizing, bootstrapping etc. Wilcox also provides R codes for all analyses in his books, which you can download for free as "functions" from his webpage (which work more reliable than the packages).

Hello Daniel Wright and Rainer Duesing , thank you for your input and advice.

To clarify: I wasn't surprised that are not the same, I am aware of basic maths; what I am confused about is that the feedback I got was referring to the non-normal distribution of the differences between scores (for within-subjects design)

- so considering the context in which Winsorizing was proposed

i.e. the exact feedback

a.' If the distribution of difference scores are notably non-normal and that doesn’t change after Winsorizing, then bootstrapping seems a good option ' / ' Bootstrapping is a good option if the normality assumption is too violated. If the issue is primarily outliers, then Winsorize (if likely valid data). '

and

b.' I might be inclined to present means (SD) if using boot-strapping. I’d probably Winsorize problematic outliers first, as they may otherwise have undue influence on means.' )

I became unsure of whether this is referring to the outliers in the independent variables (b.) or those found in the differences between scores (a.) - because they are present here as well...i.e someone whose scores have improved higher than everyone else (they might be from a different population as they have for example adaptive strategies that surpass general population which allowed them to gain a higher benefit from said intervention - I guess that's a point of interest/hypothesis to test, but similarly one could say the same thing about outliers found during baseline measures); and because testing assumption of normality requires differences between scores.

...and, after deciding what to Winsorize for testing the normality assumption, whether I'd have to use the Winsorized data either way (as proposed in b.) regardless of whether using this data continues to violate the normality assumption (as proposed in a.); running the bootstrapping with it to not have 'undue influence on means'.

I will find the book you recommended very useful I am sure, but currently, my time is limited; and in terms of what I have already read, I haven't found a definition that explains whether there are different types of outliers/if there is a different approach for within-subjects. There is something instinctive in studying psychology that is saying - outliers can exist at any level...

The world of statistics is vast and I can only assume, as a beginner, that there is a lot of information that I do not know, but should be aware of. COVID and not having my usual discussion and support groups have made this 10000 times harder.

Dear Mara,

time constraints are most often a problem, but do not excuse improper analyses, especially if you are aware of it.

First of all, according to your description, your application of winsorizing is wrong. You do not pick some values (using z-scores for outlier detection is not recommended. which cut-off did you use?) you dont like and "winsorize" them. Winsorizing applies to your whole vector and you define how many % of the data left and right are winsorized. The same is true for trimming. There are several inferential techniques that use trimmed means and the winsorized sample variance for example, but not single winsorized or trimmed values. This is more in the vein of p-hacking to change single "wrong" values.

What do you mean with "different types of outliers"?

Maybe you can check also this book by Rand Wilcox: https://www.amazon.com/-/de/dp/1119061393/ref=sr_1_4?__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&crid=367JVUXDIVWPL&dchild=1&keywords=rand+wilcox&qid=1597916787&sprefix=rand+wilco%2Caps%2C287&sr=8-4

It is less technical than other book from the same author, but similarly easy to grasp as the book recommended by Daniel, but here he also delivers the R codes/examples for all analyses directly in the book itself. Chapter 11 is dedicated to caparison of two dependent samples.

Thank you Rainer, of course, hence why I am trying to understand it with the resources available.

I am using SPSS and following Field's recommendations ( https://uk.sagepub.com/sites/default/files/upm-binaries/52062_Field_4e_SPSS_II_proof_Ch_05.pdf ): ‘95% with absolute value less than 1.96, 5% (or less) with an absolute value greater than 1.96, and 1% (or less) with an absolute value greater than 1.96…no cases above 3.29’. Same with Winsorizing: ' Winsorizing: Substitute outliers with the highest value that isn’t an outlier. '. I was actually aware of Winsorizing variance after researching for the past two days for an answer, but I was not sure if this was the same thing considering the dissonance between the definitions...so I went along with what I studied and was recommended to use by my supervisor. I have read about p-hacking as well...I hope you understand my agitation with these conflicting views.

'different types of outliers' - are outliers only considered at baseline, at post measure or as a difference in scores in case of dependent samples? Thank you for suggesting Chapter 11 above for informing this decision.

Compare your paired samples t-test results with and without outlier/outliers.

If results are the same, retain outlier/outliers.

If results are different, closely examine the outlier/outliers and ask yourself are they are conceptually and empirically defensible?

If yes, retain outliers.

Since your sample size is small, the effect of outliers on mean may be significant.

I agree also Rainer Duesing , because your sample size is small, using z-scores as a outlier detection method may not be appropriate. Instead using graphical methods of outlier detection may be more useful.

I like a article about this issue is, you can also check this article.

Aguinis, H., Gottfredson, R. K., & Joo, H. (2013). Best-practice recommendations for defining, identifying, and handling outliers. Organizational Research Methods, 16(2), 270-301.

Best wishes,

Good luck with your study.

Mara Nicola I haven't read all above replies. It seems that you are too concerned about outliers and winsorization. My recommendation is to use a paired Yuen's t test, if you are urged to feel that you need something more sophisticated.

Mehmet Sinan Iyisoy yuen's test for dependent samples uses winsorization, doesnt it?

Rainer Duesing Yuen's independent t test uses trimmed means and Winsorized variances. But I am not aware what it uses for the dependent case (I can't reach Yuen's paper right now).

I have come accross an interesting (maybe, old) review on modern methods by the way while searching. Namely,

Article Modern Robust Statistical Methods An Easy Way to Maximize th...

Ertuğrul Şahin , can I suggest doing that in the other order. Ask if the outliers are conceptually defensible (which I assume means part of the target population) and then exclude or not. Doing the tests first, looking at the results, and then deciding sounds like a recipe for p-fishing.

Dear Daniel Wright as you know,

As a general rule, data screening is conduced with order checking 1. accuracy, 2.missing values, 3. univariate and multivariate outliers (in case of multivariate analyses) and 4. examining assumptions of used statistical analyses.

If any problem arise any of each stage, the problem must be solved in that stage before going other stage.

I recommend this as a robustness analysis not for changing the order e.g. Is the intervention significant in the presence or absence of outlier/outliers?, If I can say yes, without outliers I am more confident to my results

researcher must decide what to do in third stage before conducting statistical analyses but in this case, up to me as robustness analysis ıt may be useful.

However, as you are a well known expert in the field, your interpretations are more likely to be more accurate than me, and specific recommendations for this case may be more useful from your statistical point of view for researcher and us.

I liked the point a recipe for p-fishing, I did not think that before you say.

have a good work and thanks for your novel interpretation, it helped me to realize something new.

Mehmet Sinan Iyisoy Thank you for suggesting the Erceg-Hurn & Mirosevich's article - it's a great summary to this discussion. Also, I came across Keselman et al.'s (2008) robust approach recently, but this article made it very easy to understand.

I've also found a short description of the alternative Winsorising technique used in the Andy Field reference (recommended by my course):

'These include modified Winsorization, which involves transforming only one data point (i.e., asymmetrically) in the distribution'

found in Reifman, A., & Keyton, K. (2010). Winsorize. In N. J. Salkind (Ed.), Encyclopedia of Research Design (pp. 1636-1638). Thousand Oaks, CA: Sage.

(not that this is correct - as @Rainer Duesing suggested it could be classed as 'p-hacking' )

Thank you everyone, I highly appreciate your willingness to share your expertise! This community is so valuable and I am learning much more than I did during my course.

Dear: Outliers must be detected and addressed before analysis

Just to add another good reference for dealing with outliers by Ratcliff: https://psycnet.apa.org/doiLanding?doi=10.1037%2F0033-2909.114.3.510

I'm a bit late to the party, but my article (and dissertation) on the Winsorized t-test may be of interest: https://www.journalcjast.com/index.php/CJAST/article/download/8256/14711