Hello everyone.
I am planning a one-group pre-/post test design (I am aware of the limitations, including a control group is unfortuntately not feasible) and want to measure whether var 1 predicts var 2 at time 1 (simple regression); whether the intervention changes the value of var1 and var2 (paired t-test); and whether the change (if existent) in var1 predicts the change (if existent) in var2. In brackets are my suggestions on how to approach these questions statistically.
First question: I will have a small sample (violation of assumptions incoming) and might therefore conduct a Wilcoxon Signed-Rank test instead of a t-test, does this make sense?
Second question: Is there a better way to establish the change in this design (again, also considering that I have a small sample size)?
Third question: What is the best way to go about the last question, so whether change in var1 predicts change in var2 (again, also considering that I will have a small sample)?
Thank you.
Hello Carina,
It's not sample size that determines whether parametric (e.g., t-test) or non-parametric (e.g., Wilcoxon signed-ranks sum test) is more appropriate; it's the assumptions about data scale/strength and normality. A very real problem with evaluating normality assumptions when you have very small samples is that the tests are typically underpowered (as well, small samples are more volatile than larger samples).
So, to your questions:
1. Do scores on var 1 predict scores on var 2 at time 1? If you think that the scores are interval strength, then the Pearson product-moment correlation will address this. You're looking for whether the correlation is different from zero. If you think the scores are of ordinal strength, use the Spearman correlation.
2. Do scores change from time1/var 1 to time2/var 2? If you think the scores are interval strength, the paired t-test is a better choice (see qualifier). If you think the scores are ordinal strength, the Wilcoxon test is a better choice. Qualifier: you can always use bootstrap/resampling estimation for the probability, which allows you to ignore distributional assumptions.
3. Do var 1 changes (from time 1 to time 2) predict var 2 changes (from time 1 to time 2)? This implies using difference scores as the variables of interest (e.g., score on time 2 - score on time 1). There are a number of technical problems associated with difference scores, one being that, unless the scores are perfectly reliable, difference scores will have lower reliability than either of the two scores used to create them. That results in less power to detect relationships or differences. So, before rushing into this one, first you need to be confident about the technical quality of the scores.
If you are confident, then the same answer for #1 would apply for #3.
Both for #1 and #3, the analog method would be either (a) simple, bivariate linear regression; or (b) ordinal regression.
Good luck with your work.
It is also whether you are wanting to estimate a parameter! I assume you'd like to estimate the size of a possible intervention effect.
Question: What is your sample size? Also, a scatter plot post with pre will be the best statistical procedure I can recommend. This can also help us to understand what approaches might be useful. How closely associated are pre and post (and for example, are some people at pre at ceiling)?
You may run regression (despite the fact that you have a small sample size), adjusting for your var1. This way, everyone in your sample starts at the same place and the estimated changes would reflect the true-r changes adjusting for other baseline variables. Refer to the graph on slide #8 from this PDF file: https://opa.hhs.gov/sites/default/files/2020-07/tpp18-pre-post-webinar-slides.pdf
- Badges
- Science topic