Hello,
I have two very important questions. These are as follows;
1- How many respondents do I need for test-retest reliability and criterion validity?
2- For test-retest reliability must I apply the scale to same participants or to different participants? For example, I have collected the first data from psychology students and one month later the second data again from psychology students but different ones.
Thank you in advance ...
Hi,
for your first question it depends on a few factors - how many questions you have, how many factors etc. If your measure is for a particular smaller population then fewer participants are acceptable. I'd suggest consulting Tabachnick & Fidell (2007) for details
for your second question it should be the same participants - you are trying to find out if the same participant gives a consistent response to the items. If they aren't consistent then you don't have a reliable measure of that trait.
BW
Ed
Dear Edward,
First of all, thank you for your response. What I know is, your first reply is about CFA and EFA, right? In other words, the number of respondents depends on the number of items or factors. Is this rule of thump the same for test- retest reliability and criterion validity?
Thank you in advance!!
As Edward said, test-retest reliability should (must!) involve the same people twice.
As for your first question: You don't need to be running factor analyses in order to examine reliability or criterion validity. If you will be running some sort of multivariate statistics (including factor analysis or multiple regression), then many authorities suggest at least 10 people per variable.
In the broader picture, what you're asking relates to statistical power. Sample size is one of the two keys here. The other is effect size - that is, how large/robust do you think the effect will be? This applies whether you are trying an intervention (i.e., experimental research) or just examining a correlation (i.e., observational research). You can find some guidance on power analysis in any decent statistics textbook (often it's one of the chapters not assigned by professors, but it's there). You can also just look at the table showing how large a correlation you need in order to achieve statistical significance with increasing sample size. What you'll probably notice is that you make large gains up to a sample size of 40 or so, more moderate gains thereafter, and pretty much peak out by the time sample size reaches about 200. Besides, statistical significance tests are really intended for moderate sized samples - when you start working with huge ones almost anything will be significant, because after all, everything is connected to everything else.
I guess I'm saying that you don't want to mess around with samples of just a dozen or so and don't need to bother with samples in the hundreds - unless you're engaged in a lot of multivariate work.
Dear Stephen,
Thanks for your explanation. We are going to use the same participants for test-retest reliability.
For test- retest reliability and criterion validity, we only want to see the size of correlation therefore we will use maximum 100 participants.
Kind regards
Hi Both,
I agree with everything Stephen says although I'd aim for 15 participants per variable. The other thing I'd suggest looking at is how strong similar correlations in the literature have been - this will give you an idea of the effect size, which you can cross reference with the table of r-values and p-values
BW
Ed
Dear Stephen
1. For the first question, the total number of participants should be grater than 30.
2. For the second question, and in order to assure the stability of your test, the tests should be developed (tested/retested) for the same group.
Regards,
Zuhair
- Badges
- Science topic
- Similar topics
- Psychology