Hello Zengfang,

it is always difficult when a supervisor requests something. Even if this is completly nonsensical as in this case, what would be the optimal recommendation.

With a N of 30, even calculating a correlation and reliability coefficient is on very shaky grounds, because of the immense sampling error. That is, no matter how positive or negative the results will look, you cannot generalize this to the population or a larger sample (of the main study).

Instead of estimation, pretests should focus on things about which even a small number of people can inform, for instance, error in the questionnair or filter, or question wordings. With regard to the latter, you do a kind of qualitative approach to check the validity which is far more important and informative than quantitative stats.

Perhaps the attached plot can convince him. It shows a small simulation, where the true alpha was .72. You see three scenarios: N = 200, N = 100, and N = 30.

I hope this is enough to understand that you cannot conclude something meaningfull in the latter case. BTW: of the 1000 samples, one third (231) had an alpha < .65 (which would for many people count as "too low reliability".

I am scratching my head for years, why it isn't common practice to report alpha with a confidence interval. lIke any other statistic, it's an estimate of a population parameter.

HTH

Holger

This approach may require some advanced programming, but you may try to run a factor analysis with small sample bias correction.

The small sample size solution is relatively new, but rapidly growing field in statistical science. Here's a free book for your reference: https://www.taylorfrancis.com/books/oa-edit/10.4324/9780429273872/

Zengfang Lin Factor analysis is regarded as one of the most powerful techniques to demonstrate construct validity and is the most often used method for establishing construct validity as assessed by an instrument.

Zengfang Lin I completely agree with Holger Steinmetz that Cronbach's alpha is one of the better choices for confirming the internal consistency of the scales (for a low sample pilot study ) and is the least sample sensitive compared to factor analysis. While on the other hand, factor analysis is highly sample sensitive given that it works on inter-item correlations and the factor structure is highly likely to change with the increasing sample size due to the variations in inter-item correlation coefficients.

Woa, Imran, wait a minute :)

that was not what I suggested. My simulation was a reaction that she already computed alpha. The simulation showed that even such simple thing as alpha has a huge sampling variance and can NOT be trusted.

We can repeat the exercise with a more complex model but this will rather be worse.

My recommendation: In low-N pretests, don't do any fancy stats and focus on pretesting and errors of the procedure (even 1 person is sometimes enough to locate problems).

All the best,

Holger

D. Eastern Kang Sim Thanks a lot for the recommendation of the book! Really helpful!

Holger Steinmetz Hi, Holger! Thank you so much for your recommendation and explanation!

I agree with you on the focus of the pretests. That's what I did in the pilot study: checking the wording and spelling errors based on experts’ and participants’ feedback. I followed the recommendation of a sample size of 10-30 for a pilot study in many previous studies. But the minimum sample size for run factor analysis is recommended to be 1.5 times of the items (I have 38 items, which means at least 57 respondents are needed). Some people even recommend having more than 100 participants. I am just so confused about the conflicts here.

Thanks a lot for the supporting evidence with the Cronbach's alpha. That’s really helpful. But I am looking for references (like citations) to support the action of not running factor analysis (EFA or CFA) in the pretest, do you happen to have any articles or books to recommend? Thanks a lot!

Hi Zengfang,

take a look into this one:

Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7(3), 286-299.

Beware of using rules of thumb. The sample size depends on the clearness of the factor structure (optimally you have crystal clear items with no double loadings) and the factor loadings.

In the paper, they say that with loadings of .80, even an N = 50 would be ok. I strongly doubt that. From the CFA field, I know that a sample in the 100s would be better:

MacCallum, R. C., Widaman, K. F., Zhang, S. Z., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84-99.

Boomsma, A., & Hoogland, J. J. (2001). The robustness of lisrel modeling revisited. In R. Cudeck, S. du Toit, & D. Sörbom (Eds.), (pp. 139-168). Chicago: Scientific Software International.

I attached this one.

Best,

Holger

Holger Steinmetz Hi~Holger,

thank you so much for the references! Will read them! Great help.

THANK YOU AGAIN!

Best regards,

Zengfang

No it is not.

Hello, Lin... You need 7 to 10 questionaires by each item... 287-410 at least. Best regards.

Hi! If the items are taken from different sources or for the newly developed items researchers need to use the factor analysis for the construct validity. Exploratory factor analysis is used for such instances. However, in case of sample size it is always recommended to have more than 30 (even if for pilot study) just to enhance the reliability and validity of the outcome. Thanks!