If you have enough items left for each factor after removing the item that cross-loads so substantially, I would say go ahead and remove it. Otherwise you have an item that measures both factors, which probably makes it harder for you to justify computing a scale score for the factor that includes the item in question.

Hi Christian and Nadira,

Thank you for your input. You both made valid points. The problematic item only has a |B| of 0.109 with Factor 2 (where I had meant it to load). So it seems like there are some issues with my theoretical definition, or perhaps the wording of the item was unclear. So it does makes sense to remove it.

I will definitely need to take another look and see if this can be salvaged. I did not personally create the survey items, so I wasn't sure how to interpret and present the findings. Thank you both again for your help!

Joanne Lim, given that you have 16 items but only 104 participants, an underlying problem might simply be a lack of participants. The notion that the number of participants should be 10 x the number of items, I think is not a bad rule of thumb.

Unless your communalities are nice and high (e.g., all above .50), perhaps you really are cutting it a bit fine to expect "accurate" factors from emerging out of your data.

From what you have written, none of your items has loaded across both of your items to an extent that is worrisome. That's good. However, I'd look carefully at the item that has loaded unexpectedly on the "wrong" factor in case there is a good reason for that. Nevertheless, I'd go back to being concerned about an insufficient number of participants to drive EFA.

Incidentally, you seem to have done other things correctly, i.e., you used PAF, an orthogonal rotation, and a process of eliminating poorly performing items.

Please be aware that EFA is a data-driven technique, so it might be that your sample is just different.

Sometimes a scale can be inherently wobbly, however. Recently, I have been helping people who have used a scale that, in prior research, yielded three quite different factor structures, and in the research I'm helping with it has yielded two more, quite different, factor structures. In that case, I think the scale's instructions and response options might be the main culprits.

All the best with your work.

Hi Robert Trevethan

Big thank you for such a comprehensive answer! The issue of participant numbers was definitely something I'd been wondering about as well and had been meaning to ask. I referred to Costello and Osborne (2005), and they too emphasised the importance of large sample sizes.

As for communalities, the problematic item had the lowest factor loading (not counting the two I removed for being

Joanne Lim, I'm glad my post was helpful. The Costello and Osborne article is informative and authoritative. If you want more information about sample sizes for EFA, feel free to ask me.

I think we might have a misunderstanding about communalities versus loadings. In SPSS output, the communalities are provided before the loadings, and it's the extraction communalities you need to pay attention to. I think they need to all be above ~.50 if you want to work with a smallish sample.

Your loadings look OK to me.

You could be spot on when referring to wording of items in your scale possibly not being satisfactory. With regard to the scale that I referred to in my post above, I'd forgotten to mention that a number of the items have wording that I regard as quite unpolished. Some wording is even silly.

One last note: Please don't worry if the coefficient alphas for your two factors are not high. With as few as 6 or 7 items, the alphas can be quite low (down around .70, maybe a touch less) without cause for concern. Check out the interitem correlations!

Hi Robert Trevethan

Thank you for your kind offer to help! I do apologise, I must admit I'd not been sure what communalities meant (I had to look it up). I used JASP to run the data, not SPSS. It doesn't look like it gives the communalities directly (I may be wrong here), but it does give the uniqueness, which I think is 1 - communality(?)

If so, then it doesn't look great because the factors' uniqueness are between .34 to .81 (8 out of 13 have > .50 uniqueness). And they are all in Factor 2. The average interitem correlation is .47, item-to-rest is .44 to .73 (Factor 1) and .35, item-to-rest is .31 to .63 (Factor 2). Is this very bad?

Furthermore, do you have any advice on how to report this? More specifically, with my sample size and results, what kind of conclusions can be drawn? I'm hesitant to say this is a valid or reliable scale, but I don't want to bash it either. It definitely needs more adjustment, and to be tested again on a much larger sample.

Thank you again, this has been really helpful information.

Ah, sorry. I reran the data with SPSS to double check. the communalities are between .20 and 63, pretty evenly spread out. That doesn't seem very good.

Joanne Lim, it's good to hear back from you. First up, with some of your communalities being quite low, I suspect that having only 104 participants is too risky. Furthermore, it would seem that some of the communalities are too low to be satisfactory. There are no hard-and-fast rules about how high communalities should be, but it's nice to see them up around .50 or higher (though, in my own research, I've let variables sneak through with communalities down around .45 if they have high loadings on a factor).

I appreciate that you don't want to "bash" the scale you're using, but unfortunately some scales simply are not very good - or they are not very good with a particular group of participants.

I think the only way to report your findings is to be totally honest! :-)

Please let me know if I can be of any further help.

Hi Robert Trevethan

I agree; it wouldn't make much sense to say the scale is anything more than it is. Huge thank you, all of this has been really helpful! I will be working on how to properly critique this study.

Best wishes to you!