Hanneke Leeuwestein You need to consider the results from both a theoretical and statistical perspective. I'll break my comment down into a discussion of each:

Theory:

-(Content validity) Do you think your questions captured all the dimensions/aspects of the construct?

-(Construct validity) Do you have enough questions to adequately measure the construct? Six questions per factor is on the low side, but not unheard of. Usually, a researcher starts a factor analysis for a new measure with a large number of items (far more than you'll probably need) for each factor - anticipating that you'll be dropping uninformative items. Bear in mind, this depends on the nature of what you're measuring; namely, how complex it is. Some constructs can be adequately measured with 3-4 questions, others might require 10-15.

Statistics:

- Usually a researcher specifies a cutoff for factor loadings. For example, only items with a factor loading of at least .4 are kept. If you want to be more strict, you could use a .6 or .7 cutoff. Then re-run the analysis and see how this changes the covariance structure.

-Depending on the software you're using, you can see how overall variance changes if each individual item were removed. For instance, there are packages in R that allow you to conduct such item response analyses.

Moving forward:

-After you have a satisfactory EFA, researchers generally strive to further establish the construct validity of the new measure by seeing how scores on the new measure correlate with scores on related measures. For instance, I might have participants complete my new questionnaire and also a questionnaire for a related construct (aka convergent validity). The logic being that participant scores should be correlated across the two measures - if I'm indeed measuring what I think I am. (For instance, if there are 3 current tests intended to measure life satisfaction, I should score similarly on all 3 tests). At the same time you can see how participant scores on the new measure correlate with UN-related constructs (aka discriminant validity). The logic here being that participant scores on my new measure should NOT correlate with scores for an unrelated construct.

- You might also assess the test-retest reliability, criterion validity, etc, etc,. It depends a bit on what you're measuring and which method(s) are most feasible. In the end, you want to subject your new scale to as much scrutiny as possible to ensure that it's a reliable and valid measure before publishing it.

At the time of items development while designing the questionnaire, you had to ensure conceptuality i.e. face and content validity of items along with psychometric properties. Moreover, while designing the questionnaire, guidelines for reducing the possibilities of common method bias laid down by Podsakoff et al. (2003) also needed to be followed. At last, before going for final data collection, you should have gone for piloting with a small sample to see whether items are loading in the desired direction or not.

@Blaine Tomkins, thank you for your extensive answer. I am indeed aware of the importance of assessing other types of validity. We also collected data for this, e.g. 3 related questionnaires were filled in to assess convergent validity and divergent validity, criterion validity is taken into account with other measures and test-retest reliability was also assessed. All these things look good!

So for further development of the instrument, my concern really is the factor structure of the questionnaire which isnt as expected and is theoretically difficult to interpret. Do I understand correctly that you suggest to remove all items with factor loading lower than .4 (or stricter cutoff score, as you mentioned), and do this all at the same time? And then rerun the exploratory factor analysis?

@Imran Anwar, thank you! I wasn't too clear in my outline above, but before collecting data from these hundreds of people we defenitely did these things (face validity with an expert group, content validity since our items have a strong theoretical foundation). You're right indeed I should have executed a factor analyses with a small pilot sample, but we didnt. Do you have any suggestions how to continue? Thank you for your reply!

Hanneke Leeuwestein Yes, since these items are not very good measures of any particular factor, they should be collectively discarded or otherwise modified (the second option would require collecting more data).

When it comes to EFA, my approach is to let the data do the talking. If the structure is contrary to expectations, first thing I do is check if I'm modeling the correct structure (orthogonal vs. oblique). Next, I check the "problem" items to see if there are any glaring issues (i.e., poor wording, negative phrasing, etc,). If everything checks out, too bad for my expectations!

I've created many items for new measures in the past, and sometimes I'll have what I think (intuitively) is a great item, but the data show otherwise. Or I expect a certain structure among my items, but observe something quite different.

Hanneke Leeuwestein, you have received some excellent advice above, and you have also responded in ways that are appropriate.

Perhaps I can add some things that might be helpful. First, I recommend that researchers use parallel analysis rather than the Kaiser criterion or scree plot to determine the likely number of factors in their data. Maybe you did use parallel analysis, however. In case you need information about it, I'm attaching a document that might help.

Second, I would remove only a few items at a time in iterative steps in EFA because sometimes the items that looked a bit "bad" end up improving when the worst items are removed. Perhaps you could do two series of EFAs, one in which you were extra cautious and removed only the worst items each time, and a second series that was a streamlined version of the previous series that you'd write up in a report /article. The second would be less fussy to recount, but it would be informed by the more cautious set of analyses.

Third, I'd pay some attention to items' communalities as well as their loadings. Hair et al. (2014) recommend doing that, and I've found it a good idea. Here's the reference for them:

Hair, J. F., Jr., Black, W. C., Babin, B. J., & Anderson, R. E. (Eds.). (2014). Multivariate data analysis (7th ed.). Harlow, UK: Pearson.

Finally, I almost always find that an oblique rotation (promax??) produces a more interpretable solution than does an orthogonal rotation. That's understandable because items chosen in an EFA are usually related to each other.

All the best for your research.

Robert Trevethan Thank you for your answer! The parallel analysis information is helpful, thanks, I will try it. We indeed dit a oblique rotation, and currently assess item communalities and complexities in order to decide which items to remove. This brings me to another relevant question, if I may....

Blaine Tomkins Imran Anwar Robert Trevethan The EFA I described was based on a sample with regular students and in this sample, we expected a right-skewed distributon of scores (which was indeed the case). However, the aim is to develop an instrument that is useful for different populations (regular studets, refugee students, special education students). For the 2 other 'groups', we also collected data (until now around 60 for each group, more will follow). Our first idea was to use the regular student population for instrument validation (EFA, convergent validity, criterion validity) as a kind of reference group. However, I'm currently doubting whether we should include the other 2 groups in the EFA as well: the distribution of items vary quite much between the groups (les right-skewed distribution). I don't feel too comfortable removing items based on the EFA of the regular sample, it could very well be the case that e.g. all students from the regular sample score extremely low on certain items but these items look more important for the other groups.

I'm curious whether you think it would indeed be better to include these other populations in the EFA as well. Or, alternatively, should we do multiple EFA's for each population (but then we need way more data, which is not feasible I'm afraid).

Thank you in advance for sharing your ideas on this, if you have any!

Hanneke Leeuwestein, I'm happy to submit my two-cents worth, and also happy if others disagree with me because it's often from disagreement that we learn.

I think there is a risk of getting unreliable/invalid EFA results if samples are too small, but in the last few years I've discovered that the rules of thumb such as needing 10 x as many participants as there are items do not always apply. See, for example, the following article:

de Winter, J., Dodou, D., & Wieringa, P. A. (2009). Exploratory factor analysis with small sample sizes. Multivariate Behavioral Research, 44(2),147–181.

The authors of the following article also discuss situations in which smaller-than-usual samples might be adequate:

Gaskin, C. J., & Happell, B. (2014). On exploratory factor analysis: A review of recent evidence, an assessment of current practice, and recommendations for future use. International Journal of Nursing Studies, 51, 511–521. https://doi.org/10.1016/j.ijnurstu.2013.10.005

So, I would be inclined to run separate EFAs unless some of the samples are extremely small. If I were you, I would also look carefully at the nature of the raw data and some basic descriptive stats from them (e.g., means, SDs, and interitem correlations).

I guess there are also tests of measurement invariance that you could apply to identify whether the comparison groups are sufficiently similar in terms of factor structure to be used in your analyses.

Robert Trevethan thank you a lot! Could you explain why seperate EFA's for each group are more favorable than an EFA with all groups combined? In the end, we aim to make it an instrument valuable for all these types of students, also the regular students. It is mainly relevant for those who score 'high' (comparable to most of the students from the other 2 groups). Therefore I'm also considering doing one EFA with all this data combined.. But perhaps there's a fallacy in my reasoning here.. I would be very glad to hear more about the different considerations on this decision.

Hanneke Leeuwestein, I suggested conducting different EFAs because the factors could be quite different depending on the samples. If the factors were different, and if you were later to compare different groups, it might not be possible to do that meaningfully.

Furthermore, if you combined data from different groups, you might end up with a factor pattern that applies to none of them.

I suspect that the more you become familiar with your data and how it shakes out, the better.

I hope that's helpful.

Yes, thank you. I'm already quite familiar with the data (a.o. inspection of means, SDs, range, group comparisons more).

I get the point! On the other hand, for the final instrument development we do not aim to focus on group differences. Just a focus on differences in scores and patterns where some type of students may be overrepresented in a 'category' of certain scores (e.g. high on one factor, low on another factor) compared to other types of students, but based on the data I think it will be a mix. Therefore I thought of combining everything in one dataset en conduct one EFA. Well, I'm notr trying to convince this would be the best method, but just wanted to explain a bit more and share my reasoning and doubts. ;-)

Hanneke Leeuwestein, I'm quite happy to walk / have walked with you on this. Doing research seldom permits us easy answers to simple problems. I think I've gone as far as I can to help you - though feel free to get back if you think I might be able to help more.

Maybe some others could step in and make helpful suggestions.

But apart from that, I'd be interested to see what became of your use of parallel analysis. Was the outcome similar to what you'd found before, for example?

Thank you again Robert Trevethan , I'm grateful you walked with me in this! I will run the parallel analysis asap (before continuing with the other analyses), and will report back to you!

Hanneke Leeuwestein If you notice clear differences in factor loadings across the different groups, I would not recommend including all the groups' data for the purpose of instrument validation.

I think the procedure should be: validate a measure first, then test the measure with a different population(s). Trying to do both at once leads to circular reasoning: I'm using a new measure to look for different response patterns among subgroups, I'm using different patterns among subgroups to validate my measure.

P.S. Recall that the goal of EFA is to assess the items, not the participants. It's fine for subgroups to differ with respect to mean, SD, etc,. So long as the consistency among items is the same for all subgroups, you're in the clear.

Blaine Tomkins, I very much appreciate the points you are making. One thing that concerns me is how it is possible to determine that "consistency among items is the same for all subgroups".

Essentially, that's about measurement invariance (MI), isn't it? And if so, I wonder about the extent to which MI needs to be established. At which point do researchers tip from allowing some discrepancy in the consistency between items to becoming, well, obsessively demanding about equivalence?

Do you have any ideas / suggestions? At the moment, it's an issue that's confronting me, but I'm not sure how to move forward responsibly with it.

Robert Trevethan I should have clarified that I'm using the phrase "the same" very loosely above. Expecting perfect consistency would be absurd indeed. After all, one can observe variability with repeated testing of the same individual on the same measure in some cases.

You could conduct item analyses with the data of each subgroup separately (assuming you have an adequate sample), looking at inter-item correlations and variance metrics in order to see if the results are roughly similar to the original results (with all participants included). Deciding how similar they need to be is, I think, to some extent subjective.

Maybe this is a situation where only discrepancies that are deemed 'obvious' should be considered problematic. For instance, if total variance for an item set is 78% in one subgroup, but 35% in another - something major is going on with the sample.

Blaine Tomkins, thanks for getting back. I agree with all of what you've written. I'm not sure whether you're aware of the following by David Funder, but I found it refreshing:

https://funderstorms.wordpress.com/2020/01/31/misgivings-some-thoughts-about-measurement-invariance/

Cheers.