Hi Md Nawaz Sarif

I am not familiar with the strategy you propose, but that may just be me. Why do you not want to use EFA? Just a sidenote: Usually, you do not want components but actual factors (EFA with oblique rotation). Also, there are many more indicators for removing individual items including theoretical considerations.

Best

Marcel

Marcel Grieger thankyou for your response, I have reviewed a very few studies where researchers have used t-value with 27% upper cases and 27% lower cases for item reduction. However, I'm at the beginning of the scale construction process and have just developed items based on theoretical constructs. Here could you please let me know few more indicators for removing individual item besides theoretical consideration that you mentioned if I go for EFA.

Hi Md Nawaz Sarif

like I said, I have never come across the t-value being used in that fasion, but then again I am not a statistician. Other criteria for eliminating items are:

- low MSA

- low communalities

- multicollinearity/redundancy

- substantial cross-loadings on more than one factor --> probably the most used indicator because it is easily detectable

- increase in realiability (McDonald's Omega or Cronbach's Alpha) if item is deleted (usually corresponds with low inter-item correlation)

For more specific cutoff-criteria I would kindly refer you to any textbook on the matter. Most of my material is in German, but I do recommend Andy Field and you will also find many articles on Google Scholar.

Best of luck

Marcel

Thankyou for the suggestions and insights Marcel Grieger

Md Nawaz Sarif, Marcel Grieger has given you some excellent advice. The following references, all in English, might be of use to you:

Beavers, A. S., Lounsbury, J. W., Richards, J. K., Huck, S. W., Skolits, G. J., & Esquivel, S. L. (2013). Practical considerations for using exploratory factor analysis in educational research. Practical Assessment, Research, and Evaluation, 18(1), 6.

Briggs, S. R., & Cheek, J. M. (1986). The role of factor analysis in the development and evaluation of personality scales. Journal of Personality, 54(1), 106–148. https://doi.org/10.1111/j.1467-6494.1986.tb00391.x

Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309–319. https://doi.org/10.1037/1040-3590.7.3.309

Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10, 1–9.

de Winter, J., Dodou, D., & Wieringa, P. A. (2009). Exploratory factor analysis with small sample sizes. Multivariate Behavioral Research, 44(2),147–181. https://doi.org/10.1080/00273170902794206

Fabrigar, L. R., Wegener, D. T., MacCallum, R., C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272—299. https://doi.org/10.1037/1082-989X.4.3.272

Gaskin, C. J., & Happell, B. (2014). On exploratory factor analysis: A review of recent evidence, an assessment of current practice, and recommendations for future use. International Journal of Nursing Studies, 51(3), 511–521. https://doi.org/10.1016/j.ijnurstu.2013.10.005

Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191–205. https://doi.org/10.1177/1094428104263675

Matsunaga, M. (2010). How to factor-analyze your data right: Do’s, don’ts, and how-to’s. International Journal of Psychological Research,3(1), 97–110. https://doi.org/10.21500/20112084.854

Preacher, K. J., & MacCallum, R. C. (2003). Repairing Tom Swift’s electric factor analysis machine. Understanding Statistics, 2(1), 13–43. https://doi.org/10.1207/S15328031US0201_02

Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432–442. https://doi.org/10.1037/0033-2909.99.3.432

Incidentally, I'm pretty sure that MSA refers to the measure of sampling adequacy, also known as the Kaiser-Meyer-Olkin index.

All the best with your work.

Robert Trevethan Thank you for sharing those articles.

Md Nawaz Sarif, you're very welcome. I should, perhaps, have said that I have never seen the 27% upper and lower setup. Are you able to give us an explanation for that "rule", or provide a citation or two regarding it?

Incidentally exploratory factor analysis, as well as interitem correlations, could be used to reduce the number of items in a data set. Marcel Grieger provided some ideas concerning that in his most recent post above here. Items that don't load satisfactorily on any factor are also candidates for removal.

Of course, you would need to have a sufficient number of participants to conduct factor analysis. If you don't have enough participants, examining such things as patterns within interitem correlations might be the best you can do.

First, I don't understand what you are doing. Can you spell this out in more detail? In general I would be cautious using any fixed statistical criterion based on sample data as definitive.

Second, you haven't said what the purpose of the items are and why you want to reduce the number. Is this a multidimensional scale?

Robert Trevethan, Hi, I have attached the following references where t-value with 27% upper cases and 27% lower cases is computed for items of a scale.

Bozdogan, A. E., & Uzoglu, M. (2012). The Development of a Scale of Attitudes toward Tablet PC. Online Submission, 2(2), 85-95.

Zembat, R., Kuru Turaşlı, N., Güven, G., Sezer, T., Akşin, E., Yılmaz, E., & Bayındır, D. (2016). Validity and reliability of the DeMoulin Self-Concept Developmental Scale for the 36-72 month old children.

Daniel Wright, Hi, I'm constructing a scale for assessing students' related constructs i.e., extraversion, resilience, mindfulness, etc. and yes, it is a multidimensional scale.

Md Nawaz Sarif, thanks for sending me the references for two articles. Unfortunately (and sorry), I find them both so badly written and filled with trivia that I didn't look carefully at either of them and would not put much faith in them at all. The texts were so painful that I couldn't bear to spend time on them. The first article also contains what are, in my view, some glaring misconceptions - for example that 30% is a sufficient amount of the variance being explained by the retained factors. (I like to see at least 50%, and preferably up around 60% of the variance being accounted for by the retained factors.) From the scree plot, also, there are probably three factors in their data, not one.

So, although these researchers have used some procedures involving items with something at 27% and t-tests, I don't put any trust in their analyses. Unless I overlooked something in these articles (and I could well have because of my haste to get away from them), I can't see where they got the idea that working with the 27% was valid.

I would look at other methods for identifying which among your items are satisfactory, and which not.

Please do feel free to get back if I might be able to help more.

Hey Md Nawaz Sarif

I absolutely agree with Robert Trevethan - sorry Robert, I wanted to highlight your name, but RG keeps killing my text. Weird.

Neither articles instills any confidence in me and that is only partly due to their bad English and poor formatting. The funny thing is that they do use some elements of EFA, but they miss vital parts and I still cannot wrap my head around the 27 %. Long story short: Stick to the EFA (or maybe CFA in your case).

Best

Marcel

Marcel Grieger: Thanks for the support, Marcel. It seems I'm not the only one who can't get RG members' names to appear where we want them to at times within posts.

With regard to the two articles in question, I'm often fascinated by the dreary, trivial, misleading manuscripts that can be accepted for publication. Where are the good reviewers? Where are the good researchers?

Thank you Robert Trevethan and Marcel Grieger for insights on 27% upper and lower setup. Yes, I understand and do agree with what you both agree upon, if I use EFA and then CFA for constructing and establishing scale then the scale would be more valid. And, yes, EFA and CFA are more sophisticated and acceptable techniques compared to 27% setup for developing a new multidimensional scale.

Md Nawaz Sarif, you're welcome. If you do use EFA, then CFA, please be aware that you should not use the same data for both sets of analyses. At least, you shouldn't go from EFA to CFA using the same data. Perhaps you could try CFA first, and, if it didn't work out, you could use the same data to conduct EFA.

Every good wish for your research.

Hello all,

The upper 27% vs. lower 27% comparison comes from many decades ago, in classical item analysis. It is one way to assess item discrimination: whether scores on a specific item correlate (positively, one hopes) with scores on the overall measure. Where did the 27% come from? It would correspond to scores beyond +/- 0.75 SDs from the mean in a normal distribution.

Is this (t-test/means difference test) useful? Not any more so than ordinary Pearson (or point-biserial) correlations would be. In fact, possibly less so, because: (a) statistical significance is sensitive to sample size: with a large enough N, any mean difference in extreme score groups on item performance will show up as "significant;" and (b) you're actually discarding data in comparison with the correlation coefficient (which uses all cases in the data set).

As others have suggested, factor analysis will likely give far better insight as to the structure of a measure than simple item discrimination (though this is one starting point; inspecting inter-item correlations would be another).

Best regards to all.

David Morse, thanks for the explanation about the 27%. Now something makes a bit of sense. However, I can't for the life of me work out how individual items on which there is a significant difference between the top and bottom 27% of responses might be related to a total score, and I can't see how this might be useful in situations where scales that assess "personality variables" are involved. Perhaps, this procedure is useful in testing situations where items are desirable if they can discriminate between the most and least capable respondents / students. Maybe I'm being obtuse . . .

Hello Robert,

If high (overall) scorers--the top 27%--perform/respond differently on an item than do low (overall) scorers (the bottom 27%), then item performance correlates with total/overall performance (though the more direct quantification is an item-total correlation coefficient). If all items manifest positive item-total relationships, that must imply that the items correlate with one another as well. These two features help to offer some assurance that the total score is largely a coherent (e.g., unidimensional) indicator of some trait / attribute / ability. That's the rationale, in any event.

How would it apply in personality scales? Only if: (a) the scale was meant to assess a single characteristic, such as sensation seeking; or (b) in a multi-trait measure (such as a "Big 5" measure, you'd have to restrict the "upper" and "lower" characterizations to a single subscale or subscore at a time.

Hope that helps.