Let's say the three items are all True/False questions. You ask three of them to get a bit more reliable measure because people can guess correctly often. You take the number correct for this (or a weighted average which is PC), and then use this in a PCA with a bunch of other questions. I don't think people would object to this. True, you will get a slightly different solution if you do it all in one, but if you want a single variable to stand for those three responses, nothing against this.

Also, it depends what you are using the analysis for. If the analysis is to construct some smaller number of variables that tell a good story, then if the variables tell that then the proof of the pudding is in the eating (i.e., if you see patterns in the data you did not see before, then the procedure showed you something). This is approach is different from fishing for a significant result. The purpose of PCA/EFA can be to reveal patterns. Revelation is one of the goals of exploratory data analysis according to Tukey.

If you already know what you are looking for, I'd probably follow Albert's advice.

Dear Michalea Otis,

If I understand correctly, you are trying to remove redudancy in your problem. As mentioned, the three similar question items are grouping or getting resolved to one factor, It does not matter whether you are going to add other question items to the new factor or treating all question factors together.

Dear Michaela,

I would recommend You to use other factor extraction methods such as maximum likelihood (if Your data met the assumption of multivariate normality) or principal axis factoring, because the main aim of PCA is only to reduce the number of items and using PCA You can speak only about components not the factors. Moreover You should bear in mind that to distinguish the number of factors You should use e.g. parallel analysis or minimum average partial instead of Kaiser criterion or scree test.

Unless there is a theoretical reason for analysing the 3 items as one variable I see no reason why you should not (re)analyse the 3 items separately with the additional items.  They will cluster as a separate factor if they truly are distinct from the other items.

Dear Michaela,

The most important question in your problem is not technical or statistical. This is theoretical. Why? You have to justify the inclusion/exclusion of any items by means of a theory-guided process. Content validity is the most important source of validity evidences from a questionnaire. The rest of sources are nested with the contents of the items. My question is: how many items could embrace the operationalized definition of the construct you are trying to measure? Is it really necessary to include three indicators for the same content, for the same part of the construct? Are all the other parts of the construct equally represented? In summary, are you overrepresenting this content? First of all, my counsel is to answer these questions, and after that implement a dimensionality reduction technique. In this point, I suggest a CFA (Confirmatory Factor Analysis) for testing the latent structure a priori. PCA could capitalize chance and overfit the best latent model for your sample, but not for the population. I think it is better to do a CFA if you are sure about the structure (dimensions and item assignments). As a psychometrician, all the test construction process must be theory-guided.

I hope you could develop an efficient instrument.

All the best.

Michaela,

There is a procedure called an extension analysis that can be used to see the correlation between a factor (core items) and other items (extension items) in an unbiased fashion. This is done automatically through UniMult (http://unimult.com/) but can also be done in the 'psych' package in R (fa.extension).

Raf

Let's say the three items are all True/False questions. You ask three of them to get a bit more reliable measure because people can guess correctly often. You take the number correct for this (or a weighted average which is PC), and then use this in a PCA with a bunch of other questions. I don't think people would object to this. True, you will get a slightly different solution if you do it all in one, but if you want a single variable to stand for those three responses, nothing against this.

Also, it depends what you are using the analysis for. If the analysis is to construct some smaller number of variables that tell a good story, then if the variables tell that then the proof of the pudding is in the eating (i.e., if you see patterns in the data you did not see before, then the procedure showed you something). This is approach is different from fishing for a significant result. The purpose of PCA/EFA can be to reveal patterns. Revelation is one of the goals of exploratory data analysis according to Tukey.

If you already know what you are looking for, I'd probably follow Albert's advice.

Thank you very much for your responses. I found them very helpful. I am using PCA for exploratory purposes with the idea to reduce dimensions from questions that were already theoretically justified, which might then be used independently. Therefore, it has been decided to proceed with extracting a dimension incorporating a previously extracted variable along with other items. Thanks again.