I am analyzing learning outcomes from an educational intervention using analysis of pre/post survey data and conducting factor analysis to reduce the number of dimensions in the analysis. I have been using PCA-based approach in SPSS. I would like to construct scales that combine survey items and reflect latent variables (e.g., content knowledge, affective response, motivation to change behavior based on intervention). In doing so, I would like to use factor analysis and reliability testing to find out which survey items hang together, calculate scales for pre and post survey data separately, determine the post minus pre value in those scales (change as a result of intervention), and use regression and other approaches to analyze the statistical significance and drivers of those changes.
My questions are:
- Should pre and post survey responses be combined in factor analysis?
- If yes, in constructing scales, how should items be weighted and decisions be made for which items to include? I.e., I assume that the same items used to construct a given pre-scale should be included in the post-scale and that they should be weighted based on loading values?
Thanks for any feedback.
Dear Juliette,
preferably of respondents before and after combined in the factor analysis. should the data coming from the respondents' answers were analyzed validity and reliability in an excel sheet. I submit the following data is how valid and reliable.
Dear Juliette,
Instrument factor/reliability analysis and data analysis are separate issues. I would not combine answers together in the former process. Perhaps the best approach is to use your pre intervention survey data for the former.
If you are trying to establish scales you should first carry out an Exploratory Factor Analysis (EFA). There are various different forms of this. The one you have mentioned (Principal Component Analysis, PCA) has a drawback as it measures total variance rather than shared variance. This means it does not provide a measure of the ratio of the two, known as communality. It is wise to remove any item with a communality score less than about 0.2. I would therefore only use PCA later - see below.
As you are planning to extrapolate your findings to a post intervention survey I would recommend a sample-based inferential technique, such as maximum-likelihood or Kaiser's alpha.
Before running an EFA, check for high bivariate correlations between items and remove one item from any pair with a value grater than 0.8.
Use an orthogonal factor rotation, such as varimax. This should improve the quality of your model without the drawbacks of oblique rotations.
When you run the EFA, check the factor scores. Suppress any less than 0.3 in absolute value (an option in SPSS). You are looking for items with scores above 0.4 on single factors. You also need to take into account your sample size and there are several rules for deciding how many factors to include. For a factor to be included it should have at least three items with a component loading greater than about 0.4 and low loadings on the other factors. This might mean changing the number of factors you are extracting. Remove unwanted items one at a time, starting with those which are least useful.
Once you have done this, select these items for each factor in turn and run a PCA on them, checking the size of the largest eigenvalue and its component scores, which should again all be above 0.4. You can then save this as the weighted regression model and use this in your analysis. See my guide on the subject:
https://www.researchgate.net/publication/280936182_Advice_on_Reliability_Analysis_with_Small_Samples
The sources I have looked at are Field, 2013, Chapter 17 and Tabachnick and Fidell, 2014, pp. 660-729.
Technical Report Advice on Reliability Analysis with Small Samples
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS