Hello Ad,

Given that you've dramatically changed the contents of the survey, I think you're wise to worry about collecting data to support statements about both the reliability of scores and the underlying structure of the survey.

However, for me, the most important question here is: How do you intend to use the preference and expectation judgments? What is/are your specific research question/s that involve the scores from this amended survey? This should guide your choice for how to proceed.

My inclination is that, if you wanted to undertake either exploratory or confirmatory factor analysis, you'd be best served by considering the preference scores only. The reason is that the expectation scores could be far more situation dependent. As well, the two scores are hardly comparable, other than both are measures of affect.

Finally, if you do attempt factoring of ordinal strength variables, I'd suggest you consider one of the following options:

1. Instead of Pearson correlations among variables as the starting point, consider using polychoric correlations instead.

2. Use item response theory instead, as there are several models from which to choose for scaling of ordinal scores. As well, Mokken scaling could be considered as part of this universe.

3. Optimal scaling (a.k.a. nonlinear factor analysis). While perhaps more frequently used within PCA (principal components analysis), there's no reason that transformed scores couldn't be used as input to a factor analysis.

Good luck with your work.

Hi, David Morse,

Many thanks for your kind, thoughtful, informative reply!

My intention with this specific item set is actually to track longitudinal changes in each of those scores -- preference and expectation -- as well as the difference between them -- which I interpret as a sort of "fit" metric. In addition to tracking changes over time descriptively, using techniques including latent class and profile analysis, I plan to model the effect of that "fit" metric (IV = pref - expect, preliminarily analyzed using only beginning-of-year data) on a behavior change metric (DV = behavior change score, which is the difference between end-of-year and beg-of-year responses to a different behavioral scale) -- likely through SEM, though I may be limited (techincally and time-wise) to other techniques.

So, all quite complex. With lots of murky latent vars, conceptual challenges, etc!

Some thoughts:

1. I take your point that preference might prove the most "stable" item type for factoring. I wonder, however, about your thoughts on another approach: creating that composite "fit" score, and conducting regular EFA on that single value. Would this remedy some of the above problems?

2. I will certainly use polychoric correlations as a starting point, great idea. I haven't been able to et the polychoric package to work in Stata yet, but would just share for now that "collin" on all vars showed all VIF < 2.5. And a Pearson revealed low to moderate correlations.

3. Could I possibly get your thoughts on how to better consider whether IRT would be a better model for my Likert-like data? Is this simply because, despite often treating Likert-like repsonses otherwise, this data is ordinal? So, I'd need to recode the responses to each as binary, making it appropriate for a categorical data analysis technique like IRT?

4. Same with optimal scaling: I assume this would be to avoid the normal + continuous assumptions of traditional factor analysis? What would my rationale be for chosing this over polychoric starting point, or IRT? (Note that sktest shows Prob>chi2: 0.000 for all vars, so perhaps this is my only path forward...not sure how much my n = 3000 confounds these analyses)

Thank you, again, for your kind support!

Hello again Ad,

You could use discrepancy scores between judged importance and expectation as a variable, and characterize it as akin to a goodness of perceived fit indicator for an individual.

Whether these resultant values are better handled as a scale vs. grouped, common factors depends on a number of concerns (again, back to specific research question/s, as well as data collection scheme, empirical data characteristics). As you've indicated that these data would be collected at a common time (once; at beginning of term/year), that helps to avoid the volatility influence.

Ordinal IRT models don't require collapsing variable scores into dichotomous form.

LCA can use ordinal scores as well as dichotomous scores for categorical variables. However, if you are successful in scaling or factoring the fit scores, you can argue that the resulting scale scores or factor scores are interval strength, and use one of many clustering methods instead of LCA. I mention this because it sounds as if you are very interested in subtyping your cases into qualitatively distinct groups. Alternatively, cases may be the objects of interest in factoring instead of variables. In Cattell's explanation, this is referred to as "Q-type" factoring.

Good luck with your study.