I am going through the long arduous task of psychometrically validating a psychodynamic adherence scale. We wanted to get an idea on which items load strongly on which other items. So we got some experts to view some stimuli that had previously been selected to represent each scale item, and asked them which scale items 'could' be rated on for that specific stimulus. We specifically did not ask for ratings of adherence as this will be investigated in follow-up factor analyses. So we are left with identification frequency data for each perceiver for each item on the scale. Which statistical analysis technique would be most suitable for this type of data with this purpose?
What's your sample size (expert judges)? How many stimuli, and how many scale items? If you didn't have a plan for analysing the data, then how did you calculate your sample size?
Imam, I have never heard of a polychoric correlation matrix but I will look into it. What did you mean by creating dummy variables to produce dichotomous variables? Did you mean a variable where the relevant stimulus to each individual item is given a '1' (i.e. correct) and all others as '0' (i.e. incorrect) and then analysing where the errors are occurring?
Paul, our sample comprises 7 expert judges. Our sample size is limited by logistical feasibility rather than the required sample size for sufficient power. We have 37 items on the scale and we have selected 37 videos that were identified previously as exhibiting high adherence for each item (i.e. 1 stimulus for each item). This data is meant to be more about giving us a preliminary idea of which items can load upon which other items so we can make more informed decisions in the factor analyses that we perform on more rigorously collected data later on.
What I would do is create a matrix of values, 37 rows representing your videos, and 37 columns representing your items. For each cell, you would create a score of 0 to 7 for how many of your experts agreed a particular video could be used for a particular item.
You could do a PCA as Imam suggests, but a sample size of at least 100 is generally recommended for meaningful results, and that's with more than just binary data. I think the best you can do is just examine the correlation matrix I suggest (having it colour-coded or grey-scaled by value to aid interpretation), and use the results to guide your choice of items. I have attached an example of the kind of thing I mean.
Pauls proposal seems me to be a solution to get an idea. But the problem seems me to be that you have a sample size usually used in a qualitative survey and now you want to analyze the results by quantitative techniques. Therefore if your experts would be available for a second round may be a qualitative second round using the results from the first round would be usefull. Another approach could be clustering as this would not only create a item to item result but much more a comparison of expert judgments.
- Badges
- Science topic