I launched an online survey of personal religiosity in April and have since received entries from almost 700 respondents (hoping to reach 1,000+). At approx. the 600th respondent mark, I added several new items to the survey that I didn't think of at the outset. Thus, when the time arrives to test for internal consistency and assemble index variables, many of my items will have sample sizes of 700+ while fewer others will have samples of 100+. My question is whether I can, in fact, combine items of different sample sizes into an index variable. Specifically, is it okay to impute the mean of the new (smaller N) items to the respondents that never had the opportunity to answer them? Or what is the recommended procedure in this case? Thanks in advance for all your help!
One approach would be to set up your data file with a code for "missing data" for the original 700 so that only the available data for that one are analyzed.
I'm using SPSS, so in somewhat similar cases (i.e., for items where N=400, which entails 300 'missing' cases) I would compute the index variable in syntax such that it only includes those respondents who answered a minimum of, say, 5/7 questions while assigning them the mean score of the (unanswered) remainders. Is that what you mean?
If I understand your question correctly, you want to assign the mean of the new variable (which you thought of belatedly) to the 600 respondents who did not get asked that question originally. That's one way to do it, and not without precedent, but the danger is that you could be missing some patterns: maybe the respondents who did answer the question answered in certain ways if they were female but not male, or their answer depends on their locus of control level, or something like that. Also you would be weakening the variation of this questionnaire item by artificially assigning the mean of its value to most respondents. This would wrongly limit its predictive power.
It may be better to analyze the responses of the respondents who did actually answer the question and see if they answered it in any identifiable pattern in relation to the other questions or domains in the questionnaire that you're looking at. If you found a reasonably strong pattern then you could, I suppose, "predict" how the first 600 respondents would have answered the question (based on the questions they did answer) and assign the phantom question's answers to them. But that is a strategy best taken when you wish to create a construct of some kind, (let's say 'religious guilt' here for example) that depends on having an answer to the belated question along with some other questions that were actually asked to assess a "guilt" domain based on several questions. In that way you could construct a 'religious guilt' subscore for those first 600 respondents based on their existing answers and what they would have answered for the belated question---but only if you're pretty sure how they would have answered it based on how the 100+ actually did answer it.. But even that could make for questionable validity.
The most likely best course of action would be to validate the belated question as best you can on the 100+ respondents who actually answer it. You can still do an item analysis and still incorporate this variable into an index variable, although you might not have high enough significance level for the belated question to be valid---which could be a sample size issue, or could be because it simply wasn't a good question that discriminates anything. If that's the case---if the belated question is not statistically significant---you should certainly do further analysis to see if it's a sample size issue or something else. If you're doing principal components analysis or multidimensional scaling, you may be able to see where the question "would" fit in an index variable, and then discuss why it couldn't be used with certainty.
If the belated variable turns out to be valid and strongly predictive, especially in a way that stands on its own in relation to other variables, then there should be no reason why you couldn't use it as a component of an index variable. Index variables are used all the time in various fields, often without verification in the study population they're applied to, as long as they were verified originally on an original study population, somewhere. An example is a "health index" of a census tract using GIS analysis where the index combines number of clinics, age of population, death rate, etc. in the tract to come up with a population health score or index for that tract. Those factors were associated with health outcomes somewhere else, although saying they have an effect on health in the census tract you're analyzing now is educated guesswork, really. So yes, you can include the belated question in your index variable even if it is validated on fewer than the full 1000 respondents. I hope this helps.
By the way that sounds like a good response number for your survey, although I don't know what percentage of questionnaires were not answered!
- Badges
- Science topic