I have undertaken several longitudinal studies using unstructured/semi-structured interviews and investigating the potential for using surveys. I have identified one particular issue with surveys in longitudinal studies - would welcome some robust discussion on whether there is any justification for continuing a longitudinal study with known flawed questions that are not going to be changed. Or, from another perspective, just how much does altering any question undermine the validity of the longitudinal study?
Assuming the interview/survey produces scores that are compared over time, it is permissible to modify questions if equating studies are done with the old and new items and appropriate adjustments are made. That would require that the study be conducted at least once with both the flawed and corrected items so the differences in performance could me assessed. If this were not done, any longitudinal comparisons would not be entirely valid.
By trimming down the initial RQs, we could kill the beauty of longitudinal studies-a longer period of observation renders unexpected/unforeseeable results. Perhaps acknowledging in the result in the form of a discussion would make the research piece an honest and even enlightening one.
I do agree with James and Jae contributions. I am not sure how your interview questions are integrated into the study, but if you are using them to further illuminate the survey results then perhaps you can address the flaw in the survey questions by asking appropriate interview questions and explain the findings accordingly in your discussion.
Many thanks,
Debra
One has to be extremely careful to take up such studies. My experience suggests dynamic behaviour of any situation originally envisaged. Metrics static in nature would tend to bias. I tend to agree with observations made by friends.
If the item in question is part of a scale, then you can examine how it relates to the rest of that scale using some fairly simple quantitative techniques (such as Cronbach's alpha and Exploratory Factor Analysis). If that item is indeed "contaminating" the scale, then I would favor removing it, rather than replicating the errors of others.
David Morgan's made me realize that Edward's question is not about research questions but survey questions.
I think that what Edward means by "flawed' and David describes as 'contaminating' should be clarified. Possibility of eliminating some questions in the follow-up survey could depend on the content of every such questions. To set a simple example, suppose it had a question on the marital status. Edward could describe it as 'flawed' in the sense of being irrelevant to the research topic. 'Flawed' can also mean, for example, incompletely formulated (e.g. only married and single options were in the first survey missing out others, e.g., divorced). In the first case, the question could perhaps be dropped. In the latter question, it should probably be modified.
Furthermore, I am afraid that if many survey questions are changed, it could pose the problem of lack of substantive issues to follow up in time (longitudinally). weakening the raison d'etre of the research. The qualitative and quantitative nature of the study also will affect your decision to modify/drop questions. In the case of quantitative studies, I would be extra cautious (surely more than David Morgan) in eliminating 'contaminants'--it could look like manipulating the study method to match initial hypothesis of the researcher, and it will look pretty bad.
A slightly different approach on this might include path analysis, or structural equation modelling (SEM), depending upon your measures. If you have strong theoretical views on this flawed measure and it is a mediator or outcome, you might find providing a 'fix' part-way through useful and substantive.
- Badges
- Science method
- Similar topics
- Mathematical Sciences
- Statistics
- Applied Statistics
- Survey Design