Hi!
What is the scientific value (in terms of reliability and validity) of customised questionnaires as opposed to standardised questionnaires?
I am collecting data systematically through an online questionnaire survey that I have designed myself, because my research topic does not neatly fit any of the more common standardised questionnaires. Although I will borrrow some items from standardised questionnaires, I need to customise most of the questionnaire content, withouth having the leisure to rigorously test its validity and reliability in any thorough manner.
My topic is related to Human Computer Interaction and online software for community engagement in urban planning.
So the question is: Are customised questionaires defensible in terms of scientific value relative to industry-trusted standardised questionnaires?
The sort answer is no.
Questionnaire design for the purpose of validating theory needs to conform with the instrument design principles that are used. If you cannot find a set of factors to measure what you wish to evaluate in your study then:
a. You didn't search thoroughly or
b. You have identified a research gap for which you need to step back and create it your self through a scale development process. This can be for example by interviewing users first and trying to develop a scale that measures what you cannot measure already.
My suggestion is to look back in the literature to find related scales to operationalize your constructs. Especially for HCI there is a ton of related scales that you can use (e.g, TAM, UTAUT2 etc).
Hope that helps.
Dear Nikos,
Thank you for your concise answers, they are extremely useful.
I will tweak some of the items from SUS, TAM and UTAUT2, among others, to fit my particular case, and will also follow your advice for developing my own scale together with users. I won't have the resources to go thorough as thorough a process as researchers such as Davis and Venkatesh and colleagues have done, but that will help to improve questionnaire item validity and reliability.
I will also investigate how open questions can complement standardised closed-ended questionnaire items, as part of a mixed methods research approach.
For any new instrument, you basically need to demonstrate its reliability (typically through Cronbach's alpha) and validity (typically through a pattern of predicted correlations in line with construct validity).
Thank you for your input David, this confirms my initial doubts. I will work it out as best as I can.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire