Dear all,
I wish to hear from your advice in pilot testing self-administered questionnaires. I have several individually validated questionnaires (e.g. IPAQ, DASS, SF-12, Perceived Barriers to Healthy Food, Tobacco questionnaire), will be used to collect data via self-administered by respondents, in my study later. Prior to actual study, I need to pilot test these questionnaires due to different study populations. So, my questions are as following
What are the aspects required for analysis to be done for the entire package of these individually validated questionnaires besides reliability analyses?
I understand there is a need to do test-retest reliability in this pilot study Hence, the test-retest reliability should be reported as a whole for the entire package or reported individually for the each tool?
Dear Raymond,
Firstly, are you using the Malay version of all those validated questionnaires? If yes, then you can skip this paragraph and go on to the next one. If you are not using the Malay version or the one which is validated for a more local context, your task is an arduous one. You will have to validate the questionnaire (factor analysis and reliability testing) and then determine which questions/factors which can be retained for a more Malaysian society.
By performing test and retest, you are attempting to assess temporal stability. In my opinion, you might not need to do this for all types questionnaires. When you utilize Theory Of Planned Behaviour, you need to do a test and retest as the constructs require you to do so to examine if there is stability in your indirect (unarticulated thoughts) questions.
You might need to explain more regarding your validation intention - 1) Are you validating the questionnaire? 2) Do you intend to merge all these questionnaires to create a comprehensive one? 3) Do you want to conduct or test a population with these questionnaires?
Rabin
Typically researchers undertake Cronbach's alpha test for the whole and also as each tool as well (as if the is an anomaly in alpha it may be due to one tool). The pilot serves to identify early anomalies, which may/may not disappear at the full survey.
In addition there are usually a series of questions posed by the researcher to respondents concerning clarity, ease of use, understanding with respect to questionnaire in a pilot.
In addition what colleagues mentioned above, researcher usually do pilot study for many purposes that all lead to stronger design. So, finding if there is any typo, grammar, unclear text, meanings..all lead to modifying the questionnaire to become a valid one. you also can estimate the needed time to fill the questionnaire, which is important to tell the participants ahead of time so they estimate how long it will take ,,, which will be reflected on more compliance in filling, and ultimately, more valid results.
hope that helps
Dear Yogarabindranath Swarna Nantha, Robert James McClelland and Prof. Muayyad Ahmad,
Thanks for your inputs provided.
The actual study will be done to assess the lifestyle practice among a young adult population. Therefore, several questionnaires will be used to assess their lifestyle and health outcomes in different domains mainly Physical Activity, Dietary habits, Tobacco use, Mental Health, Perceived barriers.... So eligible participant need to fill up all questionnaires that measures all these outcome interests.
I neither intend to combined all questionnaires into one nor to modify any of them but to execute all these questionnaires together as a package primarily in English language. However, those who are not well-comprehend will have Malay version (validated in Malaysia) as alternative.
Since all these measures accounts individual behaviours, hence my idea is I need to pilot test these all questionnaires via test-retest for respondents consistency in providing similar results (Do correct me if I'm wrong). So, at this point, here comes my doubt where how should I analyze the so called test-retest reliability? Analyze the entire package as whole, or analyze individual questionnaire separately?
In addition, since all these questionnaires have been validated in other studies, done among young adults population but different countries, is there any additional analyses required to be done during pilot testing? As far as I concern, there are none except test-retest reliability (correct me if I wrong also). At the same time, I am wondering whether would there be any discrepancies in the response if one administered one of the questionnaires vs administered the whole package.
I agree with Robert James McClelland that you should use Cronbach's alpha instead of test-retest reliability. The basic idea in any assessment reliability is to eliminate the possibility that your results are affected random error, and alpha address this issue directly. (The idea behind test-retest is that if the measures have low reliability -- i.e., high random error -- then they will not correlate across time.)
As for validity, I would concentrate on "construct validity" (another idea that originates from the work of Cronbach). This applies to the pattern of correlations among your variables, so that each measure performs as predicted with regard to the other measures.
Finally I agree with Professor Muayyad Ahmad that you should be careful about your assumptions regarding the adequacy of your translation as well as the meaning of the items to those who have English as a second language. This typically involves pilot testing, rather than pre-testing -- where the former works closely with a small number participants to get their reactions to the questionnaire, while the latter uses a larger preliminary sample in circumstances that are as realistic as possible.
One increasingly popular form of pilot testing is called cognitive interviewing. Here is a ling to an article about that topic:
https://www.researchgate.net/publication/259631901_What_Do_Our_Respondents_Think_We%27re_Asking_Using_Cognitive_Interviewing_to_Improve_Medical_Education_Surveys
Article What Do Our Respondents Think We're Asking? Using Cognitive ...
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire