It is common practice to pre-test a questionnaire before it is used in the field by asking experts to evaluate the questions.
This really depends on several factors, like research topic, your experience in the research topic, access to experts, budget, time, etc. In my opinion, the pretest in your research population is much more important. Because there you may identify which questions are problematic under realistic conditions.
The best pre-test is to use a small sample of the proposed study population as their feedback may be more useful than that of an expert who is not part of the population. For student dissertations, it is acceptable to use the supervisors' feedback as a pre-test due to time and other constraints
I think you should distinguish between(a) Expert Review' and (b) Cognitive/Qualitative Testing of the questionnaire. Both (I believe) are valuable, but they are different. Expert Review is normally done first, and involves experts in either the topic being surveyed, or in questionnaire design generally, or both. In such cases, it is typical to use several experts (3-6, perhaps). Cognitive testing of members of the population would then follow, after modifications are made based on the experts' comments. The issue of how many to test is unsettled, and as Jean Philippe Pierre Décieux points out, depends on your resources, complexity of the instrument, etc. A simple pretesting plan may involve no more than 6-9 such interviews. A very good approach seems to be iterative testing that involves both Think-Aloud and Verbal Probing techniques -- after this first 'round' of testing, you modify the questionnaire based on what you have found, and then conduct a second round. Generally, two rounds of 9 are better than one round of 18. Researchers who really want to comprehensively evaluate a questionnaire may conduct many more interviews, however, especially when multiple language translations are involves. I have several resources on cognitive testing that I can send you, if you would like to contact me directly.
Gordon gives an excellent response! This is exactly the approach you should consider in developing and initially testing your survey instrument.
Thank you all for your valuable comments abour my concern the number of experts used to evaluate a survey questionnaire.
I strongly agree with Ariel about Gordon's input. I usually rely on a group of eight experts to conduct such an exercise.
in special field i suggest 4 to 7 person should be help you as expert mans.
According to references [1, 2] and based on Cochran formula, the sample size of questionnaires can be determined as follows:
n= ((z^2 p(1-p))/e^2 )/((1+(z^2 p(1-p))/(e^2 N)) )
where N is population size, e is margin of error, z is confidence level, and p is percentage value.
In my case study (Reliability Engineering), N can be total number experts in specific area of reliability engineering (For example, number of reliability experts who work on offshore wind turbine reliability). To have ±3% margin of error, e = 0.03. Confidence level considered as z = 0.95. p = 0.5.
[1] W. Cochran, Sampling Techniques (3rd edition), New York: John Wiley & Sons , 1977.
[2] H. B. H. B. Sarmah, "An Investigation on Effect of Bias on Determination of Sample Size on the Basis of Data Related to the Students of Schools of Guwahati," International Journal of Applied Mathematics and Statistical Sciences, vol. 2, no. 1, pp. 33-48, 2013.
You can also check the following URL:
https://help.surveymonkey.com/articles/en_US/kb/How-many-respondents-do-I-need
I found the following web page and I think it can be useful regarding the question:
https://towardsdatascience.com/sampling-techniques-a4e34111d808
Dear Gordon B Willis,
Thanks for your excellent response. I was wondering is there any review paper or reference including both methods to reflect more details/information?!
hi,
there is no facts or significant number of panel for evaluation, as it all depends on your research. some use 15 some use 4 some use 6, it depends on you how many do u want to choose, but its good if more than 3 as u can get a better result from the panel and suggestion from panel. if u put 10 panel u may have more than 10 suggestion and it give more work to do so i suggest if above 3 and not more than 10 for social science reseaarch.
- Badges
- Science topic
- Similar topics
- Statistics
- Survey