The validity of a questionnaire can be checked using a panel of experts who examine its face validity and content validity, ascertaining the questionnaire items are within the proposed theoretical construct. A common way of ensuring reliability is through piloting the questionnaire and conducting Cronbach's alpha to test the internal consistency reliability. Furthermore, you might need to conduct EFA and CFA if you designed, not adopted, the questionnaire. You could see Bolarinwa (2015) for the basics of questionnaire validity and reliability and Ball (2019) for tips on conducting an online survey.

Ball, H. L. (2019). Conducting online surveys. Journal of Human Lactation, 35(3), 413–417. https://doi.org/10.1177/0890334419848734

Bolarinwa, O. (2015). Principles and methods of validity and reliability testing of questionnaires used in social and health science researches. Nigerian Postgraduate Medical Journal, 22(4), 195. https://doi.org/10.4103/1117-1936.173959

Good luck,

Hello,

beyond what Mohialdeen said a few comments. These are comments that fit EVERY research process

1) I agree to make heavy use of pretesting your items--however, beyond using a panel of experts, I would (also) do cognitive pretests/probing with respondents from the target populations.

2) If you plan to use statistics (e.g., internal consistency estimates such as alpha or omega) or approaches (e.g., EFA/CFA) be sure you know the meaning of the common factor model and generated or select your set of measures based on that. 90% of failures result from diverse sets of items pushed into the factor analytic framework.

3) After the qualitative piloting mentioned, to a quantitative pretest. This is to check variances or the efficiency of the whole questionnaire. I would be careful to estimate statistics (e.g., models or alpha) as these provide point estimates with a huge variance/inefficiency based on the low N. Hence you may get bad estimates although the measure is good or the reverse.

4) Regarding the use of so-called "validated scales", I would recommend testing the validity of the measurement model (IF it concurs with the factor model) in the actual samples. Measures are interpreted by respondents on the basis of the study context (the sensemaking by the individuals about the whole study) as well as the local context in which the item is placed.

Hope this helps

Holger

Daniel--I am just wondering what type of survey you plan to administer. Is it an affective instrument? I also have a couple of thoughts. First, reliability is generally considered a characteristic of the data collected with a given instrument. (Hence, we typically estimate reliability for every sample within which we administer an instrument.) And validity is a characteristic of the score uses/interpretations and inferences drawn based on those scores. So, neither reliability nor validity is really a characteristic of the test, itself. Here are a couple of handy resources that I frequently use when teaching about these concepts:

McCoach, D. B., Gable, R. K., & Madura, J. P. (2013). Instrument Development in the Affective Domain (3rd ed.). Springer. Book Instrument Development in the Affective Domain: School and C...

*AERA, APA, & NCME. (2014). Standards for Educational and Psychological Testing. American Educational Research Association.

*Note that you can get an open-access copy of this text from AERA's website: https://www.aera.net/Publications/Books/Standards-for-Educational-Psychological-Testing-2014-Edition

Best of luck with your work!

~Sarah

As the others have indicated, it is the content of the survey (i.e., the variables) that needs to be assessed for reliability and validity, not the mechanical tool such as Survey Monkey that is used to collect those variables.

Daniel Julian Otto, I think that the words reliability and validity can be used in relation to the mechanical aspects of obtaining data (e.g., SurveyMonkey or Qualtrics) as well as to the information that you seek (e.g., from scales) that is used as the data you subsequently analyse and report.

In your case, I think you might need to distinguish between those two things (mechanical aspects for obtaining information versus the actual information / data). It might also be necessary for you to define clearly what you mean by reliability and validity. Unless you do that, I suspect you'll wallow around in mud, and so will those who come in contact with what you produce.

All the best for your research.

Article Developing Valid and Reliable Online Survey Instruments Usin...

Daniel Julian Otto Please find the above link to get more information on survey instruments.

I agree with David Morgan that this issue of reliability and validity pertains to the questionnaire not the mechanical tool. It's only a concern if your questionnaire has specific guidelines for its administration that cannot be done using Survey Monkey. I use the service all the time and recently had an article published using the data from the Survey Monkey system.

Check https://www.researchgate.net/publication/334001429_Developing_Valid_and_Reliable_Online_Survey_Instruments_Using_Commercial_Software_Programs

Also check https://www.surveymonkey.com/curiosity/how-reliable-are-surveymonkey-panelists-around-the-world/

Chinaza Godswill Awuchi, thank you for providing the URL in your post immediately above here. In my view, it addresses the topic that was raised in the question at the top of this thread, namely, "I'm planning on using survey monkey as a data collection tool but I do not know how to determine its validity and reliability". That refers to the tool, not to such things as scales.

I acknowledge that the words reliability and validity are very often (almost always?) applied to scales, but I'd argue we shouldn't always confine their meanings to that context. Furthermore, I'd argue that, as much as we try out of good will to help the person who generated a question, we need to try to address the issue(s) that questioners raise, not impose our view on what we believe questioners should have raised.

There's only one concern I have with the URL you provided: It doesn't look like a conventional citation that would be acceptable in a thesis or journal article. Maybe a conventional citation isn't necessary, however - and I still appreciate your having provided the URL. Maybe other RG members can come up with something that is similar in content but more "citeable".

For what it's worth, a colleague and I have conducted research with large numbers of preservice and inservice teachers using Qualtrics. These two groups provided different patterns of responses in terms of missing data and straightlining (i.e., using the same response option for all, or many, items). My impression is that, had we used face-to-face administration of our surveys, we might have obtained different data - i.e., data that were more reliable and/or valid. If so, I think that the method by which our surveys was delivered would have influenced data quality.

It seems there are two different issues here. Are you going to use your own list of subjects or are you going to use subjects provided by Survey Monkey? If you are going to use your own list of subjects, Survey Monkey simply collects the responses.

I'll cite a Streiner's chapter book here, about computarized administration, in general:

"One potential disadvantage of computerizing scales stems from the direct transfer of existing instruments to a computerized format. In most cases, it has not been established whether or not the translation has adversely affected their reliability and validity. Paper-and-pencil questionnaires allow subjects to see how many items there are and to pace themselves accordingly; to skip around, rather than answering the questions in sequence; and to go back easily to earlier questions, in order to change them or to check for their own consistency. While scale developers may deplore these deviations from the way the instrument was intended to be taken, the results of the original reliability and validity studies were conducted with these factors possibly playing a role. Modifying these factors may affect the psychometric properties of the scale. However, the evidence to date suggests that if there are any differences between the paper-and-pencil and the computerized versions, they are very small (e.g. Merten and Ruch 1996; Pinsoneault 1996; Watson et al. 1990)."

You can check chapter 15 for more info on this topic.

Streiner, David L.; Norman, Geoffrey R.; Cairney, John. Health Measurement Scales (p. 321). OUP Oxford. Edición de Kindle.