Questionnaire methodology has long been used for data collection in many fields. It is nevery known whether the respondent are filling in the blanks with all correct and true information in his brain. It is also impossible to identify whether the results from questionnaire had suffered from subjective bias from social role, real-time thought, or results that would like to be known.
So, do you believe in results collected from questionnaire?
There are certain things that only surveys can do (e.g., examine people's internal thoughts), and times when alternative data sources (such as big data) cannot provide representative samples. Obviously, surveys can be done badly, and there are many potential sources of error, including respondent mistakes, poor questionnaire construction, problems in the sampling frame, sampling issues, data collection issues, and low response rates. That just means that surveys need to be done with great care, with awareness of their strengths and weaknesses. Note also that different types of questions present different types of issues. Not all questions are subjective or raise social desirability issues, for example.
They are reliable and valid but best if you create and field test them yourself. When you write up your assumptions, you would assume the participants are telling the truth. It is easier to tell the truth if the questions you ask are relevant to your study and your participants so creating your own questionnaire would decrease some of that bias. I created and field-tested my own questionnaire and interview questions for my study. Thank God I did. It would've been a disaster otherwise.
I agree with Laverne, it is true that there are many biases in this type of study as in most quantitative studies, so they must meet the requirements of evidence of validity that any instrument requires and have a representative sample to then conduct further studies with a reliable and acceptable survey.
All measurement instruments are subject to some degree of error, and while we can cope with random measurement error with sufficient sample size, systematic sources of error can be problematic at any sample size. As both Laverne and Christian suggest, there are many grounds for improving the design of survey instrument items to reduce or avoid known sources of systematic error.
Rather than asking if we can trust results, I prefer to investigate possible sources of error and changing the design to either avoid them or provide measurements to help estimate the size of the distortion. For instance, in some cultural contexts asking about income can lead to non-response, and drop out. Those same respondents may have answered a question like "How often do you replace your cell phone?" which, among a set of other items can be used to construct an index which is highly correlated with individual or household income.
From a methodological and statistical perspective error is just another parameter to be estimated and understood.
Howard T. Welser I strongly agreed. I would like to use an alternative approach. In my own research, I am figuring out to use some big-data methodology instead of traditional questionnnaire towards a similar scientific question. I surmise some error from the questionnaire methodology are not systematic or random ones, but technological ones that are be made by subjective or unconscious biases. Thanks for sharing the response.
Laverne A. Davis Thanks for taking part in this discussion. I would prefer to use a suitable questionnaire for the study with specific aim. Of course all studies with questionnaires should use the hypothesis that all respondents are telling the truth although we know they cannot actually. To impose a hypothesis, however, may be not the first choice to optimize this type of studies.
Cristian Ramos-Vera I like your statement of a reliable and acceptable survey. We all know maybe they may not be by the questionnaire methodology anymore, right?
Yes, even if there may be some uncertainities it can be minimize by pillot test/pretest.
There are certain things that only surveys can do (e.g., examine people's internal thoughts), and times when alternative data sources (such as big data) cannot provide representative samples. Obviously, surveys can be done badly, and there are many potential sources of error, including respondent mistakes, poor questionnaire construction, problems in the sampling frame, sampling issues, data collection issues, and low response rates. That just means that surveys need to be done with great care, with awareness of their strengths and weaknesses. Note also that different types of questions present different types of issues. Not all questions are subjective or raise social desirability issues, for example.
Yes, provided the questionnaire has been tested for validity and reliability. If not, you must do it yourself through a pilot test .
Yes, its beneficial but most times , people could not bother it just see and tick even didn't learn it, so it must be cleared that data should sent to competent authority so that will help for further study.
Generally yes as survey data represents micro level data which could be consolidated to come up with macro level analysis and trends
Teresita Bascos Deveza Yes So for small scale research where only few of respondents are requested to fill in blanks the questionnaire is not suitable.
Yup. Focus group discussion or group assessment/sharing would be more appropriate using a well structured set of questions/issues for resolution/assessment/discussion/recommendations/stepsforward.
Small samples are seldom useful for any kind of statistical analysis, because they have large standard errors and thus low power to detect significant effects. If you are limited to a small sample, you might indeed want to consider the possibility of doing in-depth qualitative interviews rather than surveys.
Maybe we should rather operate some observational tool into people's brains to check whether their answers are actually true?
jk
Probably, sometimes yes if it a small research and atimes data collected through questionaire can be unreliable because we can not determine the genuinity of the data collated!
Hi Hongxu Wei and all. I point to the resource question. Most often we have to live with the uncertainty. In some areas it is possible to use biological markers (e.g., in medical research), but most of these markers also have uncertainty attached to them (differences in sensitivity - specificity). I once tried to use a biological marker to check the authenticity of patients' report on alcohol use and I was accused in the local newspaper for trying to check whether patients were lying. I took the blow, but could not used the data anyway; the test (MCV) is sensitive to alcohol use above a (quite high) level, but it has very poor specificity and I still had to rely on self-reported data only.
Best
John-Kåre
With all due respect, a questionnaire has to be tested for validity AND reliability before being used for research.
How do you do that? There are companies that specialize in the subject, by testing and retesting the questionnaire, until the company decides a maximum of biases have been eliminated They will sell you the questionnaire at a high price.
If you don't have the money to buy it, you will go through the method of the pilot test.
There is enough literature on the pilot test methodology.
Jacob Aschkenazi I think test-retest reliability is a rather out-dated practice. Instead, most researchers now use multiple measures to minimize unreliability in the form of random measurement error.
I never felt reliable on data collected through questionnaires. The reason is that most of the time questionnaires are susceptible to subjective bias. Being a respondent myself I have felt that having a lengthy questionnaire (some time comprised of 4,5 pages or 50+ questions) creates anxious feelings in the mind which may produce false results. Furthermore, responses of the population vary which cannot be normalized by a factor meaning that results of questionnaires are populations specific and cannot be generalized.
Ajmal Khan Cann't agree more. In addition, it is acceptable that questionnaire data fail to follow the normal distribution but, look at the distribution, many weird situations can be arisen which are not natural response. This should be due to the obvious bias.
I feel there is too much pessimism here. Every method of studying human behavior has its limitations, so the use of surveys can be a good decision when chosen appropriately.
David L Morgan If one is a questionnaire supporter, the discussion may be disappointed because people here are mainly talking about the weakness and the possible promotion. As the weakness of new approach that may replace the questionnaire methodology has not been fully revealed, it is fine to figure out an alternative way that can be better than self-reported scores.
Hongxu Wei, to be fair, your original question sought information about whether researchers believed in the validity of data obtained from questionnaires. You didn't ask people to (solely) criticise questionnaires - even though, reading between the lines, that is what you were seeking. In light of your actual question, therefore, I think that David Morgan's contribution is on the ball and helpful.
I would add that, as several posters to this thread have already pointed out, certainty is difficult (impossible?) to achieve. Furthermore, I think it behoves researchers to look at their data carefully to see where invalidity might lie, to engage in responsible data cleaning, to use only the clean data in their analyses, to perhaps analyse their data in different ways and report any differences that emerge, and so on.
I think researchers should also feel obliged to take care when creating questionnaires / surveys / scales that those who respond will be maximally inclined to provide valid data. I see a lot of surveys that are so badly designed I am tempted to sabotage the research. Usually, I simply cease responding as soon as the incompetence of the researchers becomes too manifest for me to bear any longer - but, at times, I have indicated that I'm 300 years old and have 150 children living in our residence.
Researchers tend to have their own favorite approaches. I remember watching one discussion where a researcher was advised to use each of six different techniques, each offered by someone particularly adept in that particular technique. The fact is that many approaches are useful, and often they are most useful when conducted in a complementary manner. Each technique has something to offer, and each has weaknesses. The biggest issue, I often find, is not what technique is used, but whether the researcher is asking the right questions.
There is a point I guess. I always believe in using a mixed method which would help to overcome this problem. In-depth interviews with a reachable sample would help to interpret data derived from the questionnaire successfully than handling data derived from questionnaire alone.
Robert Trevethan
Thanks for anaylzing the whole board of discussion for this question. Yes, I put forward a question for open answers with a potential attitude that I, as well as some invovlers in this discussion, feel unreliable on data from questionnaire. This is a subjective question although it is not as so obvious as it is on the surface of words.
I also do screen and analyze the data before further calculate and analyze them. However, I usually do this kind of analysis on data that were collected from experiments from subjects of plants, e.g. seedlings and community forest trees. I never analyze in advance on data from human subjects from self-reported scores because I am not sure the data in my hand are reliable or not.
Lasni Buddhibhashika Jayasooriya Thanks for replying. Actually, many questionnaire data are not Gaussian followers hence in many situations the mixed model is invalid to analyze non-normal data. If I changed to use another moder with goodness of fit acceptable, I need to use some maximum likelihood model where repeated measures cannot be used. Some journal reveiwers they do not accept to use LS means in a maximum likelihood model. This is an issue.
Bradford Chaney Yes I like the statement that a methodology should be used to anwser the scientific question. If the scientific question does not require too high precision of data and can accept some fraud respondents, it is fine. However, the fact is, some researchers put forward of a scientific question and anwser it without too much consideration about the validity of questionnaire. Maybe some people can be aware that this is an issue that should be considered when seeing this debate.
to a large extent, i do... for example, in my PhD study all volunteers were very cooperative and tended to follow up for results... so their remarkable interests reflects somehow the reliable given data...
Currently, I am involved in two surveys, neither of which collects subjective data. One collects financial data from organizations, and another collects enrollment data from colleges. So let's not automatically dismiss survey data as subjective.
Having said that, subjective are often quite valuable. When working with people or with programs involving people, often it is helpful to know their motivations or inner thoughts. The data can still be very accurate. Election polling actually does quite well, though there are surprises. People do change their minds, and sometimes they are not honest. But, by and large, it works.
As noted by others in this discussion, all data are subject to error. One needs to be aware of the sources of error, seek to minimize error, and allow for it.
Paula Romanos It is lucky to recruit some nice respondents who are host and very careful to answer the questions. One of study topics is to assess people' emotional response and mental status to an experience in urban forests. In this kind of study, however, I need respondents to fill in the blanks with their realtime natural thoughts. If one person is too concentrated to think about what is the exact thought and feeling at the moment, his/her results will be over nervous and stressful. Therefore, I agree with some scientists that the scientific question is the most important thing to evaluate the reliability to employ the questionnaire methodology.
It all 'boils down' to analysing questionnaire data carefully, especially in light of questions asked.
As we all know, there are many influencees on respondents answering a questionnaire; thus, whether this research instrument is returning valid data for a particular study, is a matter of judgment.
There are scientific statistical methods in planning, conducting, and analyzing surveys that will ensure the objectivity and reliability of survey results. These methods could be learned/accessed using the internet, studying survey methods in universities, or attending trainings. Moreover, there are survey statisticians or professional survey instituions which could be tapped in undertaking surveys.
Teresita,
absolutely---regarding the more quantitative aspects of surveys; however, in my previous post, I principally had in mind the more qualitative aspects.
Philip,
scientific survey methods could also be applied to qualitative surveys. There are a lot of materials on this in the internet. Please see business expectation surveys and consumer expectation surveys undertaken quarterly by yhe Bangko Sentral ng Pilipinas as well as similar qualitative surveys being done by other centrsl banks. Thanks.
- Badges
- Science topic