In statistics in general, there are assumptions that we must make for us to continue our investigation. So the assumptions that you listed above are among them. If the assumptions hold, our conclusions are fine, generally speaking. If we are conducting a large enough sample, we are also assuming that the Law of Numbers will kick in and, in the case of social research, that respondents are at least honest in responding. These are the parameters of imperfect research.

You're not paranoid. In fact, this is one of the biggest concerns in social science. However, by being clear in your statements, asking the right questions in the right order, you can minimalise said issues. Also, statistics (as Cordell suggests) can provide clearity in these issues. For example, cronbachs alpha shows if people are randomly filling in indexes/scales or if they actually give coherent answers. So does a factor analysis.

The question I have though for you is more theoretical: does it matter? At least in social sciences we're looking for opinions, attitudes and emotions. Does it really matter that they don't know the first thing about something? Isn't it all about perception? If people perceive something to be true, then thats all that should matter for your research. People create opinions based on what they know, whether they're correct or wrong doesn't matter. If you say "their answer is bullshit cause it isn't true" then you're bias yourself and thus creating assumptions about the respondents that cloud your objectivity and by extension your study.

Cordell, Sander, thank you both for your answers. I do agree, that literature gives this general advice that the researcher is responsible for minimalizing effect I ask about. But there are some "buts" still disturbing:

- large samples generate “good” p-value but it has nothing to do with effect size, which is main thing researched, so I am not so sure that the law of numbers guarantees solution

- actually, it is well shown by mentioned Cronbach’s alpha, because this indicator is quite prone to sample size and number of variables (larger samples with bigger number of variables give higher value of Cronbach’s alpha)

- I understand humanistic coefficient rule - I ask not about subjective opinions but about opinions constructed ad hoc, without full reading of questions or/and cafeteria, just because respondents are asked to do it and do not want to make an impression that they do not know something (important, because the researchers ask about it)

- as far as I know, the respondents’ truthfulness depends on cultural factors – e.g. in traditionally protestant countries they are more … honest (?)

I don't know, I mean - there are so many questionnaires conducted every day. Some are good, some others aren't. There is this huge and vital discussion between "quantitative" and "qualitative" researchers. But I rarely meet arguments considering these basic problems of respondents' attitudes and actions. What do you think?

First off: you're completely right, p-values are becomming very fast, very useless due to very large samples because of the internet. In fact, there is a very big discussion in the scientific world about the value of the p-value. Take for example nordic countries. Sweden apparantly has nation-wide surveys with a 90%+ response. Does statistical significance mean anything? If you have 90% of the population we're no longer talking about generalizing the results from a sample to the population because our sample IS the population. So true, significance only means somehting when you're working with small samples on a population. Also, how important is your statistical significant beta coefficient of 0.12 in your regression when you compare it to a (marginally) non significant result of 0.76 beta? Take a look at the strength of the coefficient as well as the signficance and, based on your theoretical argument, conclude whether it is useful or not.

Your last two arguments are a bit fuzzy to me. What do you mean about opinions constructed ad hoc? You mean you give them a hypothetical problem and they have to figure it out on the spot? Isn't it still about perception then? If they give an ad hoc solution, then you're still measuring how well they that solution is and therefore can conclude (for example) that people are more likely to come up with innovative solutions when they're unfamiliar with a subject than those that already informed about the issue.

And protestant countries being more honest then i.e. catholic countries, well that is still something you can account for in your results. Remember well that there is a reason why most scientific areas are based on falsification: you can never ever rely solely on the truthfullness of people. You can never truely say "you're theory is correct". Still, by using latent constructs and reversing items in that latent construct, as well as statistics and common sense, you can solve many of these problems.

Concerning quantative and qualitative research, both have pros and cons. Personally i prefer quantattive due to the generalizing character. Problem with it is the social desirable answers screwing with your results. Qualatative on the other hand, can go much deeper (and according to some therefore better) into a subject and how people respond to and perceive that subject. Ofcourse the problem there is that a bad interviewer can fuck up the data more than 1 respondent filling out a survey at random in quantative research...

Starting next week I'm following an advanced course on survey design, addressing these exact problems, I'll keep you updated if my view changes during the course.

ps. there are two books I have to read for said course, perhaps some literature for you as well.

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press.

de Leeuw, E. D., & Dillman, D. A. (Eds.). (2008). International handbook of survey methodology. Taylor & Francis.

Sander, thanks again for your answer. Especially the first book seems to fit my concerns the best. I also look forward to hearing from you about your experiences from the course. I do conduct research on samples up to 6000 respondents, both PAPI and CSAQ. I also work with qualitative techniques. The question results from 10 years of experiences and it’s a trial for discussion with other researchers.

Once again, I appreciate your engagement in the discussion and await for more :).