Christian - 

Data editing at the statistical agency where I worked for many years was based on data that fell outside of expected bounds, which made those data subject to scrutiny, not automatic expulsion.  In the case of regression, one can graph predicted y on the x-axis, and observed y on the y-axis.  Then it is often obvious when some data were reported in the wrong units, for example.  Responders were called to ask about suspicuous data, but I tried to emphasize that the responder should not be led to a result, but just asked to review it. 

If you do not have access to the sources of the data, it may be much harder to prove some data need to be removed/replaced.  In your case, it sounds like your data may not be salvageable.  It is not generally acceptable to drop data just because it is suspicious. 

You might want to search on the terms "data editing," and "data quality."  I did have an affiliation with a data quality journal at one time.  You might find some articles.  

In Snijkers, G., Haraldsen, G., Jones, J., and Willimack, D.K.(2013), Designing and Conducting Business Surveys,  John Wiley & Sons, Inc. there is some information on edits.  On pages 491 and 492, "statistical edits" are noted, and it is stated that these edits "...detect data values that are likely to be wrong."  Note the word "likely." 

It appears to me, Christian, that you may find the same thing.  Remember that one percent of the data are supposed to be outside of a 99% confidence interval.  You may well know what data are suspicious, but you won't know what are truly incorrect.

In some cases suspicious data may be 'counted,' but given lower weights, but how you justify that may be subject to debate. I think I saw something about that when glancing through the book noted above, but I suspect there may be many versions of such a weighting scheme.

(In weighted least squares regression, there are weights based on heteroscedasticity which are well justified, but I doubt that that applies to your case, though I think sometimes people assume an outlier when what they are actually seeing is heteroscedasticity.)

Best wishes - Jim

Hi James, thanks for your answer! I’m looking for some articles, theories or research that can be fortify the approach of leaving outliers in the sample. I do not have the change to justify the answers by asking the participants.

Hello Christian,

If you think that "participants have understood the question and answered it correctly" then you should keep their data. The reason to remove outliers is because you think that they are in some sense wrong. In your case it is perfectly plausible that some people would have much lower annual incomes (e.g. work part-time) or much higher annual incomes (e.g. millionaires) than others.

There is advice about handling outliers in books such as "Handbook of Statistical Analysis and Data Mining Applications" Chapter 4.

Also, you mentioned that you are interested in calculating the mean of the data and noticed that it is very different depending whether outliers are included or excluded. Instead/as well as calculating the mean, I would recommend calculating the median which is a more appropriate measure of average when you have skewed data. You could also report the skewness and variation of the data.

Best wishes, Christina