What do you consider an outlier? If data values are impossible or obviously incorrect, they should be removed. But if data don't fit your model, it is your model that should be changed, not the data.
It is not usual for it to rain 15 cm in a day where I live, but it does happen. If that data point where excluded, it would give an incorrect impression of the total rainfall and the distribution of rainfall.
For some data sets, it can be difficult to find appropriate models, but that doesn't justify discarding data just because it doesn't fit a model you're familiar with. Sometimes simplifying the analysis to be able to use a nonparametric test is a good solution.
To answer your question, how much outliers affect statistical analysis depends upon the analysis. Some methods are quite robust to outliers and some are quite sensitive. Consider using the mean or the median as a measure of location. The mean is sensitive to outlying values, but the median is not.
Outliers are not necessarily a bad thing. These are just observations that are not following the same pattern than the other ones. But it can be the case that an outlier is very interesting for Science. For example, if in a biological experiment, a rat is not dead whereas all others are, then it would be very interesting to understand why. This could lead to new scientific discoveries. So, it is important to detect outliers.
This being said, if the aim of the analysis is to explain the overall pattern in some population, then removing the outliers and do the analysis again without them is a good idea, since they can alter the results and interpretation. For example, one outlier could lead to reject the normality hypothesis.
This has been the subject of dozens of chapters and hundreds of papers. Undoubtedly, it is not possible to explain it in several paragraphs. But if you are interested in exploring the problem in the field of geochemical data processing, I strongly recommend you to read the precious paper by Dr. Peter Filzmoser et al. entitled: "Outlier Detection with Application to Geochemistry".
Article Multivariate outlier detection in exploration geochemistry
Outlier Affect on variance, and standard deviation of a data distribution.
In a data distribution, with extreme outliers, the distribution is skewed in the direction of the outliers which makes it difficult to analyze the data.