If "outlier" means physically implausible or even impossible value, then such outliers should (must) be removed.

if "outlier" means a value just being a bit away from the rest,the action depends on whether or not you can identify some external reason that might explain the outlying value. If you have such an explanation, then the kind of explanation tells you if you should remove the outlier (example: you made measurements on cell cultures. It turned out that the outlying measurement came from a plat where cells were unwantedly infected with some fungus or bacteria - this value should then be removed). If you don't have an explanation, the outlier might be worth noting/reporting but it should not be removed.

In general, if "outliers" are nor really rare, then your experiment or data collection/processing is flawed and should be revised. Wrong assumptions about the expexted distribution can also lead to the idea one had ouliers (e.g. if the distribution is actually log-normal, gamma or beta but you wrongly assume it is normal). So finding outliers may indicate that you also should revise your assumed distribution model. If the experiment and data collection/processing are fine and the assumed distribution model is also (known to be) appropriate, outliers must be rare. Rare events are not impossible, so they do happen. Because they are rare, they don't harm - statistically, so there is no statistical reason to remove them.

Thanks Jochen Wilhelm for such a overwhelming response. It was indeed informative.

Regards,