I data mining in pre-processing step we have to standardized and transformed the data to be in a form of consistence and integrated for extracting patterns and creating models with robust results.
What are the differences between: outlier, noise, extreme values? and how we can handle this type of data?
outlier = usually a value that is recognized as being clearly seperated from the rest of the values (either visually or by some statistic); it can also be a value that is highly unlikely under a given distribution model. There is also a practical definition: values that are impossible or highly implausible in the experimental context are labelled as outliers.*
noise = variation in the data that we can not (or don't want to) explain.
extreme values = highest and/or lowest values.**
---
* they are usually produced by errors in measurements, recordings, or data transfer. Example: a patient's age recorded as 1941 (years). Such an age is highly impplausible for a human being, it's likely that here the year of birth was recorded instead. Outliers should be rare. If they are not rare, your assay/experiment/data recording is bad, or you are likely to have a faulty assumption about the distributional model.
** in a univariate case, an outlier is neccesarily an extreme value (the inverse conclusion is not neccesarily true: an extreme value does not need to be considered outlying). A multivariate outlier does not need to be extreme in any variate, and --again: a multivariate extreme value does not need to be an outlier.
Greetings,
you may wish to evaluate your data with a new approach, irrespective of it contains outlier, noise or extreme values; or none of them
Article TO DETERMINE SKEWNESS, MEAN AND DEVIATION WITH A NEW APPROAC...
outlier = usually a value that is recognized as being clearly seperated from the rest of the values (either visually or by some statistic); it can also be a value that is highly unlikely under a given distribution model. There is also a practical definition: values that are impossible or highly implausible in the experimental context are labelled as outliers.*
noise = variation in the data that we can not (or don't want to) explain.
extreme values = highest and/or lowest values.**
---
* they are usually produced by errors in measurements, recordings, or data transfer. Example: a patient's age recorded as 1941 (years). Such an age is highly impplausible for a human being, it's likely that here the year of birth was recorded instead. Outliers should be rare. If they are not rare, your assay/experiment/data recording is bad, or you are likely to have a faulty assumption about the distributional model.
** in a univariate case, an outlier is neccesarily an extreme value (the inverse conclusion is not neccesarily true: an extreme value does not need to be considered outlying). A multivariate outlier does not need to be extreme in any variate, and --again: a multivariate extreme value does not need to be an outlier.
@Ette, I strongly disagree here. Extreme values are the minimum and the maximum, in statistics the smallest and the largest value in a sample or the limits of the domain of a distribution (if such limits exist; for instance, the extremes of a beta distribution are 0 and 1). This is entirely unrelated to outliers. A sample always has a smallest and a highest value, but it does not need to have any outlier. Read the paragraph from Wikipedia carefully:
"If the sample has outliers, they necessarily include the sample maximum or sample minimum, or both, depending on whether they are extremely high or low. However, the sample maximum and minimum need not be outliers, if they are not unusually far from other observations."
https://en.wikipedia.org/wiki/Sample_maximum_and_minimum
In 6 sigma boxplot we may see in a graphical manner the minimum, quartile 1, quartile 2 (also called median), quartile 3 and maximum. Outliers are marked with asterisks in case the deviation to the mean is significantly high compared with the rest of the population. Extreme values are the minimum and the maximum of the population once outliers are removed.
We encourage you use Minitab to illustrate this. this is one of the quality tools we use in GE.
Jochen explained the differences well. Let me just add a few clarifications:
1. Noise is that part of an observation that we regard as non-repeating, which is why it is often called random. That does not mean it is completely arbitrary. It tends to fall into one of typical patterns of randomness, which we call random distributions (Binomial, Gamma, Poisson etc). Often this pattern be anticipated upfront and Normal distribution is rather the exception than the rule. Keep that in mind, when using boxplots for outlier detection, everyone!
2. Hence, outliers can be defined as observations that clearly fall outside the anticipated pattern. To give a less trivial example: reaction times in moderately complex cognitive tasks, such as the Stroop task are typically below 1 second. If one observation is 3 seconds, it can be assumed that some other process than the one under interest, happened, e.g. an external interuption.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining