I am building a statistical distribution model for my data. And I am working with several data sets, each has different data range.
In order to compare the results of my model I needed to unify the range of my data sets, so I thought about normalizing them. A technique called feature scaling is used, and it is performed as such;
(xi -min(X))/( max(X) - min(X)) , where xi is the data sample and X is the data set.
However, by checking the mean and standard deviation of the normalized data, it is found to be almost zero. Does that affect the statistical distribution of my original data?
.
this will happen when you have very large rare events (i would not call them "outliers" before having looked at them in detail !) : max(X) may be very large but related to very rare events, most of the xi are much smaller and the resulting distribution will be concentrated at zero
in such cases, you may adapt your normalization strategy
- log transformation may help
- Z transform may help
depending on the shape of your distribution ... and your ultimate analysis goal !
another rough trick is to replace min(X) and max(X) by appropriate quantiles (say 1% and 99%), thus having the same kind of simple normalization and avoiding the effects of the rarest and largest events
.
oh ... and this kind of normalization is just a translation followed by a multiplicative scaling, so it does not really change your distribution (as would be the case with log or Z transforms) : a translated and scaled gaussian is still a gaussian, but with a different mean and variance ...
.
.
this will happen when you have very large rare events (i would not call them "outliers" before having looked at them in detail !) : max(X) may be very large but related to very rare events, most of the xi are much smaller and the resulting distribution will be concentrated at zero
in such cases, you may adapt your normalization strategy
- log transformation may help
- Z transform may help
depending on the shape of your distribution ... and your ultimate analysis goal !
another rough trick is to replace min(X) and max(X) by appropriate quantiles (say 1% and 99%), thus having the same kind of simple normalization and avoiding the effects of the rarest and largest events
.
oh ... and this kind of normalization is just a translation followed by a multiplicative scaling, so it does not really change your distribution (as would be the case with log or Z transforms) : a translated and scaled gaussian is still a gaussian, but with a different mean and variance ...
.
This is a very nice solution Fabrice, i.e. to replace min(X) and max(X) with appropriate quantiles. Looking it over, I found that some use the 5th quantile and 95th respectively. Do these two values hold specific meaning. And is there a way to determine which quantiles gives the best data representation without loss of information?
Thank you.
.
first, this kind of transformation does not loose information
there will be information loss if you "clip" the normalized values inside the [0, 1] interval (that is normalized values less than 0 or larger than 1 will be normalized to 0 or 1) and in that case, obviously, if you use 5% and 95% quantiles, 10% of your values will be recoded out of [0, 1] and clipping will loose information for 10% of your values
5% or 95% have nothing magical about them, just "somewhat extreme" quantiles ; most of the time i use 1% and 99% ... more a matter of taste than a matter of deep analysis !
.
and if you do not use clipping, you loose no information, you only translate and scale your data so that most of them are recoded in [0, 1]
more extreme quantiles will recode more values in [0, 1] but will come at the expense of having more and more concentrated normalized distributions
.
Dear Hiba
Most important you have to mention that this results when the log was used or other transformation
I think it may be better to divide every value by the mean of the set, so you would obtain a normalized mean which is equal to 1. This will not alter statistical moments such as the variance. I have done this in the following papers:
2016, “Improved Shape Parameter Estimation in K Clutter with Neural Networks and Deep Learning”. International Journal of Interactive Multimedia and Artificial Intelligence, Vol. 3, No. 7, pp. 3-13.
2015, “A Neural Network Approach to Weibull Distributed Sea Clutter Parameter’s Estimation”. Revista Iberoamericana de Inteligencia Artificial, Vol. 18, No. 56, pp. 3-13.
2015, “Estimation of the Relation between Weibull Distributed Sea Clutter and the CA-CFAR Scale Factor”. Journal of Tropical Engineering, Vol. 25, No. 2, pp. 19-28.
Good Luck!
- Badges
- Science topic