I have to normalize data which has values for 100-1000 (numeric values). I read some material regarding normalization techniques e.g. min-max normalization, cosine normalization, non-monotonic normalization etc., but I don't know which technique is best for me. I read a paper suggesting non-monotonic normalization is better but haven't found any good material regarding it.
Supplementing Jens Peters answer, I would argue that there is no best normalization per se without knowing the analysis task. It could also be helpful to investigate the means, median, and stdv and outliers of your range of values - before AND after the application of various normalizations. For example, in many cases a uniform distribution of values is desireable. Accordingly, a normalization should be chosen that performs this task best. Have fun!
Actually i am trying for stock market prediction and want to normalize stock data because i read somewhere ANN performs better with normalized data.
Well, seeing your data which has limited range and not much variability between the minimum and maximum, and has one dimension, I think min-max would be a suitable option for normalization.
I agree with Rakesh. Since the linear min-max normalization does not distort the relative pairwise distances (because it`s linear), it seems to be appropriate for making your various stock charts comparable without producing additional errors. Linear min-max normalization: (max-X)/(max-min). Don`t forget to check if (min==max) in case of constant time series ;-) hf
.
min-max normaization may lead to awfully slow learning when the input data has a fat tail (or outliers but this is an easy check) and the "informative" part of the distribution is the head. It may therefore also induce strong differences between variables after the normalization step.
local linear rescaling on the unit interval using quantile values (map the value Q(n) of the nth percentile to n/100 and scale linearly [Q(n-1), Q(n)] to [(n-1)/100, n/100] if you use percentiles, for instance) worked fine for me most of the time : easy, alleviates the fat tail problem to a great extent, easily deployed on the test set (of course, you keep the quantiles as calculated on the training set !).
pure order statistics (which is the limit of the above procedure) also worked nicely most of the time (somewhat slower on "easy" distributions where the "holes" in the density are the informative parts, since these "holes" diseapear in order statistics).
.
The old and glorious standardization by z scores (i.e subdividing by mean and dividing by standard deviation) is a very effective way for normalising. Moreover, if you normalise on a data set A and you then have a single observation Xi not pertainng to A you can immediately verify the homogeneity (e.g. z < 3) with the reference collective.
Z- score normalization is very common but it depends on your dataset and what you want to achieve. You can refer to my paper which provide a brief analogy to normalization techniques
I suggest you to try this transformation:
For x in [a,b] define y=(2*x-a-b)/(b-a) and your data is normalised at interval [-1,1].
(As Jürgen mentioned you have to be carefull for the case a=b)
It has worked nice for my reseach work.
Z-normalisation is useful if you have information that your data is 'normal', but this is not close to the truth for your case.
Please inform me about the results of this transformation.
Hello Prashanth,
Could you point to the paper which attests non-monotonic normalization to be better?
Also, I am trying to understand how it works, could someone explain the reasoning behind it?
This is what I got
exp(–z2/2)
z = (xij – xj0) / sj
xj0 is the most favorable performance value w.r.t. attribute j
Why would using the most favorable value help in normalization, and more importantly, how would you define the most favorable value?
Hello! I am a postgraduate student and I write my dissertation this period. Could anyone please help me and suggest me an easy to understand literature which compares the normalization techniques?
Thank you in advance!
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining