I have data input for a neural network with one output, the data range between 0 and X, I do not know the exact value of X because it changes with a time, I mean may be at a time t the max value of X=1234, but at t+n , X can take values higher than 1234. I'll use it in an online application to do some predictions.
I would like to put this data (normalize it) in [-1,+1] Are there any good methods to do this?
Hi Nana
I think that this course Slide on data preparing can solve your problem (5. Time-Dependent Data):
http://cis.poly.edu/~mleung/FRE7851/f07/preparingData.pdf
is it helpful? make know.
Hi, nana,,, first assess the presence of outliers, second, analyzes the trend of the histogram accordingly analyzes the component distribution through a p-plot or q-plot,
If the distribution is logarithmic, apply a exp (), if exponentiation Ln (), if quadratic, sqrt (),,,,, etc ... No fixed form .. depends on the trend and the number of data
this formula, just transform data N(mu,var) to N(0,1),, The data transformed will are in (-3,49; 3,49) approximately, when data have Normal Distribution.
Thank you for all your responses.
Mr Guersi Noureddine, thank you for the course, it is very helpful...
For the outliers, if I suppose that data higher than a threshold as outlier, Have I to delete it from the data set? or may be I have to save its positions in my set, but only do not use it in my application? for example at time t the value of X= val, where val will be considred as outlier, so I think that I do not have to delete this value, because before all, X must have a value at time t, otherwise, my predictor will not run correctly!!
please, correct me if I'm wrong
If you have a variable depends on the weather. Why better consider a distribution based on stochastic process?
I do not see where they treat it as a regular data distribution.
Hi Nana
if you have not already solved your problem
I propose an empirical method which is applicable and not too complicated if you are familiar (you have a priori knowledge) with your application and your data sources.
Before proceeding to the real-time (or online) application, try to save your data (for an acceptable time, greater than or equal to the simulation time) in any file (inputs and outputs), seeks the data min and max .
After that you can use this min and max as max and min thresholds in your online application regardless of the new data value.
In this case you can apply the Matlab normalization mapminmax
[ y = (ymax-ymin) * (x-xmin) / (xmax-xmin) + ymin ]
where ymax = 1 and ymin = -1 for example for the interval [-1,1], x is the current value of the data, xmin and xmax will be the max and min thresholds that could have your data, and y is the normalized value.
This is not a very scientific method, but when you work with for example voltages and currents, you have usually an idea of the max and the min amplitudes (values) of these variables.
I had tried to use different kind of pre-procesing techniques with neural network and the results showed that min-max performed better than others.....but still it depends on what kind of data you are dealing with.....
Everyone talk about the data, other very important aspect is the configuration of the neural network you are using, if the net is trained bach or on-line. If is the second one, the net will automatically adjust the structure to adjust to de data characteristics. Also if the net is trined off line, it is important determine the range of the data to use, if is [0,1] o [-1,1]. In base of this you can decide what type of normalization is better.
Thanks for all these responses,
for my input data, it is explained above, in addition, I do not use the data it self, but the delta between two successive records, for example X1=231, and X2=300, so delta=-69, this what I want to normalize,
the output of my network will predict the difference between the actual and the next record, for this I've used a tangh as activation function==> output will range [-1,+1]...
If I were you, I used EMA (Exponentially Moving Average) for time dependent data. You will have no worry for change in your max, since historical data will be considered in each recent data and then use one of the regular methods of transformation between [-1,1].
- Badges
- Science topic