I would be very cautious with chart-based, or time series based stock market prediction.

You implicitly assume a Markov property, where a value within a time series at time, say S(i+1) is somehow of function of the present value S(i) and the past values S(i-1), S(i-2), S(i-3), ... .

So, you are trying to model (learn) the mapping from [set of {S(i-tau) }] -> S(i+1)

The stock market is NOT following the Markow property!

The value of today S(i) is not necessarily a function of the past (S(i-1), S(i-2), ...)

but the reflection of the assumption of the brokers how the value of tomorrow will be!!

As an accademic playground it is fine.

But be carefull in investing any real money.

There was a big hype in the beginning of the nineties, using all sort of ANN for predicting stock market behavior. A lot of money was lost.

A better approach is to use a set of econometric indicators (inflation, oil price development, Gross domestic product (GDP), ...) and try to predict just the change of the value, say predict the sign of the first derivative

(some companies, including financial institutions did this during the nineties).

See for example:

PASE 1991, (2nd) International Workshop on:

Problem Solving from Nature - Applications in Statistics and Economics

Zurich, December 1991

(Proceedings can be easily found by google,

but the URLI have is toooo long to include here)

Dear Prashant,

If you could tell us what type of neural network you are using, then we might be able to help you to choose an appropriate preprocessing technique. What results were you expecting when you say it "doesn't seem to be working"?

Regards,

Nicholas Lee

I am using Jordan type Recurrent Neural network and i am using Back-propagation through time(BPTT) as my learning algorithm with five unfolds.

I want to predict next day closing price of a stock.

According to me input parameter trade volume have values from 10000 to 1000000 so while normalizing with simple min-max in between values become very small like 0.006,0.0005,0.1.......so importance of this input parameter might not remains same with respect to other parameters.

I think you are saying that linear (min-max) scaling is causing a loss of sensitivity. I think you may be able to solve this as simply as taking the logarithm of the input data values before you apply them to the neural network, and then applying an anti-logarithm to the output values.

actually i tried log + min-max i mean i first took log of values and applied min-max normalization to it and its working. Though not getting good results but getting upto 60% accuracy but it can be improved with some changes.

Thank for your help.

Can anyone provide an article describing why taking the log is a good idea as discussed here? Many thanks.

The reason that log-transform is a good idea is to activate the central limit theorem. The noise can be best modeled by multiplication noise instead of additive noise. In this sense the stock prices follow a log-normal distribution. As a result the Stock prices at time i denoted by Si can be predicted from the closing price as time i-1 denoted by S(i-1) such that Si=S(i+1)*noise. By taking the log-transform the multiplicative noise is transformed to an additive noise source such that the Central Limit Theorem gets activated. The Log(Si) is Gaussian distributed or Si is log-normally distributed. This is the whole basis of the Black-and-Scholes stochastic differential equation and financial calculus. A very accessible article starting from these simple principles of the CLT in financial prices is in "LOGNORMAL MODEL FOR STOCK PRICES" by Michael Sharpe found on the web at: http://math.ucsd.edu/~msharpe/stockgrowth.pdf

Have you compared it to previous work results to see if someone got a higher accuracy. Also, you should not rule out the possibility that temporal resolution and BPTT might be the problem. I would suggest a time delay neural network or NARX as an alternative if all else fails.

I would be very cautious with chart-based, or time series based stock market prediction.

You implicitly assume a Markov property, where a value within a time series at time, say S(i+1) is somehow of function of the present value S(i) and the past values S(i-1), S(i-2), S(i-3), ... .

So, you are trying to model (learn) the mapping from [set of {S(i-tau) }] -> S(i+1)

The stock market is NOT following the Markow property!

The value of today S(i) is not necessarily a function of the past (S(i-1), S(i-2), ...)

but the reflection of the assumption of the brokers how the value of tomorrow will be!!

As an accademic playground it is fine.

But be carefull in investing any real money.

There was a big hype in the beginning of the nineties, using all sort of ANN for predicting stock market behavior. A lot of money was lost.

A better approach is to use a set of econometric indicators (inflation, oil price development, Gross domestic product (GDP), ...) and try to predict just the change of the value, say predict the sign of the first derivative

(some companies, including financial institutions did this during the nineties).

See for example:

PASE 1991, (2nd) International Workshop on:

Problem Solving from Nature - Applications in Statistics and Economics

Zurich, December 1991

(Proceedings can be easily found by google,

but the URLI have is toooo long to include here)

I hope you can verify with your transfer function selection to get the appropriate network results. Probably the normalization you have done is between -1 to +1 and if you use your transfer function in the network with 0 to 1 then, you will lead to output with large error values.

actually i tried log + min-max i mean i first took log of values and applied min-max normalization to it and its workingProbably the normalization you have done is between -1 to +1 and if you use your transfer function in the network with 0 to 1 then, you will lead to output with large error values.I would be very cautious with chart-based, or time series based stock market prediction.

In fact data you used is time series data. Hence employing recurrent NN is appropriate. I propose you to convert all series into return series. For eg. if price index at time t is P(t), then the return series at time t is r(t)=P(t)-P(t-1). Use the return series for training NN. Normality assumption is not a requirement for NN modelling. Use transfer function and other NN parameters appropriately.

You can see more details in papers published by me.

I have a fundemental question. Is it legitimate to normalize the data (min-max or otherwise) during stock price prediction? Time series and especially financial data are best examples of evolving systems. We dont know what the price of a stock is going to be tomorrow. Consider, you trained your network on data in the range [0 200] and tomorrow the stock price is 425, how will any network behave? Normalizing data each time a new input comes, is also out of question, due to the above reason.

Ofcourse, some methods can be formulated to overcome these problems, such as using "past change in price to predict future change in price", rather than predicting the price itself. But it may not work well in event detection or sudden change in values.

Hence, I am of opinion that, the network we choose should be able to handle the unnormalized data.

To Kartick Subramanian:

normalizing the time series data is fine, if you normalize your data in a

> reasonable range

The idea given above is good approach in training in an offline fashion. But financial prediction (time seried prediction in general), problems should be considered as a typical candidate for life long learning scenario. Here, the data comes in continuously and the prediction system should predict and learn together. When ever there is a change in data dynamics, the system should adapt itself to fit to the dynamics.

In the above said scenario, data normalization (including any of the above said methods) cannot be implemented due to the lack of a "fixed" min and max. As I said in my earlier comment, the range of data cannot be predicted. As for the expert, even they cannot forsee with "certainty" what holds for tomorrow. For benchmark comparison of algorithms, the above said methods will work. But with any considerations for real-world applicatioins, i am skeptical about the working with normalized data.

@Kartick Subramanian "past change in price to predict future change in price"