A large number of machine-learning models have been built to predict stock prices in literature. What are the main reasons why no one has achieved success so far?
Zhenyu,
1) Machine-learning methods HAVE been successfully used by various individuals and institutional 'in-house' groups, but most 'public' individuals, such as yourself, will NOT learn of 'THE' SPECIFIC methodologies that have yielded 'lucrative' returns and results. When 'huge' money is involved, and this IS the case when 'dealing with' the financial markets, NO ONE is going to publicly 'share' their 'edge' derived from applying THEIR successful methods to trading....hence, you're not likely to hear of, nor see, detailed studies and reports of such successes.
2) MOST 'academic' researchers who publish papers attempting to apply computer-processing algorithms to trading markets simply do NOT truly UNDERSTAND the underlying 'dynamics' of market price behaviors, so 'naive' applications of methodologies are attempted and 'researched', with the result that 'less than stellar' outcomes are generated frequently. To be 'effective' in developing 'successful' trading methods requires a rather 'deep' understanding of 'general underlying dynamic behaviors' of what makes the markets 'tick'. In particular,....
3) Markets (stocks, futures, forex, options, etc) generate data that form (statistically) NON-STATIONARY, time-series of numbers over ANY period of 'time window' that one may want to examine, 'forecast' upon, and trade. 'Prediction' (which is highly 'precise') is essentially impossible, but to a greater or lesser degree, 'forecastability' (less 'precise', but more 'probabilistic') IS applicable to market time-series data, with the exception of what are called 'event shocks', such as USA's 9/11, October of 1987, 'flash crashes', and similar types of 'events'. (From a 'risk-management' standpoint, any 'good' and 'effective' trading strategy/system MUST make provision for such occurrences in order to protect trading capital and prevent financial 'disaster'!)
4) From an engineering (and computer science) perspective, a 'trading system' can be 'thought of' as a 'combined' mathematical/logical TRANSFORM that uses 'appropriately-conditioned' time-series 'market' data as input and then attempts to 'functionally' convert this input into a monotonically-increasing 'capital-capture' output time-series. Before attempting to EFFECTIVELY design such a 'transform', one MUST have a relatively 'decent' understanding of the characteristics AND 'character' of the time-series 'input' data to which the 'transform' is to be applied.....MOST researchers don't have an adequate, NOR realistic, market-dynamics UNDERSTANDING....hence, their market MODELS are 'inadequate' and THIS is another reason why you rarely see public information of 'successful' machine-learning methods as applied to trading the markets.
5) Lastly here, but not 'finally', 'patterns' DO frequently recur in market-oriented time-series data that CAN be 'exploited' when designing a 'transform' such as mentioned in the previous paragraph. 'Pattern', in this context, does not necessarily mean a 'visual stock-chart formation' only....it can be comprised of various 'features' that are often-times 'embedded' in the time-series data. These 'patterns' and 'features' can certainly, and EFFECTIVELY, be discerned by means of machine-learning methods.
So, to summarize, in the spirit of 'hand-waving' guidance, the real 'trick' to applying machine-learning methods, or ANY other methods for that matter, to successfully 'extract capital' from the financial markets is to become well-versed in HOW time-series data (typically 'price' series, but not exclusively) is 'created' by the dynamics of how market-participants behave and act upon the market trading-vehicles of interest. The usual 'admonition' applies: "Apply and Use the proper
'tools' to 'solve' the problem/challenge presented".....THIS requires a PROPER understanding of the various 'elements' of the 'problem at hand'. AND, if YOU want to employ a machine-learning approach to SUCCESSFULLY make money using the
financial markets, or just research this as an 'interesting' academic pursuit, you're probably going to have to do the research yourself, OR do it in collaboration with other(s) who have a similar purpose. I hope my thoughts here may be of some assistance in giving some 'direction' for your quest........
Have fun and ENJOY!! (;->)
Buzz
The set of arguments used by those models is just a subset of the superset. Find the superset, create the superset model, profit :) The problem is that some arguments of equation are not obvious and sometimes difficult to formalize.
In "The Black Swan", Taleb says that some of his colleagues even used some statistical physics models to make stock prices predictions (and he has doubts about the evaluations of such methods...). Anyway, personally I think that you can aim at modeling that, and as long as there is no "hidden intervention" (from somebody that can actually influence the market and you can hardly account for him), you might have pretty decent results.
Plot yourself today's returns and tomorrow's returns and if you are somebody acquainted with statistics, you'll get your answer.
the topic is TOTALLY different from the content. what is the definition of *success* here?
Elsa, I fail to see why.
success = non-random returns
independence prevents this from happening.
I do agree with Shchadenko. I mean so many researches have been carried out. Guess at some point in time, there have been so many diverse factors catered in for the predictions. We surely need to understand the statistics involved. And to which level of acceptance of predictability we are talking about ... needs to be defined
yes but you will have to factor in very many things and unfortunately you can either narrow down the prediction to one single product over a relatively long time with desirable accuracy, or you can predict the prices of a lot of products based on common attributes (quantified differently depending on how much they affect the end result and if they are discreet, symbolic valued attributes or otherwise). this has proven rather less accurate and becomes rather unreliable...
The answer is “yes” if the stochastic process is continuous on the “left”!
Hmm.....but before anyone says "yes", please think about the question "are stock prices predictable"?
In machine learning what we often do is learn influences to the past performance to predict the future. But the influences of the past is very small. Stock prices are often more oriented to the future performance of a company and its sector. Also I think the influence factors on performance do change a lot and gradually over time. What may have been a stumbling block back in the early 2000s is now something that everybody has access to, that is abundant and that is not an influence factor anymore.
Yes to some extent.
But quality of outcome is depended on various factors, if you can accommodate all those factors into learning system, the result should improve
The best results for stock prices prediction are obtained using RNN trained with EKF.
No, because the stock prices are basically noise. The best invesment strategy is the Random Walk. Any Learning Machine can obtain good results only in the training data. If some information exists in the price serie, one or several arbitragists are using such information and the resulting price serie is only noise.
Of course, You can predict stock market price. You can apply supervise ML techniques for training with some historical data to make a mature prediction model. Further, this matured model can be used for predictions. You can refer some of the related literature ...
You need to apply some sophisticated ML technique for the training that actually should agree with the data. Identification of relevant parameters for stock price prediction is the main challenge for the model to be successful.
Yes but try to use the liner regresion ,is it useful in this type of applaction..
My opinion is „Yes”, the theory of economical equilibrium give a prediction of the prices. The predictions depend on the possibilities for value functions construction (evaluation) used in the equilibrium model. Then the question becomes, can we evaluate such value and/or utility functions that can describe the real and the future moments based on the empirical (stochastic) information. The prediction will be of stochastic nature.
In essence, many research efforts from machine learning community hold the philosopy as the data say itself. To this regard, I believe we could model the underlying dynamics of stock price based on the data in the past. And for sure, it makes great sense on "predicting the future" in terms of decision support rather than decision-making.
Depends on what you are trying to achieve. You will not achieve any results by just throwing data at machine learning algorithms. The best suggestion is that you first try to model an investment strategy. For example, some stocks are cyclical in nature. You have to take this inherent property of the stocks that you are trying to predict into account. To do this you must first determine the granularity of the cycle that you are trying to predict. If you select microsecond interval it will be of no avail to predict seasonal shifts.
Also remember that as markets become informed of new techniques, the market will adjust to reflect such advantage. Learning from markets with machine learning techniques could be considered to fall under adversarial learning.
Hope this helps
Can machine learning do------Yes
Can current machine learning approaches do,,,,,,,,,,,,,no
Problem------- Many factors control it like budget proposal, reveal of interest rates, Take over of a company.....
Possible solution..... NLP+ adaptively filtered large data + deep learning
Current possible best scenario....similar to Bogdan Oancea
said... dont even try to poke offline predictors or supervised learning
Dear Zhenyu, I suggest you to also ask this question:
Have anyone ever succeeded to use machine learning techniques to predict any stock price?
I think the main question is that machine-learning models returns are self destructive, that is, if many 'investors' implement those models and try to take advantage of them they turn useless.
“Keynes’s description of uncertainty matches technically what mathematical statisticians call a nonergodic stochastic system. In a nonergodic system, one can never expect whatever data set exists today to provide a reliable guide to future outcomes. In such a world, markets cannot be efficient” (Davidson 2002: 187).
“Keynes … rejected this view that past information from economic time-series realizations provides reliable, useful data which permit stochastic predictions of the economic future. In a world where observations are drawn from a non-ergodic stochastic environment, past data cannot provide any reliable information about future probability distributions. Agents in a non-ergodic environment ‘know’ they cannot reliably know future outcomes. In an economy operating in a non-ergodic environment, therefore – our economic world – liquidity matters, money is never neutral, and neither Say’s Law nor Walras’s Law is relevant. In such a world, Keynes’s revolutionary logical analysis is relevant” (Davidson 2006: 150).
I would put the answer to the question more in terms of:
1) Machine learning cannot be used to predict the stock market as it has well been established in the sources of the previous posts. The number of decision making is way too big to analyse to get a meaningful pattern and all one will get is noise
2) Machine learning could be meaningful in obtaining patterns from certain stocks. This will most likely be cyclicals. The use of machine learning in this kind of stock will act as a kind of low pass filter and give you the advantage of not having to determine the range of the filter.
3) Machine learning could in principle help you in a strategy system such as a rule based system and help you put in place a stop loss.
The important message is that machine learning algorithms could have use, but they are not crystal balls. If you take a signal processing class you know that all signals have noise and imposes limits on the information carrying capacity of the signal, but this does not make signal processing worthless. If signal processing techniques are used judiciously they can be very useful in processing signal, so will the use of machine learning if used correctly.
Roberto,
I would tend to agree that the general question is no. However, I think that progress can be made towards specific sub-problems of certain economic data and get some general results on those areas.
For example, we now are making some progress in machine learning in accepting that some datasets have training data that does not follow the same distribution as the test data. The earliest reference of the analysis of algorithms that may tackle this problem (covariate shift) is from 2000 by Hidetoshi Shimodaira.
Another problem is the general use of toy problems in machine learning(yes they are still being used) instead of real world data. while the results will be discouraging at first because you cannot generate good results in your publications it is the first step in accepting that the field of machine learning needs to improve. As long as we do not accept that we need better algorithms in ML and that we need to initially accept that we are doing pretty bad we will not improve results in fields such as stock analysis.
Again, I do tend to think as you that in general the answer is no, but still we can make progress towards better understanding of stochastic systems (and as you say have knowledge of the domain, which is something we researchers try to abstract to no avail) and the use of ML to tackle some restricted domains.
Nassim Taleb has written about the stochastic models that are used to estimate the risk in the stock market. He argued that these models systematically under estimate the risk related to black swan events.
" - The disproportionate role of high-profile, hard-to-predict, and rare events that are beyond the realm of normal expectations in history, science, finance, and technology
- The non-computability of the probability of the consequential rare events using scientific methods (owing to the very nature of small probabilities)
- The psychological biases that make people individually and collectively blind to uncertainty and unaware of the massive role of the rare event in historical affairs"
Roberto,
I think that your comment on better requirements of what success means goes to the heart of the matter. This is one thing that we must look at very carefully in ML. We must also look seriously at the ML community to set higher standards not just on what is success but how it is measured. Not just for this particular problem but in general.
While we may have different views I think we agree that better standards are needed in this area( I think that both our points are very valid and both require taking care of). I also think that looking at the thread, there is consensus on the difficulty of the problem and that it cannot be tackled by our current techniques and knowledge. Nonetheless I choose to remain cautiously optimistic that we can at least make some progress on this area
it also depends what markets your dealing in. Some markets are more calm and predictable than others and make it easier to create models capable of "semi-accurately" predicting the way the stock prices will go.
In general, ANN is an interpolator, so the extrapolation is not reliable. On the other hand we could design a feedback or recurrent network using time-series data for short term future prediction of stock prices. To get a better results i recommend to consider some exogenous parameters into model too.
This question is an interesting one indeed and I spent nearly 3 years of my life trying to answer it (working for an equity market neutral hedge fund).
In an "ideal world" (no transaction costs, a very deep market where your trades don't influence the price,...) you can indeed achieve a forecasting success rate e.g. for daily stockprice movements of around 52-53% - this is as good as a really good proprietary trader from an investment bank without insider information!
However, if you want to implement these strategies in real-life trades you need to "get in" (i.e. buying/selling) and to "get out" (i.e. selling/buying back). And like in quantum physics where you as a subject interact with the microscopic object, your market impact (i.e. the impact of you on the actual share price by buying or selling this particular stock) cannot be neglected. And usually your market impact destroys the "ideal world forecast".
However, for a certain limited number of stocks (worldwide!) one can do both: one can find for these stocks robust forecasts which survive ones own market impact provided one does not buy or sell too much in those given stocks.
So where is the problem? Unfortunately, one cannot run a billion USD fund with these strategies - the "predictable & tradable universe" is too small. And after all, earning fees of approx. 1.5% of the assets under management and getting 20% outperformance fees over Libor + 5%, say, is a nice thing but not if your fund size is small and all quants expect a high salary + bonus.
Last remark: there might be life at the end of the tunnel as there exists this famous Renaissance Technologies hedge fund. Its founder is the famous mathematics professor James Simons. This guy is not only an outstandingly excellent mathematician but his fund is rather large and doing very well. So I would put the question as follows: "What is the secret of Renaissance Technologies' Medallion Fund?".
It is even a tough question if human experts can predict stock prices.
I think it's an interesting topic. As far as I know, machine learning can be applied to language process and other activites of human (Of course there're so many things I don't know since I'm a novice). Prediction of stock prices can be useful. Some factors, such as psychology, politics and so forth, can be added to the determining factor when predict the prices. Everything can be showed as a number, and every number can be calculated and used to obtain information. My Opinion.
THE MACHINE LEARNING TECHNIQUES FOR PREDICTION, DOES NOT ABLE TO PREDECT THE PSYCHOLOGICAL FACTORS OF HUMEN , ON THE PRICES OF THE STOCKS and others. the human behaviour can not be measured by any machine as it is unpredictable. the market is run on the Greed and Fear of the investor. further determined by the Price and sentiments. this why no machine model can predict the correct price.
However in the normal days of trading such behavior can be discounted but however it will be $ 100 million question.
There are certain AI techniques that have been used successfully, but as the out of sample period increases, the forecasting accuracy decreases.
In my view the AI techniques will get success in this domain by adding more input variables which affects the market performance. If you look at the theoretical results by various research papers, few/very few input variables have been used & may fail in the actual market results. But I can say, AI techniques(hybrid) will suit for stock market prediction.
If you use the Edgeworth box,then the set of the Walrasian equilibriums and the appropriate prices p = (p1, p2) (two goods) calculated based on the demand utility functions, the contract curve and the initial endowment could be a meaningful prognosis of the market equilibrium. The demand utility functions could be taken as analytical description of the consumer's preferences (convex).
why not? but requires a huge volume of input to study the time line behaviour of the stock. ANN can be used,
I'll argue that the journey of predicting stock prices accurately begins with an all-inclusive feature space definition.
By an all inclusive feature space definition, I mean having a list of attributes that measure a large number of the factors that come into play in determining stock prices, especially those attributes that are psychological or political in nature. But how do we define or quantify these? I'll leave that open for discussion.
If we can achieve this, then having ML models to solve this task may just be a stone throw away.
Yes you can. I am doing my PhD on that at the moment. Many things you need to look at before you start your research. The model selection itself, the current ranking and situation of the stock market, the type and quality of data you're going to use, the inside procedures you're going to use while you employ the test ( e.g. model learning step, model testing step and validation step) Also if you need to evaluate the accuracy of the model try to compare it with three groups, one with classic models, other one with advanced models and the last one with hyprid models. Don't forget the term DATA MINING. Go and search for comprehensive book you can read. Try to have the basic before you start and remember that this field of research is very interesting and you're definitely be able to find at least one solution for the current debates and for the tricky volatility game in the international stock markets.
Another idea: if I run a trading house and I think other trading houses uses algorithms to pick up stocks, then I must try to guess what the algorithms are , parameters , etc... but the other trading house can think exactly the same. So unpredictability increases. If you have read science fiction book Foundation (Isaac Asimov), he express this better than I can, regarding psychohistory (science for predicting social evolution) : " in order to allow the science for statistically prediction of behavior of society, nobody but the forecasters can know the science, because if people know the science, they can react in a different way, so number of variables explodes and you can not predict anymore"
yes, there are many machine learning techniques that are being in use for stock price prediction. e.g ANN, Genetic algorithms etc. here is a brief survey of different techniques. hope that it will help you.
www.ijera.com/papers/Vol3_issue6/EN36855867.pdf
Yes there are many Algorthms/Techniques of Neural Networks/Machine Learning for prediction but the Problem is the Accurate Sample Data for Training/Learning and also require the Training/Learning after a Suitable time period again with the latest sample data, that's why the required success is not achieved so for...
Maybe because the trading market is in itself so random that it is just useless to try to catch patterns and make predictions.
You might be interested in reading this: http://www.technologyreview.com/view/512696/computer-simulations-reveal-benefits-of-random-investment-strategies-over-traditional/
Which talks about the paper: Biondo, A. E., Pluchino, A., Rapisarda, A., & Helbing, D. (2013). Are random trading strategies more successful than technical ones?. PloS one, 8(7), e68344.
yes , ML is being used to forecast the stock markets.
The reasons for not achieving near 100 percent success because of
1. the data used for training,[ it needs to more accurate]
2. Stochastic models are required.
3. More new methods needs to be discovered.
ML can be used. As stated earlier, the challenge is mostly the availability of representative data for training the model.
hi
In the literature there are plenty of machine learning algorithms are available. But simply you can not take any algorithm and blindly apply to any share market problems.
The MOST IMPORTANT question is that "Which model gives you the highest accuracy for a particular data set "?
You need to perform some pre-processing of the data to choose a ML algorithms.
One should learn that ML algorithms are data dependent. So the quality of data that you use for training is the major issue. Again that's not enough.
Apart from that Optimization is another issue that one has to address.
There are some other issues too, which I can not write for want of space. Look at the literature.
I hope this answer will convince you to some extent.
Thank you
Best,
Indrajit Mandal
I think it is helpful to look more closely at what machine learning models do ie. the math.
For example: in stock market analysis, you assume that the (typically) 2-D graph of stock prices is a function of some variables of interest. You might then decide to use neural networks to model that unknown function. Here's that catch: neural networks can only model continuous functions. But your stock market data is _discrete_ (the lines connecting the dots are artistic), there is nothing that says the data follows a _continuous_ function. And if the stock market follows a discontinuous function, then neural networks will never give you a perfect answer. They may however, give you an answer that works for well but only for an indeterminable length of time ... but that is unsuitable for long term trading.
There multiple typical patterns of change that can be learned and recognized and so a certain amount of prediction can be done. But there are major events which are not forecast ahead of time by a typical pattern and those will not be predicted.
there is a large number of publications, ML and regression methods used attempting to do such a task, but most of them are either over-fitted or do not take into consideration the unlimited number of factors that affects the stock market, most of these are unpredictable, i think in ideal world those techniques may work, but using them in such a rapid changing world is almost impossible, as some researcher consider such a problem as a "random walk" this means the change in the stock market value for a specific firm is random, it is difficult to model random behaviors.
for example what happened to stock market after 9/11? could any one predict the 9/11?
most of those methods work on regression made on the history of the stock value, however this history is one factor in addition to unlimited factors which cannot be modeled, those include and not limited to:
1- hype about a company and new product
2- world events, oil prices,
3- political events
4- economical events
5- national events
6- Exchange rate
7- inflation
8- interest rate
9- internal development
10- external development of a competitive product
11- insiders,
12- internal events, such as the death of the owner, the risgnation of an important CEO
13- etc.
each of the previous major factors includes unlimited number of examples, how can we model all these alternatives if we can't actually count them?
Oh, forgot an important factor:
the buyers and sellers of stocks who normally play/trade with the dreams of the firms owners and workers, in addition to dirty games that are played every day to sell more/less stock to affect the prices....
Find the most informative predictor; then also a crappy method will work. If you don't know which is this predictor, get everything possible and apply multilevel data mining; unsupervised is best.
Zhenyu,
1) Machine-learning methods HAVE been successfully used by various individuals and institutional 'in-house' groups, but most 'public' individuals, such as yourself, will NOT learn of 'THE' SPECIFIC methodologies that have yielded 'lucrative' returns and results. When 'huge' money is involved, and this IS the case when 'dealing with' the financial markets, NO ONE is going to publicly 'share' their 'edge' derived from applying THEIR successful methods to trading....hence, you're not likely to hear of, nor see, detailed studies and reports of such successes.
2) MOST 'academic' researchers who publish papers attempting to apply computer-processing algorithms to trading markets simply do NOT truly UNDERSTAND the underlying 'dynamics' of market price behaviors, so 'naive' applications of methodologies are attempted and 'researched', with the result that 'less than stellar' outcomes are generated frequently. To be 'effective' in developing 'successful' trading methods requires a rather 'deep' understanding of 'general underlying dynamic behaviors' of what makes the markets 'tick'. In particular,....
3) Markets (stocks, futures, forex, options, etc) generate data that form (statistically) NON-STATIONARY, time-series of numbers over ANY period of 'time window' that one may want to examine, 'forecast' upon, and trade. 'Prediction' (which is highly 'precise') is essentially impossible, but to a greater or lesser degree, 'forecastability' (less 'precise', but more 'probabilistic') IS applicable to market time-series data, with the exception of what are called 'event shocks', such as USA's 9/11, October of 1987, 'flash crashes', and similar types of 'events'. (From a 'risk-management' standpoint, any 'good' and 'effective' trading strategy/system MUST make provision for such occurrences in order to protect trading capital and prevent financial 'disaster'!)
4) From an engineering (and computer science) perspective, a 'trading system' can be 'thought of' as a 'combined' mathematical/logical TRANSFORM that uses 'appropriately-conditioned' time-series 'market' data as input and then attempts to 'functionally' convert this input into a monotonically-increasing 'capital-capture' output time-series. Before attempting to EFFECTIVELY design such a 'transform', one MUST have a relatively 'decent' understanding of the characteristics AND 'character' of the time-series 'input' data to which the 'transform' is to be applied.....MOST researchers don't have an adequate, NOR realistic, market-dynamics UNDERSTANDING....hence, their market MODELS are 'inadequate' and THIS is another reason why you rarely see public information of 'successful' machine-learning methods as applied to trading the markets.
5) Lastly here, but not 'finally', 'patterns' DO frequently recur in market-oriented time-series data that CAN be 'exploited' when designing a 'transform' such as mentioned in the previous paragraph. 'Pattern', in this context, does not necessarily mean a 'visual stock-chart formation' only....it can be comprised of various 'features' that are often-times 'embedded' in the time-series data. These 'patterns' and 'features' can certainly, and EFFECTIVELY, be discerned by means of machine-learning methods.
So, to summarize, in the spirit of 'hand-waving' guidance, the real 'trick' to applying machine-learning methods, or ANY other methods for that matter, to successfully 'extract capital' from the financial markets is to become well-versed in HOW time-series data (typically 'price' series, but not exclusively) is 'created' by the dynamics of how market-participants behave and act upon the market trading-vehicles of interest. The usual 'admonition' applies: "Apply and Use the proper
'tools' to 'solve' the problem/challenge presented".....THIS requires a PROPER understanding of the various 'elements' of the 'problem at hand'. AND, if YOU want to employ a machine-learning approach to SUCCESSFULLY make money using the
financial markets, or just research this as an 'interesting' academic pursuit, you're probably going to have to do the research yourself, OR do it in collaboration with other(s) who have a similar purpose. I hope my thoughts here may be of some assistance in giving some 'direction' for your quest........
Have fun and ENJOY!! (;->)
Buzz
Dear Buzz,
All points you made sounds negative or pessimistic. The same points can be used for forecasting any time-series data. Yet we should not give up research and stay cool. There are good examples and works that prove otherwise to what you highlighted. Anyway, I enjoyed reading your comments and motivated to do something about it.
Good luck,
M.O.
I think ANN can be used. But there are so man I input variables and history cases to learn.
Mahmoud,
In all due respect to your comment that the 'tone' of my post is primarily 'negative' or 'pessimistic', I agree that aspects of the content ARE negative AND pessimistic...IF...one expects to learn DETAILS of machine-learning (ML) implimentations of, let's say, 'highly successful' outcomes. These implimentations are 'held close to the vest' by the developers of such endeavors, and again, DETAILS are not publicly shared....THAT is a REALITY!
This does NOT mean that any of us who are interested in applying ML methods should NOT pursue research that involves using financial markets as a 'target' application for ML techniques....to the contrary, in my opinion and observation, ML
methods are VERY APPLICABLE to gleaning useful insights to characteristics and features of financial time-series data that can ultimately be used to develop practical, usable, and effective trading 'tools' and systems. All I was attempting to convey in my previous post is: ... that the better the understanding of how markets and the UNDERLYING behavioral dynamics work, the MORE EFFECTIVE the results of the research can be, AND that (conditionally) ML methods are viable 'tools' to use. Ultimately, very successful application of ML methods to markets CAN be achieved (i.e., NET profitable gains can be made), and (IMPORTANTLY!) by using astute and prudent 'money management' methods, 'unbelievable' portfolio growth rates can be (and HAVE BEEN) achieved in the 'real world' in a consistent and ongoing manner. NOT by many, but by a 'few' diligent and astute individual traders and hedge-funders. This is not a 'negative' nor 'pessimistic' comment....it is simply a factual observation, and an 'inspirational challenge' to which we may aspire.
And, yes, there HAS BEEN some valuable research by various investigators that has provided useful 'guidance' as to what may be 'fruitful' and what may not work very well. As is 'typical' in this kind of endeavor, every 'study' provides a 'component' piece upon which an entire system may be built. I think you'd agree that some research efforts are better than others, just as some ideas are better than others when applied to solve a particular problem. Again, this is neither negative nor pessimistic...it is simply the way the human mind, and the skill and ingenuity of the researcher, all work.
I appreciated your thoughtful feedback....Thank you! (;->)
Cordially,
Buzz
Buzz, I asked similar questions few weeks ago.
Maybe you would like see the comments there.
How to make ANN based stock market predictions?
https://www.researchgate.net/post/How_to_make_ANN_based_stock_market_predictions?_tpcectx=profile_questions
Mahmoud,
I read the comments in your 'ANN-based' thread mentioned above. As a whole, I would agree with most of those commentaries and suggestions. ANN's are trained to ASSOCIATE input-vector 'patterns' with desired-output decision-making identifications (whether these be 'discrete' entities or 'continuous' functions). ANN's can be quite effective in detecting 'patterns' within financial time-series to yield trading-decision 'triggers'. The challenge is to determine, and employ, the 'appropriate' input-vector measurements derived from the available market data. If the 'measures' effectively relate to 'causal' effects that result in price movements, then these movements are 'forecastable' by a decently-designed ANN architecture. Determining 'appropriate' measures is NOT a trivial exercise....YET, this is the KEY to 'successful' forecasting (for financial markets in particular) using ANN's!
I do not have MatLab, so I cannot offer any suggestions as to its use for your purpose.
Cordially,
Buzz
I'd like to add just one more commentary to this thread. Paraphrased, the original question was: Can stock prices be 'PREDICTED' using machine-learning models?
Brief answer: By NON-INSIDERS (i.e., the 'public'), GENERALLY, and consistently, No! ('INSIDERS' are the 'big elephants' who are able to deliberately manipulate markets to accomplish THEIR purposes, so THEY already 'know' to what price targets their chosen market vehicles will move!) However,.....IF the question were to be 'recast' as: Can stock prices be generally and probabilistically 'FORECAST' using machine-learning models?, then the answer would be: YES, indeed, they can....
More-detailed answer: Using the word 'prediction' is rather problematical here. 'Prediction' implies 'precision and high accuracy'. An example: IF the current stock price is, say, $24.57, then a machine-learning (ML) algorithm would be able to specify: that in exactly 10 minutes, the price will be, say $25.31. And, this kind of 'routine' precision and accuracy would have to be consistently 'correct' and repeatable over significant periods of time. For the many reasons stated by myself and others within this thread of posts, such consistency is NOT 'routinely' possible, no matter WHAT kind of method or technique is employed, ML or otherwise.
However, if one asks the question: Can stock prices be consistently 'forecast' with some degree of accuracy and confidence using ML models?, then the answer is DEFINITELY, YES! Stock prices are 'probabilistic' in character and show statistical recurring 'patterns' of time-sequential behavior. Price-movement 'featuristic patterns' can be discerned that are repetitive, and recur with (generally) measurable probabilities. ML models can be designed to 'data mine' for recurring price-behavior 'patterns' and determine appropriate statistics of 'occurrence probability' for 'movement-extent' direction and 'arrival-price' zone-width within N time units. In other words, given current 'contextual' price (and other associated-data) conditions, a ML model can 'projEct' (i.e., 'forecast') a 'target' price, P, to 'fall within' a learned 'price-range zone' ( P1 --> P2 ), within N time units hence, with a probability of 'prob(P)', and associated 'confidence' of C%. THIS is very doable with ML techniques.
So, in conclusion, using ML models with financial-markets data to consistently and routinely:
'Predict (with 'fine' precision)' --> No!
'Forecast (to and within 'fuzzy' range' with Probability and Confidence level) --> Yes!
Respectfully to all,
Buzz
Dear Buzz,
Within the context of stock market, the words such as "prediction", "estimation", "forecasting" may be regarded as synonymous (although what you said is ok). We meat a good universal approximator or extrapolating (NOT interpolation) function.
They can as long as we are clever with filter of data and the methods. Deep Learning neural network approaches are particularly effective and long as we keep overfitting at bay.
I agree in a lot of points with B. Ross.
Modelling is an art, and it is not easy.
In the case of stock market, the prediction should include the corresponding uncertainty assessment. Also the question is ambiguous, since you di not say which piece of information you have at disposal.
I also agree on the fact that good methods are not published. Why? You do not need it. You just apply it.
Best
This is a wonderful and interesting question. There are published research materials everywhere using various methods yet, investors keep loosing money everyday. I think the researchers have to pay more attention to the intrigues involved in market dynamics. It is highly possible that most research papers do not take important country specific factors into consideration
Few machine learning techniques can truly predict the market direction up to some extent. But, some research paper mentioned accuracy 80/90 percent that creates doubts in my mind.
The thing is that increase in stock price in some cases means decrease in many other cases. Machine learning models can predict some near-optimal cases where the loss as well as gains are not too high else it would'e just behaved as a greedy approach.
- Badges
- Science topic