Imagine a standard machine-learning scenario:
You are confronted with a large multivariate dataset and you have a pretty blurry understanding of it. What you need to do is to make predictions about some variable based on what you have. As usual, you clean the data, look at descriptive statistics, run some models, cross-validate them etc., but after several attempts, going back and forth and trying multiple models nothing seems to work and your results are miserable. You can spend hours, days, or weeks on such a problem...
The question is: when to stop? How do you know that your data actually is hopeless and all the fancy models wouldn't do you any more good than predicting the average outcome for all cases or some other trivial solution?
Of course, this is a forecastability issue, but as far as I know, it is hard to assess forecastability for multivariate data before trying something on it. Or am I wrong?
When I'm faced with a new machine learning problem, my first goal is to visualize it. That means, try to get a 2D image, reduce/combine dimensions, generate a trajectory in phase space, apply Fourier transformation, make a spectrogram. If then a pattern is visible, like a chance to find clusters or something recognizable by a human, then there is hope for machine learning techniques.
Regards,
Joachim
When I'm faced with a new machine learning problem, my first goal is to visualize it. That means, try to get a 2D image, reduce/combine dimensions, generate a trajectory in phase space, apply Fourier transformation, make a spectrogram. If then a pattern is visible, like a chance to find clusters or something recognizable by a human, then there is hope for machine learning techniques.
Regards,
Joachim
Your question illustrates a very important point that is often missing behind the buzzwords of machine learning, data science, artificial intelligence, etc.
Collected data (big or small) always represent the past. You are trying to use the past to forecast the future. Is it always possible? When we collect the values of some variables (data) without actively planned experimentation just because we can collect them, then the data contain little useful signal and a lot of noise and correlations. However, only the stable signal that can be extended into the future can provide meaningful forecast using validated models. Noise is a random pattern that is unstable (disappears) in the longer run, but it is mistaken for a stable signal. The noise can also be produced by an overfitted data model. However, it is difficult in general to tell the signal from the noise. It requires a lot of subject matter knowledge. Data must be placed in a business context. Without it, there is no sure way to differentiate the useful signal from the useless noise. A forecast made with of-the-shelf or open software but based mostly on the noisy data will not be reliable, or forecasting (say, classification) will be inundated by false positives or false negatives.
The bottom line: some data systems are inherently unpredictable in the long run. The reliability of forecasting depends on quality of data within a business context, i.e. whether data contain more stable signal than unstable noise pattern, and whether the underlying model used for forecasting is valid for this particular process. To summarize: the key to forecastability is understanding of the relationship of the variables to the forecasted outcomes, i.e. causality (like developing of a physical theory) rather than a current tendency of applying just statistical algorithms to an empirical data collection.
I think that Joachim Pimiskern 's point and Alexander Kolker 's are excellent points. In addition, you might be the best in the field of ML but, you better have knowledge in the field that you are applying ML to (or have a fellow researcher that does). You might have the best ML tools and the best processes but, there might be evidence of chaotic behavior (or noisy pattern or nontrivial mixing of processes as Alexander pointed out) already established by that scientific community.
Joachim's point is excellent in that he first tries understanding what the problem is through visualization. I would extend it with additional background of the particular field you are working on. If background information is not enough to determine plausibility I would fall back to trying to establish a behavioral pattern, correlational analysis or other statistical tests before engaging in an ML effort.
You do not. There is always a chance that you missed a hypothesis or more of them. The second point is to understand the data better. It is accomplished by visualization of them and through analyzing of any correlations and trends.
There is one trick that I used in the prediction of heart arrhythmias from ECG recordings, see a prediction of heart arrhythmias paper. It is complexity analysis of data.
The original data were understood as a complex system, which we are unable to decipher. An entropy measure was applied to those data -- ECGs.
Not so surprisingly, entropy curves resulting from ECGs showed a strong predictive power whereas statistics failed, see the paper. ML methods were applied to entropy curves with a great successs.
It seems to be than not only visualization of data but as well its correct preprocessing according to their true nature can bring a surprisingly quick insights.
Take away: When you find the model of data or at least you know its class -- in our example it is a complex system -- you can use it to your advantage and distill more info from seemingly noisy data!
References:
https://www.researchgate.net/publication/340460356_Application_of_Machine_Learning_and_Complexity_in_Medicine_Prediction_of_Drug_Induced_Arrhythmias_in_Rabbit_Model_up_to_One_Hour_Before_Their_Onset_early_version_requested_due_to_COVID-19_unfinished_m
- Badges
- Science topic