I am applying random forest to a time series with both fast and slow changing processes. I will also try time series analysis based on state space models later, but for now I am curious of potential causes to the intermittent results that we obtain.
We study the effect on an optimization algorithm that according to knowledge-based models yields up to 7% savings in fuel consumption. Since this is data from ocean-going vessels, the control over the experiments is limited.
Essentially, we apply a low-pass filter to the time series data and then consider each measurement to be independent (that is, that the auto-correlative effect is negligible). I use the R caret package and has swept over different configurations of non-overlapping roll mean windows (low pass filter), different number of classification trees, different number of partitionings of data. The influencing variable where we want to check data is if an optimization of a control algorithm gives fuel savings or not.
The problem is that there appears to be random if the prediction model has an influence from the optimization or not. My hypothesis is that this depends on the randomness in combination with that the optimization has a small effect (my guess is that it is close to the error of the measurement). So, when sweeping over the configurations, I get models with and without influence from the optimization.
Any clues?
Try to see whether there is any possibility of path analysis. It may give you some clues in the data.
Hi Jonas. Have you looked at the underlying functional relationship of the data you are trying to model (i.e. plot the data for bivariatea analysis ? Most decision trees partition feature space in a orthogonal way ... that means that if you are looking for correlations decision trees per definition are not well equipped to model this and get arbitrary and often poor results (ensembles of decision trees might arbitrariy capture these decision boundaries of correlations, but not in a parsimonious form). Note also that decision trees are very rarely used in time series prediction with and without explanatory variables - try neural nets if you are keen on machine learning or good old regression and compare to a single decision tree to check general suitability. Hope this helps, Sven
@Paul: orthogonal partitioning of feature space is the conventional way to describe how decision boundaries of decision trees (in classifcation or regression tasks) are iteratively generated - this is unfortunately standard in machine learning jargon. (is this a pure stats question?). So I was indeed trying to bridge a gap here ... and I am sorry you did not understand my response as a result.
Ok, thx.
Note that in machine learning the term "Decision Tree" is used for a family of machine learning algorithms of ID3, CART, C4, CHAID (see also Quinlan, J. R., (1986). Induction of Decision Trees. Machine Learning 1: 81-106, Kluwer Academic Publishers) which are the algortihms used to constitute random forrests (see above).
@Paul, yes we treat each ship individually. No, we have not tried to standardize it (neither ipsatively nor normatively), it is hard to get data for varying conditions (weather, position, speed, plans, time of year etc. etc.), so we have a subset of environmental conditions. For this subset, we do have a lot of data. For any kind of standardization, we need more data. I, personally, believe that particle filtering with Monte Carlo simulation may be an interesting approach, e.g., (Osgood & Liu, 2014); in a similar way, we have processes where the distribution are non-Gaussian and the relationsships are not necessarily linear.
Concerning my problem, the variations w.r.t. optimization is so small compared to other so even though random forest is good at handling small variations, sometimes the random partitioning end up with training data containing no variation. This leads to prediction models that have no influence from the optimization. I ran different configuration of random forests (e.g., different moving average window periods, different number of classification trees, different seeds for partioning the data) and found no particular configuration that worked better than another w.r.t. obtaining a prediction model that have influence based on the optimization. Instead, in about 2/3 of all possible configurations we found influences and in >95% of predictions based on the subset of conditions experienced during the test, the prediction models indicated that fuel was saved. In the rest, we found no influence what so ever.
Given time and resources (depends on the company), we may investigate this further. Thanks for your answers and the discussion, it gave me a lot of hints for further investigation.
Osgood, ND & Liu, J 2014, ‘Towards Closed Loop Modeling: Evaluating the Prospects for Creating Recurrently Regrounded Aggregate Simulation Models Using Particle Filtering’, Proceedings of the 2014 Winter Simulation Conference, Savannah, Georgia, US.
- Badges
- Science topic
- Similar topics
- Engineering
- Control Theory
- Control Systems
- Control Systems Engineering