Dear Erfanmanesh ,

I think, to make sense, these values must be between 0 and 1, however, this error metric is a little bit obscure. In the original paper, are the author mention the error metric name?

I think X and Y should be representing quantitative values, for instance, the number of correct samples in class i and the total os samples in class i.

It would be helpful if you would either cite or provide a copy of the original paper. BTW what is your software? Best, David Booth

Given that the output was transformed into a time series, it is considering the output as a continuous variable and that is why the percentage error you mentioned is computed (you should also check a similar metric known as MAPE - Mean Absolute Percentage Error). If this is the case then the problem is addressed as a Regression task instead of Classification.

Trying to answer you question, y refers to the actual value and x the predicted value. Considering that the output was also normalized the range 0-1 is exactly the same as 1-35 but in a different scale.

You should check the output activation function of the NN, which MUST be linear if the problem is really being treated as Regression.

Another point, even if you get satisfactory results solving this problem as Regression, if the output cannot somehow be considered ordinal, it may not be a good idea to treat it as Regression and you should consider Multiclass Classification.

Regards!

I implement the paper with random forest classifier(features are both real and categorical, target and output are categorical) and I want to compare my results with that paper, in the paper metrics of rmse, percentage error and similarity have been computed, i want to know that comparison is possible?are these metrics have meaning in random forest classification?

I have attached the paper

You can use any classifier, as long as it can output a continuous value between 0 and 1 (the y label) given some set of attributes (the feature vector x). Then, you simply need to predict a y for every x in the time series, this should give you a similar array of y values that, when plotted, would look like the one in Figure 8, page 11. Then you can apply the formulae 12, 13 and 14 (page 12) directly, as they did. They did not explain PS, which is just 100-PE.

By the way, you should not upload the PDF of a copyrighted paper, link to the source instead. Currently (July 27, 2018), SpringerLink is trying to sell this paper for USD 39.95 and you just made it freely available, I think the publisher (SpringerLink) will have a problem with it.

Dear Tom

can classifier's output be continuous value between 0 and 1?

yes i will not upload anymore

Sorry, maybe I was a bit misleading in my answer. You don't actually need continuous output, but it may muddle your comparison with the original paper a little.

The classification here is sort of strange, as others have mentioned. This is because in the paper, they took a set of categorical, and I think mutually exclusive activities (N of them) and represented them as as a set of numbers in the 0-1 range.

The first oddity is that they represented the thing as time-series making it look like they are modelling output of a continuous function. The second oddity is that the categories are encoded numerically, making them seem ordinal.

The problem is neither continuous nor ordinal. You should use a multi-class classifier (as Ricardo Almeida suggested) and predict the categories. The class label is a categorical label, not numerical.

If you want to compare directly the output of your classifier with the output of the classifier described in the paper, then you would have to convert your output as they did, into 0-1 range, and keep the order of class labels the same as theirs. If you just want to get RMSE and other measures, you don't need to worry about converting the output.

The main issue here is that your classifier would be more precise than theirs probably is. By this I mean the following. Given three categorical labels {A,B,C} converted into 0-1 space, you would have {0, 0.5, 1} as the encoding. Your classifier would always output one of those three and nothing else. Their network might output 0.55, which is not one of the labels, so that would make their error go up slightly (artificially), unless some thresholding is done. I am not sure exactly the output they get from their network, so I am not sure if they get these slightly off-class predictions or not.

You will probably run into many strange things with this problem. One that immediately jumps out at me has to do with time.

In one sense, time is an always increasing value, for example, computers usually keep track of time by counting the number of seconds that have passed since January 1, 1970. So, how would you predict anything based on what time it is when it is always different? Well, because the target is periodic (activities happen periodically in time), then you shouldn't care what time it is, you just want to predict a pattern: given something(s) have happened, what is the likelihood of something else happening. This is how most time-series problems are approached (e.g. music, EEG, stock prices). In this case you would do time-frequency decomposition, like Fourier or wavelet analysis.

Another way to look at time, is from the perspective of directional statistics (https://en.wikipedia.org/wiki/Directional_statistics). Time is circular, so if you arbitrary end a day at midnight (0:00) and then try to predict what a person is doing at a particular time of day, you will notice that most classifiers would consider the two times 23:59 and 0:01 to be vastly different from one another. Because one is end of a day and another is a start of a day, they are not very close, like 11:59 and 12:01 are. However, in reality, they are just two minutes apart in both cases. So, unless you model time as a circular variable, you might run into issues when doing classification.

dear Tom

as you mentioned, i use my classifier and after that for comparison, i should change the scale of labels to 0-1 range, after doing that, percentage error is mush smaller than the mentioned paper and similarity is much greater, but my RMSE result is worse than paper, how can we interpret this? (i use random forest classifier)

and another problem is that as you said, labels are categorical, but they have no relation with each other. lets explain with an example, think that classifier predict eating activity instead of washing dishes, eating have label 1 and washing dishes have label 13, when i want to calculate each of the formula, these labels and their difference become important while they aren't in fact, if we suppose that these labels change to 1 and 2 in order, our result change, while It should not be that way. what's your idea?