Many machine learning techniques such as Random Forests and Random techniques are giving different accuracy on each iterations(first time produces 90%, 2nd time 85, 3rd time 83%). Is there any standard to interpret such results like taking average of 5 (or) 10 readings? If there is any paper related to this question, please let me know
Dear Dr Muniandi,
this is a really interesting question. I would suggest Dr Breiman's paper on random forests (see link below), in particular section 2 on accuracy, section 6 on empirical data, section 8 on output noise and average over 50 repetitions, section 10 with the example of m repetitions, and section 11 with the regression example.
https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf
Yours sincerely,
Rainer
Thanks for your answer Mr. Rainer, I felt this needs further investigation to find the reliability of Random technique's real-life performance.
n-fold is one of the most techniques to calculate the average of accuracy.
There is trade-off between accuracy of your estimation and computational resources. Leave-one-out estimation is most accurate, but consumes a lot of cpu time. k-fold with small k values works fast, but gives you only rough estimation of your metrics. Also it is necessary to shuffle your data before splitting them on train and test folds. It seems to me that big variance of accuracy on your folds arises from some clustering in your data.
- Badges
- Science method
- Similar topics
- Biological Science
- Ecology