I am currently working on a prediction-project where I am using machine learning classification techniques to do this.
I have already computed various classification metrics like accuracy, precision, recall, AUC-ROC, and F1 score. What I am struggling with is how to (objectively) interpret these metrics in terms of their quality. For instance, in frequentist statistics, there are established ranges for interpreting effect sizes (e.g., small, medium, and large effect sizes).
Is there a similar set of guidelines or conventions with a citable source for interpreting classification metrics? I am particularly interested in categories like "poor", "sufficient", "satisfactory", "good", and "excellent" or similar.
I understand that the context and the specific task are crucial for any interpretation, but I still need a citable source that provides general guidelines, especially because this involves an educational context.
Thank you in advance!
In results section, the data divided into two sets; part for training data set and the rest for testing data set; the first question is what are the rules for selecting them?...the second question: when you talk about e.g. “80% of data used for training and the remaining 20% were used for validation”, Why do not used the cross validation for the performance's evaluation in your model and Why do use a larger percentage of data on the training set compared to the validation set or test set? When the accuracy in the cross-validation process less, is reducing the features a good idea? And the third question: The metrics that you choose to evaluate your regression or multi-class classification is very important (such as: RMSE, MSE, Accuracy, etc.). In your paper for which criteria you choose the metrics influences and the comparisons and when the accuracy in the validation process less, is reducing the features a good idea?
And when your model has the lowest loss and cost function obtained, can it prove that your method is "significantly better than other technologies"? Your experiment uses loss and cost to evaluate the training process, and your subject is exchange rate evaluation. Unless the relevant indicators of your estimation prediction results are better than other methods, the effectiveness of the proposed approach method can be proved and the last point How to set the parameters of proposed model for better performance? The complexity of the proposed model and the model parameter uncertainty needed to mentioned in the text.
Certainly, interpreting machine learning classification metrics can be complex due to the absence of universally accepted thresholds for quality. Unlike in frequentist statistics, the realm of machine learning often lacks established ranges for interpreting performance metrics like accuracy, precision, recall, AUC-ROC, and F1 score, especially when it comes to categorizing them into labels such as "poor", "satisfactory", or "excellent". These labels can often be subjective and contingent on the specific problem domain. However, a key contribution to this topic is the work of Powers, D. M. (2011). In his paper titled "Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation", published in the Journal of Machine Learning Technologies, Powers examines several classification metrics, providing a systematic exploration of their properties and relationships. This paper offers guidelines for the interpretation of these metrics in terms of quality. While it doesn't strictly delineate thresholds for labels like "good" or "excellent", it provides valuable insights and can serve as a foundational citable reference. Additionally, it's worth noting that the evaluation criteria can often be domain-specific. In medical diagnostics, for instance, a very high recall might be preferred over precision to avoid false negatives. Conversely, in a spam email classification task, precision might be prioritized to avoid mistakenly classifying genuine emails as spam. In the context of educational datasets, it's even more nuanced. For instance, if you're predicting student performance, a "good" accuracy might be considerably lower than in other domains due to the myriad of unpredictable factors affecting student outcomes.
Best regards,
Samawel JABALLI
It is difficult to find papers and reviews that justify the threshold of criteria to assess models there are some rules of thumb with range values, for example, for accuracy 50 % is not good it means that your values were predicted randomly, more the percentage is close to 100% better the accuracy is however around 100% meaning that your model overfits, for RMSE MSE lower the value is better your model is because it is translated that your model doesn't have errors. Each model is different according to the included variables. You can run many models changing some parameters and comparing the values between them acknowledging each specificity of the model ensemble learning method will be more accurate than a simple model. You need to select your model according to your research question.