If we have three different domain of data (e.g. security, AI and sport) and we did 3 different case study or experiments (1 for each domain) and we estimate the Precision, Recall and F-measure for each experiment. How we can estimate the overall Precision, Recall, F-measure for the model. Is use normal mean average is suitable or F1 or p-value? Which one is better?
It depends on what you want to know. You may use macro-averaging or micro-averaging for precision, recall or F measure. The latter is just the harmonic mean of recall and precision. Choosing the right parameter-value (beta) you are able to create a balance between precision and recall depending on your needs by giving more weight to the one or other. Macro-average computes the arithmetic mean of evaluation metrics (recall, precision, f1) calculated for each class or dataset separately. Thus it's the average performance of your classifier regardless the considered dataset or class.
Micro-averaging aggregates the contribution of each class or dataset and subsequently calculates the metric. Therefore in environments with class imbalances it might be preferred.
The p-value is completely different story. You get the p-value if you perform one type of a hypothesis test. This is not the way to combine different results, but to check the variance of your results. You can consider the combined measure as your null hypothesis and calculate with the p-value how likeli your test results are at least as extremely as your observed results. The smaller the p-value the more likeli your null hypothesis needs to be rejected. Usually, a signifiicance level is determined. If the p value comes below this significance level the null hypothesis is rejected. That means your result is created by chance.
Hopefully this helps a little bit.
Dear Fatima Nadeem Al-Aswadi ,
It depends on your goals from the performance evaluation process and whether you are dealing with a binary problem or multiclass problem. In general, I recommend you to use MCC.
Also, I think that no need to gather all the obtained results. simply, you are testing your system when being utilized in different domains. I f you want to test its performance when dealing with data from different domains, just merge the data you are using :), and test your system!!
- Badges
- Science topic
- Similar topics
- Mathematics
- Applied Mathematics