What do you think about so high accuracies reported in so many publications especially in medicine and psychiatry? Talking about overfitting and so frequent lack of internal and external validation, shouldn't we aim at some standardization prior to publication? It has a name: Unwarranted optimism in machine learning applications
Please share your thoughts about it.
@Mireille Hildebrand registration of ML trials in https://www.cohubicol.com/being-profiled-cogitas-ergo-sum is one possibility
Hello. Your question touches on a number of issues that are not necessarily of a scientific nature. I agree that in fields where the application of machine learning (read: deep learning by convnets) is relatively new, there does not seem to be much standardization. Some attempts have been made (e.g. on Kaggle). This is not the case in more established machine learning areas like computer vision where there are numerous benchmark problems and publicly available data sets.
In order for more standardisation to occur in medicine and psychiatry, it would be necessary to adopt (1) open data standards and data bases; (2) open source code repositories; (3) standardized performance measures with their corresponding confidence intervals.
There are potential problems with open medical data since this can violate patient privacy and there is often a significant cost associated with medical data acquisition, the rights of which would normally be held by the organisation(s) acquiring the data. Aspect (2) is easy to satisfy. As for (3), there is no standard for ML performance measures and considerable confusion surrounding different notation and terminology. Relatively few authors supply confidence intervals for their performance assessments and those that are supplied are sometimes based on incorrect statistical assumptions (such as approximations to the binomial distribution).
Please recommend if you find this answer helpful.
- Badges
- Science topic