I have observed that the performance of a machine learning model trained on a larger dataset is not as convincing as the one trained on a smaller (different) dataset. What quantitative measures I can use to describe the variance across these datasets to interpret the difference in the decision making ability of the model? Do we have a generalized measure to describe these variances across datasets to choose the best set for Model training?
That's an interesting question. Can you provide more information about the datasets: Are they labeled or unlabeled? How many labeles? Are they sparse (is there a lot of missing data)? What kind of data do they contain? Etc..
Also, if you have access to the test data, calculating something like KL-divergence between the underlying distribution of the test set with those of the training sets could give you some insight on what's happening. Theoretically, the more similar (KL-divergence closer to 0) the training and test sets are, the better the results should be.
I would be happy if you could mention the quantitative measures to obtain at the image (pixel) level data and not at the feature level.
I had to do something similar while comparing quality of synthetic vs real imagery. You can use KDE plot to estimate the pdf of each data set/image samples, and calculate KL divergence between the two estimations. Do remember KL divergence is not best measure as it is not commutative or normalized between a standard range.
Hope this helps.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training