I have a landuse classification and want to do an external validation of it (which is not based on the training data).
Because 1 of my 3 classes is really rare, I cannot safely implement a random sample for the validation.
How do I pick a number for the sample of a stratified validation? Does a stratified validation need to be proportional (between the strata), or can I use the same sample size for all stratas?
As my rare class is also my most important class, I do not want to under-represent it, but I am also afraid to create some bias without being conscious of it.
And I cannot find literature that gives a clear hint or maybe I just got confused about that topic. Is there any fixed that should be used? Or is there a proportion between all classified pixels and the amount of pixels that have to be validated per class.
I am thankful for all hints or literature advice!
See Cochran, W. G., Sampling techniques third ed and the following links:
https://www.google.com/search?q=sampling+for+rare+events&ie=utf-8&oe=utf-8&client=firefox-b-1
https://www.google.com/search?q=stratified+sampling&ie=utf-8&oe=utf-8&client=firefox-b-1
The 2 most common allocation methods are proportional to size and optimal or Neyman allocation for stratified samples. Best wishes, D. Booth
Thanks a lot David Eugene Booth! I will check out your google-search suggestions immediately.
It is better if samples are drawn from the strata following Probabiity proportional to size sampling technique.
I am not familiar with land use classification. However, I came across a case with a small-scale data set in my research Conference Paper Machine Learning Classification of User Interests Across Lan...
In the Introduction, I mention other works from different areas of science that also address this issue.
In case of my research (which is a bit far from what you do), I used Monte-Carlo cross validation when you reshuffle all the samples (there were quite few - 30 in every class) and put them into the classifier. Then, I calculated the average result of n classification tries. I did the reshuffling until the average result expressed as a float number (for example, let us take the accuracy measure of 0.92) stopped varying at the third position after the dot. I.e. after several tries I got numbers like: 0.923, 0.925, 0.921 . Then, I rounded down the number (0.92 in the example above).
However, my paper was turned down after being reviewed for two conferences on the grounds that I did not give a sufficient explanation why I used Monte-Carlo method. So, for the third time, as reviewers suggested, I searched for more data and also split bigger samples into smaller ones enlarging the collection to 200 samples in each class. And then I was able to do the conventional (in my field) n-fold cross validation.
Getting closer to your research issue (as I understand it): 1. small scale data sets are an issue in different branches of science; 2. there are no steadfast solutions to them; 3. the academia will be most likely to ask you to either get more data or (if it is possible in your field) to reduplicate some samples in the small set; 4. whatever method you chose, you will be asked to give a good justification - but if you pick up a more conventional one although applied to a small data set, it is more likely to be approved. Also, maybe, some cross validation methods: https://en.wikipedia.org/wiki/Cross-validation_(statistics) - can inspire you to transform them to your task.
Apologies, if I misunderstood your question, as I am not familiar with your field.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size