In a dataset, F-statistics is inversely proportional to the number of clusters. It suggests that within cluster variance gets increased with the number of clusters. What does it suggests??
Huge within group variance. In such a case you should not get a significance between groups
Ordinary significance tests, such as analysis of variance F tests, are not valid for testing differences between clusters (due to assumptions). The pseudo-F developed by Calinski and Harabasz (1974), can be used. When this statistic decreases with increasing number of clusters then "either A. the within-cluster variance is increasing or staying static (denominator) or B. the between cluster variance is decreasing (numerator)."
If A. then perhaps use a different clustering method, B. find the simplest model (i.e. fewest clusters) where the between cluster variance over total group variance is the largest and that the number of clusters makes sense
ref: http://stats.stackexchange.com/questions/79097/validity-index-pseudo-f-for-k-means-clustering
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining