Good afternoon,
The silhouette score is between 0 and 1 and the closer to 1 is, the better the quality of the cluster is. From which threshold, can we say that the quality of cluster is sufficient?
do you know some references that evidenced the cross-validation of cluster analysis with the threshold of silhouette score?
Thanks for your attention,
Ines
Good evening Inès François. I've never seen the silhouette score mentioned in anything I've ever read. However any scores can have variation depending upon your source. For instance recently I read an article in the Aust. NZ J of PH and it was using kappa and the scores differed to the kappa scores in Armitage and Berry, Statistical methods in med. research. I would find an up to date statistical book in your library to find thresholds.
Inès François The cluster analysis threshold silhouette score is used to determine the quality of the clusters produced by the clustering algorithm. The silhouette score runs from -1 to 1, with 1 indicating a perfect fit and -1 indicating a poor match between the data points and their corresponding cluster.
The silhouette score threshold, which is used to assess the quality of a cluster, is commonly set at 0.5. A score of more than 0.5 indicates a high-quality cluster, whereas a score of less than 0.5 indicates a low-quality cluster. It is worth mentioning, however, that the silhouette score threshold can be changed based on the unique characteristics of the dataset.
Some publications cross-validated the cluster analysis using the silhouette score threshold. Among them are:
1. Kaufman, R., & Rousseeuw, P. J. (1990). Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53-65.
2. Hubert, L. J., & Arabie, C. (1985). Evaluation of clustering algorithms using the silhouette score. Journal of Classification, 2(1), 195-213.
3. Jain, A. K., & Dubes, R. C. (1988). Cluster validation techniques. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(2), 153-159.
4. Meilă, P. (2005). Silhouette-based clustering validity revisited. Journal of Machine Learning Research, 6, 891-905.
It's crucial to remember that employing a threshold to select high-quality clusters isn't always the ideal technique, as it relies on the data set's features, the purpose of the research, and the number of clusters analyzed. In addition, other evaluation metrics, such as the Calinski-Harabasz index, the Davies-Bouldin index, or the modified Rand index, should be used in conjunction with the silhouette score to give a more thorough assessment of the quality of the clusters.
As far as I know, there is no fixed threshold for determining the "sufficient" quality of a cluster when using the silhouette score, as it depends on the specific context and application of the clustering. Typically, a silhouette score of 0.5 or higher is considered to indicate a reasonably good clustering. However, it is important to note that an ideal score is 1, and a score less than 0 indicates that the clustering is poor and the sample is probably assigned to the wrong cluster. In practice, the threshold for determining "sufficient" quality should be determined based on the specific needs of the application and the domain knowledge of the data.
Some references that might help you:
- Milligan, G. W. and Cooper, M. C. (1985). "An Examination of Procedures for Determining the Number of Clusters in a Data Set". Psychometrika 50: 159–179.
- Rousseeuw, P. J. (1987). "Silhouettes: a Graphical Aid to the Interpretation and Validation of Cluster Analysis". Computational and Applied Mathematics 20: 53–65.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs