I am dealing with a clustering problem that requires me to cluster a set of objects based on 'Avg rate of change' in several factors for a give timeframe. Attached is the dendrogram I obtained. It appears to me that the data has several singletons and clusters are not well shaped. My question is :
1.) What should be the right treatment to data (in such situation) before I take them to clustering ? All variables are at the same scale.
2.) What could be the other fully unsupervised (unlike K-means) methods that I should analyze this data by ?
Hello Ashutosh,
Cluster analysis is structure seeking, but accomplishes that goal by imposing structure. So, it's not always possible to obtain "well shaped" clusters. Not only might they not exist in nature, but the choice of clustering algorithm, similiarity/dissimilarity measure, and several other options may conspire to invoke an altogether different result.
One suggestion I would make is to be sure to standardize your variables (set all to a scale having a common SD, such as z-scores). Otherwise, the variable that has the largest standard deviation will dominate the determination of cluster memberships (using any variant of distance as the similarity metric).
Good luck with your work!
As Dr Morse suggests Standardizing variables is always a good idea. This link:
https://www.google.com/search?q=How+to+choose+stopping+points+in+a+cluster+dendrogram&rlz=1C1CHBF_enUS847US847&oq=How+to+choose+stopping+points+in+a+cluster+dendrogram&aqs=chrome..69i57.33153j0j4&sourceid=chrome&ie=UTF-8
gives some guidance in how to choose a stopping point. This can be difficult.
Best wishes, David Booth
David Morse , David Eugene Booth Thanks for answering my question. The dendrogram has already been plotted on standardized variables but as you rightly mentioned the clusters seems to be arbitrarily shaped (?) or highly overlapping. Let me dig more into what you suggested.
Hi Ashutosh,
Did you try nonconvex clustering algorithms like spectral clustering or DBSCAN?
Sibylle Hess No, I have not tried DBSCAN yet, so thanks for suggesting. Also, could you share some idea about tuning epsilon and how to decide the minimum number of samples for this approach ? I used a grid search with few possible combinations and wondering if there is a better method to do.
Ashutosh Karna Tuning the parameters for DBSCAN is still a difficult problem. A grid search is a good first approach. Since you are looking for clusters which are not mostly comprising singletons, you might have some ideas about a suitable setting of the minPts parameter. You could then have a look at some statistics of your dataset. What is the average distance of the k-nearest neighbors of your data points (k could be some value around minpts). You could also make a plot of the minimum size of epsilon such that at least 70/80/90% of the data points have at least minpts neighbors in the epsilon-neighborhood graph. Apart from that, you might find some insights from this discussion:
https://www.researchgate.net/post/How_can_I_choose_eps_and_minPts_two_parameters_for_DBSCAN_algorithm_for_efficient_results
- Badges
- Science topic