I am using the CLUTO clustering toolkit to cluster software faults from multiple versions of a software project. The goal is to find a clustering solution that provides insight into the characteristics of the software, and its defects. I would like to show that I get consistent clusters on different versions, and identify differences as they occur over time.
I have not seen any papers where this has been done. So far I am thinking of a chi-square test for homogeneity. The clusters are ranked, and the ranks seem to be fairly consistent as well. Adding ranks into the comparison may help my argument, but I'm not sure which test would be most appropriate. Any suggestions on the best way to do this? Any references to see similar comparisons in other applications of clustering?
Not sure if I understood correctly but if you are playing with clustering techniques, the standard method for validating your results is by implementing different validity measures such as the Dunn's Index, Xi-Beni index, Davies-Bouldin index, partition coefficient, etc. The key is to try different indices and come up with the one which identifies the clusters you expect to see or the one which makes most sense for your problem. Does that help answer your question at all?
Let me add a little bit more information.
I am clustering software bugs. I perform the clustering on all of the known bugs from version 1.0 of the software. I get a set of clusters and their descriptive features.
Then I perform clustering on version 2.0. If the clusters are capturing meaningful clusters, I would expect a similar set of clusters, with a similar distribution of bugs.
That is why I ask about consistency. Is the clustering of the data consistent across two different datasets from a similar source. I have not seen any papers that tackle this problem, but I suspect they exist, perhaps under different circumstances.
Well, the ideal clustering technique would yield similar results for similar data sets (i.e. from similar source) which means your clustering technique may not be the right one. Which ones did you implement? Agglomerative? K-means? Fuzzy K means? Also, what is the dimensionality of your feature set?
My results look fairly good, the clusters seem to match. My challenge is in how to compare these and perform a hypothesis test that demonstrates that they are similar.
I am using the repeated bisection method in CLUTO with the I1 and I2 criterion functions. The use of I2 is similar to k-means. The use of I1 works in a similar way, but it maximizes the pairwise similartiy of each point in the cluster, rather than maximizing the similarity with the centroid.
I have ~110 features (it varies by 2 or 3 between the versions of the software). The first dataset is 2800 instances. Subsequent versions are around 5000-7000 instances.
Here was my best guess at a way to show that the clusters were consistent.
I tried running the data for three versions of the software. Then I combined all of the instances into one large dataset and performed clustering again. I calculated expected results by adding up the instances in each of the three versions for each cluster based on the descriptive features in that cluster. There was one cluster in each of the versions that did not show up in one of the others, so I increased k by one for the combined data set. Then I did a chi-square comparison of the combined results with the expected results by adding up the others.
The chi-square result was good, but I am unsure whether that proves anything.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence