Not sure if I understood correctly but if you are playing with clustering techniques, the standard method for validating your results is by implementing different validity measures such as the Dunn's Index, Xi-Beni index, Davies-Bouldin index, partition coefficient, etc. The key is to try different indices and come up with the one which identifies the clusters you expect to see or the one which makes most sense for your problem. Does that help answer your question at all?

Let me add a little bit more information.

I am clustering software bugs. I perform the clustering on all of the known bugs from version 1.0 of the software. I get a set of clusters and their descriptive features.

Then I perform clustering on version 2.0. If the clusters are capturing meaningful clusters, I would expect a similar set of clusters, with a similar distribution of bugs.

That is why I ask about consistency. Is the clustering of the data consistent across two different datasets from a similar source. I have not seen any papers that tackle this problem, but I suspect they exist, perhaps under different circumstances.

Well, the ideal clustering technique would yield similar results for similar data sets (i.e. from similar source) which means your clustering technique may not be the right one. Which ones did you implement? Agglomerative? K-means? Fuzzy K means? Also, what is the dimensionality of your feature set?

My results look fairly good, the clusters seem to match. My challenge is in how to compare these and perform a hypothesis test that demonstrates that they are similar.

I am using the repeated bisection method in CLUTO with the I1 and I2 criterion functions. The use of I2 is similar to k-means. The use of I1 works in a similar way, but it maximizes the pairwise similartiy of each point in the cluster, rather than maximizing the similarity with the centroid.

I have ~110 features (it varies by 2 or 3 between the versions of the software). The first dataset is 2800 instances. Subsequent versions are around 5000-7000 instances.

Here was my best guess at a way to show that the clusters were consistent.

I tried running the data for three versions of the software. Then I combined all of the instances into one large dataset and performed clustering again. I calculated expected results by adding up the instances in each of the three versions for each cluster based on the descriptive features in that cluster. There was one cluster in each of the versions that did not show up in one of the others, so I increased k by one for the combined data set. Then I did a chi-square comparison of the combined results with the expected results by adding up the others.

The chi-square result was good, but I am unsure whether that proves anything.

People have implemented Chi-sq before for comparison (i randomly googled it and found a couple of papers) but you can also try agglomerative and see the trends in the clusters instead of increasing k by 1. Then you can see at which stage the results match up. Hope that helps!