Your problem is related to a situation that showed up in the machine learning literature 15+ years ago.  Individuals had access to a common set of raw data and developed different classifiers based on truth data.  At a certain point, they discovered that the preprocessing had (almost) as much effect on the final classification accuracy as the method.  In these situations, they had a natural overall metric in the classification accuracy.  The preprocessing affected the representative of the records (in the case of text categorization) in vector space form.  Generally, however, the used the same or very similar metrics in comparing individual records and how the records were classified or clustered.

Two things to look at that affect your understanding.  (1) Use different preprocessing with the same classifier.  Are the results similar? How similar?  (2) Use different preprocessors and different classifiers (or different clustering algorithms).  How does this affect the classification or clustering?  If you go through a few test situations, you may be able to answer you question effectively.

Thanks but your answer (very general) do not anwer to my question.

.

if you use different preprocessings, your clusterings are built in different representation spaces so you obviously cannot rely on internal criteria for a direct comparison

(normalization may do the job in some cases : if your metric is the proportion of variance captured by the intercluster variance, the comparison is direct ... but a preprocessing leaving very little information may perform much better for such a criterion than a preprocessing leaving a lot of information in the representation which makes the comparison somewhat meaningless)

indeed, with different preprocessings, the only thing your clusterings have in common are the items they cluster ; it is therefore possible to compare clusterings in terms of how consistent they are with each other (agreement between pairs)

this may end up in a clustering of the clusterings !

but will not give you a notion of "best" clustering, you just recover a notion of which preprocessings / methods lead to similar clusterings on these data

.

Dear Fabrice,

Yes that's right ! You win one point ;-)

You could therefefore be interested by this paper :

"A Stability Based Method for Discovering Structure in Clustered Data"

https://www.researchgate.net/publication/11435997_A_Stability_Based_Method_for_Discovering_Structure_in_Clustered_Data"

Article A Stability Based Method for Discovering Structure in Clustered Data

What about this Vincent ?

http://link.springer.com/chapter/10.1007%2F978-3-319-25660-3_18

You may know this paper already.

Jean-Charles could probably share his C++ implem' of his specific quality metrics. Otherwise, I may finish soon a Java implem' for sparse matrices.

Dear Nicolas

I'm going to see but in my mind JC  Lamirel uses an harmonic mean in this paper , so dependent of the preprocessing. But I have to check. Thanks ! Best Regards

As mentioned in another answer, the only thing you have in common is the items they cluster.  Thus, each method gives you a different cluster label for a given item.  I assume by cluster you are implying an unsupervised learning problem.   You can enumerate each cluster from one method (call it preprocessing method A), giving it a class label of 1, 2, 3, etc.  You can do the same for a different preprocessing method (call it B) and label the clusters.  Order the clusters for method B so that cluster 1 has the most agreement with cluster 1 in method A, method B cluster 2 best agrees with method A cluster 2, etc.  You can quantify agreement using Matthews Correlation Coefficient.  If you are doing supervised classification, then you can skip the above part about matching the group labels.  Then you use the Adjusted Rand Index (which is identical to Cohen's Kappa) to compare whether the set of labels for method A is more similar to the set of labels for method B than one would expect from chance.  This technique is often used to compare inter-rater variability in human-based labeling, but is also a useful technique for comparing labels between methods.