Dear Firuz Juraev

Cluster validation has long been recognized as a critical issue in the success of clustering applications.

Besides, Silhouette Coefficient and Davies-Bouldin Index, Calinski-Harabasz Index technique can be utilized for cluster validation. The Calinski-Harabasz index, also known as the Variance Ratio Criterion, is the ratio of the sum of within-cluster and inter-cluster dispersion for all clusters; the higher the score, the better the results.

Best Regards;

Ehtisham

Dear Firuz Juraev

You can try different distance metrics and similarity measures (like cosine similarity, RBF similarity, and euclidean distance) or combinations of them and choose one that yields the best results with your performance measure (Cosine similarity typically works well in my experience, especially with text or other sorts of high-dimensional data like SIFT features, but it depends on the problem and data).

Which similarity measure you choose will influence the shape of the clusters, and there are conditions under which each should and should not be used. These measures include metric functions such as the Euclidean, Manhattan, Minkowski, Hamming, Pearson, Chebychev, and Spearman distances, as well as similarity measures such as the Jaccard index, cosine similarity, and the Dice coefficient.

Take a look: Article Deep Metric Learning via Lifted Structured Feature Embedding

Conference Paper Deep Metric Learning Using Triplet Network

All the best,

Samawel JABALLI

Hello Firuz Juraev

As we know that main unsupervised clustering is to help the researcher finding certain unseen patterns in the data, therefore, in addition to the indices and coefficients mentioned in our colleagues answers, we would consider the domain expert input to interpret the initial findings and work on further tuning by honoring related physics or known conditions.

Dear Ehtisham Lodhi, Samawel Jaballi, Omar Alfarisi,

Thank you so much for your answers. They are very insightful!

Firuz Juraev

We can use Inertia and Dunn Index as the evaluation of clusters.

Dear Firuz Juraev

Please take a look at the following links, these might be helpful:

Chapter Distance Metrics in Clustering and Weighted Scoring Algorithm

Best Regards,

Cecilia-Irene Loeza-Mejía

Dear Firuz Juraev ,

These links may be helpful:

https://www.google.com/url?sa=t&source=web&rct=j&url=https://arxiv.org/pdf/1905.05667&ved=2ahUKEwjB_puZ89P1AhXD_7sIHYtKCZ8QFnoECCMQAQ&usg=AOvVaw2FdUGZQ3k8fWH_BKuiOTR9

https://link.springer.com/book/10.1007/978-3-030-37826-4

Kind regards,

Ibrahim