There several cluster validation indices by which you can measure the clusters quality. Among them Silhouette Index, DB and Dunn indices are widely used. In my opinion, choosing the clustering algorithm for comparison with your new proposed algorithm is totally based on the characteristics of a database. Generally, K-means, Hierarchical and DBSCAN clustering algorithms are mainly chosen in most of the papers for comparison.If the database is dense then DBSCAN may be your suitable choice. You can get some help in the following paper.
I am not sure I understood the question. Are you looking for distance-based clustering algorithms that can be used as state-of-the-art competitors for your new algorithm?