i was wondering if there is any approach that can reformulate clustering algorithm by standard optimization and maybe relax it into a standard convex optimization with generalized inequality in proper cones like SDP or SOCP.?
It depends on the number of data dimensions, the selected algortithm itself (how to calculate distance in N dimensions from data element to cluster center), the threshold on distance between clusters, the desired numbers of clusters, desired number of iterations, since clustering is iterative by nature. So, select these things and relax some thresholded thing - allow to change number of clusters, etc. The only idea in terms of convex optimization here, which I know, is https://en.wikipedia.org/wiki/Otsu%27s_method, but it works not as good, as it claimed, since it have a particular assumption about image PDF.
i am familiar with conic optimization, but the question is that how can you state the problem by a quadratic inequality ? can you bring up an instance ?
Although it does not address generalized inequality in proper cones, I believe the deterministic annealing framework proposed by K. Rose has a potential to be adapted to this problem:
Rose, Kenneth. "Deterministic annealing for clustering, compression, classification, regression, and related optimization problems." Proceedings of the IEEE 86.11 (1998): 2210-2239.