Dear Fahime,

Related to you query, I suggest you follow ::

https://www.youtube.com/watch?v=K8wvU9YtXQ0

https://au.mathworks.com/matlabcentral/fileexchange/24616-kmeans-clustering

https://au.mathworks.com/help/images/ref/imsegkmeans.html

It is well studdied topic, so you can get a lot of its related literature via the google.

Good Luck

May be fuzzy k-means algorithm will be helpful

https://www.mathworks.com/help/fuzzy/fcm.html

Matlab returns fuzzy partition matrix.

Vsevolod A Lyakhovetskii Thanks for your response, I did K-means but one cluster include two outliers. I mean we have 2 clusters and the first cluster includes 2 points that do not look like the others.

Fahime Sokhangou, you could calculate the mean and standard deviation using the points assigned to a cluster. Using these values you can calculate a percentile for each cluster (https://www.statology.org/calculate-percentile-from-mean-standard-deviation/). Once you have that value you can exclude points outside a certain percentile range (say 95%) and recalculate the mean and standard deviation. You could repeat the calculations above until no points are excluded from the cluster, or do it once and flag the points as outliers.

Zachariah Wynne Hi, I am wondering if you have any references for detecting the outliers based on the z-score value?

Thanks for you time

Zachariah Wynne by using percentile, based on what you explained, I lost some points in the middle of the data that were not outlier. I tried 95% and 90%.

But, I discovered the points that I would like to remove them has a different z-value from the others.

z=(X-mean)/standard deviation

Now, I am trying to find a threshold to remove that points. I would like to know how I can select the threshold logically.