I am looking for a tool which can efficiently cluster large matrices, with less use of RAM.
If you have some experience in high level programming languages you could use Mathematica, MatLAB or Octave.
And here is k-means clustering algorithm.
http://cs229.stanford.edu/notes/cs229-notes7a.pdf
Hi Deborah,
I have 30 samples as columns with expression values for each probe as rows (80,000).
You can use clara algorithm, it is implemented in R (package cluster). It works by clustering a random sample from the dataset and then assign all elements in the dataset to these clusters. By the way I've never used it for matrix with 80.000 rows, but it works quite well for 50.000.
I agree with Toni, use K-mean clustering algorithm defined by this paper : (https://www.cs.umd.edu/~mount/Projects/KMeans/pami02.pdf), I have done a clustering for 30k random micro-architectural points of my compiler design space and use low power, high performance and high intensity as my factors to cluster them to k=4 classes. (in which they were the pareto's efficient).
I suggest you add at least another constraint to be able to make it multi-objective.
If you need to take a look at my constraints, feel free to check the paper at : https://www.researchgate.net/publication/259820777_A_Framework_for_Compiler_Level_Statistical_Analysis_over_Customized_VLIW_Architecture
cheers.
Conference Paper A Framework for Compiler Level Statistical Analysis over Cus...
Thanks for the answers...I have tried K-means but there I need to specify the clusters a priori, what I am interested is to cluster the data unsupervised and see how it bins into different sub-clusters...
@Srinivas: but if your clustering constraint (objective) has only one objective (only using less RAM), there won't be needed to pass through all these pains cuz you just sort them by use of RAM and that would be the results.
As I suggested, make it multi-objective and cluster them. Here you can improvise your clustering to say 4, which stands as for instance less RAM, less power (LL), high RAM high power (HH) and so on.
P.S: You could then filter the pareto points as well, having the mutil-objective optimal points in each and every cluster
k-means-lite can solve 100,000 points (and much larger datasets), in fractions of a second, on today's typical laptop. More efficient than several known 'efficient' algorithms: running time is constant in the number of data points N. Also very easy to understand and implement.
Briefly and simply described in:
https://www.researchgate.net/publication/328497599_k-means-lite_real_time_clustering_for_large_datasets
You still have to supply the number of clusters k. However, since it is very fast, you can adopt common approaches like the elbow or silhoutte methods which involve multiple runs of your clustering algorithm.
- Badges
- Science topic