I am also working with this type of problem

I would suggest you try 2 methods:

1. something simple like k-means if you are partially satisfied you may then try something a little bit more complecated like Fuzzy C-Means

2. Try something like c45 and svm

And let me know how this go with you ;-) 

Dear Mr Mortazavi,

You can try Fuzzy C-Means (FCM), because its simplicity is powerful to handle very large data, as proven in current research.

You can refer to this paper for more detail:

(Fuzzy C-Means Algorithm for Very Large Data)

http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6205366

Good Luck

Dr. Mortazavi,

    I think you should try to code an algorithm similar to k-means that would cluster according to the restrictions you specified. Alternatively, you could fit mixture models using the suite of model-based clustering approaches available in R. The only issue is, there is no guarantee that MSE and sample size will be as required.

To my mind the bes clustering for this kind of dataset is logistic regression function . This function is very efficient for 2 classes dataset.

http://www.ats.ucla.edu/stat/stata/dae/melogit.htm

Dr. Mortazavi,

   Contact DataStax and pose your question to them.  This company has a free download of a single user version of Cassandra, SPARK and ML Library.  You might try using MatLab and their ML Library; however i would contact DataStax and see if you can speak with one of their Engineers.  BTW, DataStax's Cassandra is in use at CERN.

I hope this helps.

Dear Mr Mortazavi,

A great clustering algorithm I'm working with is the Spectral Clustering.

It will satisfy your demand for clusters with almost the same sizes (solves the normalized cut problem), and works really good for big data set since the algorithm performs dimensionality reduction before clustering in a smaller dimension.

Hi Mortazvi,

The best way to pursue this problem is to have a look at the fuzzy c-mean algorithm. The spectral clustering, simple k-means, hierarchical clusterings are not suitable for large data sets. You can use dimensionality reduction techniques because clustering at the cost of losing some information. 

If your data is not that complex and you just want to make two clusters then I guess using PCA, SVD before applying k-means or spectral clustering would be fine. 

If your data is really large (trillions of objects) then I would suggest to use Hadoop, Cassandra or Strom for efficient data manipulation.

Cheers

You might try using Spark/Storm and the ML algorithms the come with the Product.

The ML code is in either SCALA or Python (try the Python stuff first).

If you are looking for something to cluster unknown number of clusters, you have also two choices:

1. try to use pattern recognition concepts to detect number of clusters then feed that to another clustering algorithm, or

2. You may look at the heirarchical dirichlet process (HDP) algorithm and its details. Basically number of clusters is dynamic (undefined prior to the clustering process)

Enjoy !

Use Fuzzy k means. You can get an idea of the performance of cluster numbers and phi by plotting that against the fuzziness performance index and the partition entropy. Also as mentioned, you can reduce dimensionality by performing PCA on your data.