Dear all,
My question is the following:
I have large datset: 100,000 observations, 20 numerical and 2 categorical variables (i.e. mixed variables)
I need to cluster these observations based on the 22 variables, I have no idea how many clusters/groups a priori I should expect.
As the large dataset I use clara() function in r (based on "pam").
Because of the large number of observations, there is no way to compare distance matrixes (R does not allow such calculations, and is not a problem of RAM), therefore the common way of cluster selection using treeClust() and pamk() and comparison of "silhouette" does not work.
My main quesitons is: can I use factors like total SS, within SS, between SS to have an idea of the best performing Tree (in terms of number of clusters)? Do you have any other idea of how can I select the right number of clusters?
Best regards
Alessandro
It is easy to do this on R. You have to download sone R packges.
Here is the R code to select brst cluster;
clara(x, k, metric = c("euclidean", "manhattan", "jaccard"),
stand = FALSE, samples = 5,
sampsize = min(n, 40 + 2 * k), trace = 0, medoids.x = TRUE,
keep.data = medoids.x, rngR = FALSE, pamLike = FALSE, correct.d = TRUE)
Here i added a link, you can get lots of info related to your question.
https://www.datanovia.com/en/lessons/clara-in-r-clustering-large-applications/
Dear Sachin,
thanks for your answer. Do you also have an idea which factor can I use to select the tree with the "best" number of clusters? Because at the moment this is the main argument and which factors should I use to make the selection. Something like clValid() does. But in my case, as the dataset is too big, it does not.
Prepare a one way table having 22 columns for the 22 variables.
Then, place the observed values in the data set under the respective columns.
This table will be a table representing one-way classified data.
Then perform analysis of variance.
If necessary, carry on LSD test.
you can find the solution in this paper
Article A New Data Clustering Based on Critical Distance Methodology
The book
" Experimental Design" by Federer
will be helpful in this situation.
- Badges
- Science topic
- Similar topics
- Mathematics
- Algebra
- Matrix