Suppose we have users, for each user, we have: user_id, user_name, user_job title, user_skills, user_workExperience. I need to cluster the user based on their skill and work experience( long text data), put the users into groups. I was searching about how to clustering text data but still didn't find a good example to follow" step by step". Based on the data I have I think I should use unsupervised approach (as the data I have is not labeled), I found that I can use K-Mean or hierarchical clustering, but I'm stuck in how to find: K "number of clustering with K-Mean". Also, I don't know what is the best way to prepare the long text before fit into the clustering algorithm. Any idea or example that can help me, would be very appreciated. Thanks in advance.
Hi Huda,
I advice you to try a few clustering algorithms. If you don't know the cluster number, you have to use hierarchical clustering methods. But you can find cluster number by estimation algorithms and then use non-hierarchical clustering methods like fuzzy c-means, k-medoids or k-means.
hi, you can also use FCM algorithm since it is unsupervised approach and you can set the number of clusters c randomly at firs time.
In this case and using this type of data I recommend the hierarchical clustering. You bid to define the number of clusters using silhouette or PCA. It may be useful to do a PCA analysis before making clustering. If your are not familiar with matlab you can use originpro to obtain your cluster and statistical report.
In addition to what was recommended above, I suggest that you should first think how better create data vectors. As I understand, you have categorical data. For example, "user_job title" is a list of professions expressed by words, not numbers. So, instead of using N digits to represent these professions in a vector you can use a binary scheme called "one-hot encoding".
Concerning the number of clusters, it is actually an important issue that shows what result you want to get and what exactly you want to know about your data. You can ask experts, how many categories they expect to see as the result, or to create a scale of "skill and work experience" and divide it into segments. Or you can ask experts to label a small group of users and use the labeled data as a standard. You can also try to analyze your data mathematically or statistically, however, the method will again depend on what you want to know about your data in the end.
the clusterR package has a lot of options and a lot of options and ways to assess how good a method is. Also random Forest is a nice approach.
I second Murat Karakoyun , you should try several algorithms and see which one works best for your data. Here's a nice overview of several: https://towardsdatascience.com/the-5-clustering-algorithms-data-scientists-need-to-know-a36d136ef68
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python