Hello all,
Pls. tell me how to do data mining on imbalanced dataset of Recommender Systems.
As per my knowledge one need to create training and testing files for each user, having their ratings as the class label. For Ex:
Suppose user have given Item1, Item4 and Item3 rating 5, 5 and1 respectively and we want to predict his rating for Item6
For a user1 the training data will be:
User1_F1,User1_F2,......................Item1_F1,Item1_F2....,5
User1_F1,User1_F2,......................Item4_F1,Item4_F2....,5
User1_F1,User1_F2,......................Item3_F1,Item3_F2....,1
// User1_F1 (shows the feature of user1) & Item3_F1 (shows the feature of item3) and so on...
testing data will be:
User1_F1,User1_F2,......................Item6_F1,Item6_F2....,?
Please correct me if i am wrong....
Here,as we can see class label 1 comes only one time but class 5 comes two times, how to remove this imbalanced dataset problem?
And also tell me how to handle imbalanced data or any tool that can do pre-processing before applying Data Mining techniques on this data?
Hi Nidhi,
The simplest way of approaching the class imbalance problem is by sampling during pre-processing.
Sampling can be either oversampling or undersampling. Oversampling is done by increasing the number of minority class examples, whereas undersampling is done by decreasing the number of majority class examples.
In your example, the minority class is "1" and the majority class is "5". If you were to choose oversampling, you could duplicate the third record. If you were to choose undersampling, you could remove either the first or second record.
In answer to your question about pre-processing: Sampling is typically done as a pre-processing step.
I would personally recommend the highly popular oversampling algorithm: SMOTE (3000 citations). There is an implementation of SMOTE available in WEKA.
I've attached a link to the SMOTE paper which describes the class imbalance problem very well.
I hope this helped!
Mike.
https://www.jair.org/media/953/live-953-2037-jair.pdf
Hello,
You can use collaborative learning algorithm.
1- Build a matrix of entries where rows are customers/clients while the columns are the items to be proposed.
2-Assume that known entries can be represented with regression model then generalize it to predict what might be the nearest neighbor of the unknown entry.
3- Optimize the cost supposing as if all entries are known
4- Find the best matching prediction.
I hope this will help.
Turgay
Hi Nidhi,
The simplest way of approaching the class imbalance problem is by sampling during pre-processing.
Sampling can be either oversampling or undersampling. Oversampling is done by increasing the number of minority class examples, whereas undersampling is done by decreasing the number of majority class examples.
In your example, the minority class is "1" and the majority class is "5". If you were to choose oversampling, you could duplicate the third record. If you were to choose undersampling, you could remove either the first or second record.
In answer to your question about pre-processing: Sampling is typically done as a pre-processing step.
I would personally recommend the highly popular oversampling algorithm: SMOTE (3000 citations). There is an implementation of SMOTE available in WEKA.
I've attached a link to the SMOTE paper which describes the class imbalance problem very well.
I hope this helped!
Mike.
https://www.jair.org/media/953/live-953-2037-jair.pdf
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining