Hi,
I confused on one thing, so I need you help in figuring out the right way:
I have introduce a supervised machine learning-based framework, which is definitely easy to evaluate because you only have labeled data, now the thing is that I have enhanced the same supervised machine learning framework to have semi-supervised capabilities. So, I have labeled data and unlabeled data. I used self-training to classify the unlabeled data and add them in the dataset if the predication probabilities of unlabeled data is above threshold. Now, my question is how to evaluate my semi-supervise learner and see whether it add value in the supervised learning model?
I am confused if I use the updated dataset (with newly added data using semi-supervised learning) for evaluation then wouldn't it add a bias in my results? because my classifier is not 100% accurate and it might have add some false positive samples from the self training part.
It will be hard to get 100% in text classification if you have many and highly diverse text snippets. I suggest to test the clusters via precision / recall. We have published two papers about it Article The Application of Artificial Intelligence Technologies as a...
andArticle Past, Current and Future Trends in Enterprise Architecture –...
Ruediger Buchkremer Thanks for you guidelines. Yes I am using precision and recall as an evaluation metric. I am confused in figuring out training and test sample. With some research I found out that I can use same training and test samples for the baselines. For the semi-supervised learning I will added new unlabeled data in the training sample and verify whether the results have improved. Do you think, its the correct way? Actually I am unable to find any standards for evaluation in semi-supervised setting.
The correct approach is build your model only from the fully labeled training set. After doing so, test your model using a test set data (different from the training set) that does't have labels. Knowing that we tend (sometimes) to provide a labeled test set to get the statistical result for clear model assessment process.
To evaluate your model, you can rely on several metrics such as Accuracy, ROC, Kappa statistic, F-Measure, Root mean squared error, and Confusion matrix. There are many other methods, so you need to explore the literature well.
HTH.
Dr. Samer Sarsam
Dear Ali Masood,
I recommend that you do a simulation with fully labeled data. You can randomly select a subset of the data to represent the unlabeled data. Any time your model makes a prediction, you would check the actual label (which you know) and score accordingly. I also recommend that you perform cross-validation.
If you plan to use unlabeled data, then you would have to manually verify each prediction and score accordingly. If the class is somewhat subjective, e.g. mood or emotion, then you would need several people to verify the predicted class and reach consensus.
- Badges
- Science topic