I am using machine learning techniques to train models for different training candidates which are actually combination of original datasets. I am creating training candidates by adding no more than 3 datasets at a time. I am using text mining in rapid miner tool to extract terms from bug summary and using them to train model. My problem is that when I am adding dataset in training set number of common terms between training and testing datasets are decreasing in comparison of number of common terms for single training dataset and f measure performance is either increasing or decreasing independent of number of terms. What is the reason behind this? Why common terms are decreasing when we are adding datasets and f measure trend is not constant?
I venture to say that your common terms are decreasing because the dataset you are adding does not have the same common terms than the first one. In a nutshell if you have lets say Hamburger to be a common term in the first dataset because it appears in 300 out of 500 documents, so its on 60% of your documents. Hotdog in this combination is only on 100 out of the 500 documents (20% of them) not making it to your 'common term threshold'. Once you add another 500 documents from the second dataset that have Hotdog in 400 of them, and Hamburger in only 50 of them, your common terms will change to hotdog in (100 + 400 out of 500 + 500) = 50% of the documents and Hamburger (300+ 50 out of 500 + 500) = 35% of the documents making your number of common terms (and which ones too) change. This is also affected by the frequencies. To further test for this, you need to know the distribution of your terms in your datasets and verify they are similar. This way when you agreggate them, you will get a constant number (and which terms) are common between datasets. If you mix datasets of Sports news and the Weather, you will lose common terms found on the individual datasets since they are not realted and very likely have just a few common terms.
- Badges
- Science topic