After completing the required steps to build a machine learning model (Training and testing). How to evaluate the model using new data ?. Assuming we built a classification model with Python using Sklearn decision tree algorithm (for spam detection), After getting an accuracy of 0.97, for example, how can we test this model with new data to check if it works fine?
Hi, you need to separate the dataset into three sets of training, validation, and test sets. Use the training data to adjust the parameters of the model and use the validation data to see how the training process behaves and avoid the overfitting. Finally, use the test data to check the generalization error on the unseen data.
It is better to use standardized training and validation data. For standardizing, the test data use the mean and standard deviation of the training data.
Dear Lakhdar Belkharroubi ,
The three main metrics used to evaluate a classification model are accuracy, precision, and recall. Accuracy is defined as the percentage of correct predictions for the test data. It can be calculated easily by dividing the number of correct predictions by the number of total predictions.
https://www.jeremyjordan.me/evaluating-a-machine-learning-model/#:~:text=The%20three%20main%20metrics%20used,the%20number%20of%20total%20predictions.
Regards,
Shafagat
Lakhdar Belkharroubi There are numerous approaches for evaluating a machine learning model with new data. Cross-validation is a technique that is commonly used.
The original data is randomly partitioned into k equal-sized subgroups in k-fold cross-validation. The model is then trained on k-1 subgroups before being tested on the remaining subset. This method is done k times, with each subset serving as the test set just once. The model's performance metric is the average performance measure across all k iterations.
A holdout set is another strategy in which a portion of the data is held aside as a test set and not used during the training phase. The model is trained on the remaining data and then tested.
To assess the model with fresh data, use the classifier's prediction method and compare the predicted labels to the true labels of the new data. To evaluate the model's performance on the new data, you may compute accuracy or other performance measures like precision, recall, and F1 score.
In your scenario, you may separate your data into training and testing sets using sklearn's train test split function. Then, using the decision tree's classification technique, categorize the new data and compare the results to the real labels.
It is crucial to highlight that just because a model performs well on the training and testing sets do not mean it will perform well on fresh untested data.
- Badges
- Science topic
- Similar topics
- Biological Science
- Biochemistry
- Biomolecules
- Peptides