Hello Siti,

I think the problem is in the choice of your testing data as you can see your accuracy becoming poor as you increase training data and keep testing data constant. I am assuming your testing data was not correctly drawn from your training data i.e your testing data is not from the same population as your training data or not having appropriate representation as it is in the training data. For good and correct accuracy, your testing data must be drawn from a similar population as your training data and must contain appropriate representation of the classes in your training data. I can see you also later mention "imbalance". That could also be a great problem to solve before classifying your data. You may need to do some form of resampling to have a relatively balanced population samples. If this is not done, your algorithm will mostly work in favour of identifying the majority class and also can erroneously classify your minority class as belonging to majority class. I do hope this helps you a little to push ahead in the right direction. Wishing you all the best. Adeyemi.

Hi Siti,

Having a very high accuracy value like yours (97%) is acceptable *only* if your model gets evaluated using, e.g., stratified 10-fold cross-validation. However, if your accuracy value is obtained from evaluating your model on the training set, then your result is not reliable because you may have an overfitting problem. Thus, I'd suggest relying on the cross-validation result when evaluating your model, and not on the external (unlabeled data). That is said, the stratified 10-fold cross-validation performs training and testing in each fold with almost same class distribution in both train and test sets. On the other hand, evaluating the predictive model on an external dataset (unlabeled data) would vary based on the provided data, so it's not recommended.

To enhance the accuracy of your model,  you may need to do feature selection and/or configure LIBSVM. However, for the new data, you should clean them up before adding them to the existing one. Probably, the initial dataset that you worked with is useful for your task, and adding additional instances may not enhance the result. 

Note that, although you have 97% accuracy, that doesn't mean your algorithm has an excellent performance. You need to explore other aspects, like ROC area.

HTH. 

Samer 

I believe your experiments are complaining from the problem of overfitting which was obviously appeared when the classification accuracy dropped from 80% to 50% on increasing the number of training images from  294 images to 310 images.

So, I suggest to use the random forest classifier which is a random decision forests. Random decision forests correct for decision trees' habit of overfitting to their training set.

Hi,

As Samer hinted above, it is important to use stratified cross-validation, especially if you have large imbalances in the classes; for example, if all of the new images you added are "negatives", it is not so surprising they don't help much.

Second, if computationally feasible, it would be helpful to perform a grid search to optimize the hyperparmaeters of the models (e.g., k for kNN, the kernel and its parameters for the svm, etc.). Or, even better, use a hyperparameter optimization technique like auto-sklearn in python; I'm sure some also exist in R, etc. It is then possible to optimize the hyperparameters wrt AUC, F1, or some other metric which better reflects the performance than accuracy. Regardless of the metric you choose, though, this should improve your performance.

Of course, for fair evaluation, make sure to use an *outer* cross-validation loop. That is, first split the data into however many folds you like; then, further split the training sets into a validation and training set, and optimize the hyperparameters using those. Finally, test on the (completely unseen) testing data originally held out for each fold. Thus, you will end with 10 AUCs, accuracies, etc. (for 10-fold cv). These can then be used in something like a (Bayesian) t-test for comparing different approaches.

If a grid search is not computationally feasible, then I agree with Khaled that moving to some form of random forests (say, gradient boosted tress) are probably a better choice; they tend to be fairly insensitive to hyperparameter choices. They also have some theoretical advantages over single trees, though it is less clear if they have theoretical advantages compared to svms.

Have a good day,

Brandon

Edit: fixed typo

Hi,

samer sersam please i didn't understand what you mean by "However, if your accuracy value is obtained from evaluating your model on the training set, then your result is not reliable because you may have an overfitting problem". why do you say that overfitting was the cause of having such high accuracy (97%). i think it is always the case (i mean we will obtain always such good accuracies ) when we test the model on the training data (because it is the training data that was used to build the model).

In my humble opinion,  i think that we can observe an overfitting when we get wery worst accuracies using test data in comparaison with those obtained using training data (test data contains also new instances that were not used in the training) which signify that the created model doesn't perform well on unseen instances and thus it generelize poorly.

Thank you.

best regards.

Your statement about resizing the images after already trying them in your model makes me think that you are preprocessing your images differently than the data that was used to train the model. Its important that new data receives the same preprocessing as the training data. It may be helpful to create a pipeline to manage this for you in a consistent way.

Additionally, convolutional neural networks are very good for image classification tasks, i suggest exploring them for your case. You can also supplement your image data by applying various transforms to your images like a shear or reflection transform (or both) which can help to further combat overfitting during training; if doing this it is important to make sure the transformed images are similar to something you may naturally receive in any new images.

Just a few tips i didnt see already mentioned, hth.

Hi Wassila,

I meant that the predictive model gets tested on the same data that it is familiar with (trained on). In other words, the model memorizes training set very well, so it knows the target label(s) of each instance. This, in turn, will definitely lead to obtaining an optimistic result. Therefore, we recommend applying the stratified 10-fold cross-validation to prevent overfitting problem. If the 97% is the result of the model under the evaluation of cross-validation, then the result is reliable and there is no cheating occurs in the learning process. 

You're right. In general, overfitting can be recognized by comparing the accuracy the model achieves on training data compared with the accuracy achieved on the validation dataset (or test set). But in the real-world conditions, the test set data is not always available; thus, having high accuracy result on the training set, highly indicates that we have an overfitting problem. 

Anyway, for more information, please have a look at this link:

http://wiki.fast.ai/index.php/Over-fitting

Cheers, 

Samer

Hi  Suhaila,

In addition to the above suggestions, I'd like to raise the following points:

• You probably need to scale your features before to use them with the selected classifier.
• Features selection are recommended if you have many feature parameters ( stepwise is one of the method for feature selection).
•  Leave-One-Out method could overcome the problem, if you have limited data.

Thanks,

Abdelrahim

Thank you samer sersam  for your explanation.

Best regards.

Better select the significant features either by PCA or GA or whatever. Then you may proceed. As others have suspected, it is a case of over fitting. Otherwise, a simple k-means algorithm would suffice for classification.

Select only the significant features. Test with any one classification algorithm then proceed with combined algorithm 

Dear Siti Suhaila,

Look the link, may be useful.

https://stats.stackexchange.com/questions/59630/test-accuracy-higher-than-training-how-to-interpret

https://machinelearningmastery.com/improve-deep-learning-performance/

Regards, Shafagat

Some very intresting references.

Here is the latest update:

1. I re-look into my training data and add more images in terms of angle and features and used SURF descriptor to detect the key interest point

2.Using the training data, I used stratified 10 folds cross-validation (using Weka) and classified using LIBSVM  with linear kernel and managed to get 95% accuracy

3.However, when I test the data using sample test with unlabeled data the accuracy varies from 50% to 76% depending on the numbers of test data and different types of images (angles). It shows that when I have many new images with a different angle that not in the training data, the result will be so poor. Should I cover more wide angles? What is the threshold that I can rely on to stop to add more data in the training data and look for others algorithm?  

I am sorry for being helpless.

Dear Siti Suhaila

you did not mention type of image (texture, face, etc ) each type would be classified better with certain features than others. you might need to get features from transformed domain. the nextstep is try to specify the best feature set you could use GA for example to find ~ best feature set, or you could use PCA to minimize feature set. for sure you might need proper image pre-processing stage.

Hi Siti, 

I recommend using the learning curve approach to find the suitable training set size. 

What I can say (without seeing your data) is that your algorithm doesn't have a good performance although that it has a high accuracy. I recommend cleaning your dataset well and making it balanced before evaluating your model with stratified 10-fold cross-validation. In addition, for evaluating the performance of your algorithm, use, e.g., ROC area, F-measure, etc. and *don't* only rely on the accuracy value. You could also try other learning algorithms, such as LibLINEAR and SMO (or whatever algorithms you like), then compare their results. I don't know your data size, but you could try attribute selection as well as selecting the suitable scale for your data.

NB: Make sure that you have reasonable predictors; otherwise, you'll not get an excellent result. 

HTH.

Samer

Dear Venus,

For your information, the data that I am using are face images (frontal face) with no specific regions because I just want to detect the whole face, half face or no face. Therefore I did not use the Adaboost or Viola-Jones algorithm instead I used SURF features to detect the key interest point. Am I in the correct platform?

The algorithm does not give expected results with adding new images. It means proper cleansing is required before test those images. As you are using only facial image without any resizing or pre processing and algorithm's performance more deteriorate. It means proper cleansing and pre processing is required before testing. 

hi .

1 ) i think u must to use beter feature or use more feature's . like HOG,LBP or use HOG and SURF together.

2) make change in SVM parameter . like change kernel to gaussian.

3) use 70% of data for train and 30% for test (select randomly)

i hope be usefull.

I think as you are doing this for face and you want to detect whole face, half face or no face. So it is clear that interest points in face region will same for 3 classes except no face. Therefore in my opinion interest point or other low level features can not differentiate these classes.

You should jump to CNN classification for this problem.

Tq all. Before I choose another algorithm should I strictly balanced the number of instances of both classes (I am doing binary classification)? Because I got a lot of negative samples as compared to positive samples in my training data. Can this make a big difference?

Hello Siti,

I think the problem is in the choice of your testing data as you can see your accuracy becoming poor as you increase training data and keep testing data constant. I am assuming your testing data was not correctly drawn from your training data i.e your testing data is not from the same population as your training data or not having appropriate representation as it is in the training data. For good and correct accuracy, your testing data must be drawn from a similar population as your training data and must contain appropriate representation of the classes in your training data. I can see you also later mention "imbalance". That could also be a great problem to solve before classifying your data. You may need to do some form of resampling to have a relatively balanced population samples. If this is not done, your algorithm will mostly work in favour of identifying the majority class and also can erroneously classify your minority class as belonging to majority class. I do hope this helps you a little to push ahead in the right direction. Wishing you all the best. Adeyemi.

I think you can improve your classification performance by using leave one out cross validation. this cross validation will divide you data according given ratio you are passed as input in the function. you are able run multiple times to find best data fit your model. this is the link of cross validation in the matlab: https://www.mathworks.com/help/bioinfo/ref/crossvalind.html

The accuracy entirely depends on your data only. Try using clear and pure data for training and testing . In case of SVM, The kernel also make sense. I don't know which you're using. If you're using "Linear" kernel, try with "RBF" and "Polynomial" kernel too. You may also try extracting features from binary images of different threshold.  

But, since your data is not reach, it's better to use Convolutional Neural Network (CNN) for building a more powerful classifier.

Arindam Kumar Paul, 

Strong classifiers cannot perform that much if you have horrible data. The main problem is having unbalanced data. In addition, the dataset is not cleaned properly. Moreover, adding additional instances that seems to be not consistent with the existing ones, affects the quality of the resulting data and the classification result, though. Once settling these issues as well as the right number of the training set to be used, the learning process can be correctly accomplished with several classification algorithms as long as the evaluation of the learning process has performed under the stratified 10-fold cross-validation method.

HTH.

Samer

Did you try applying edge detection on the dataset images before training and testing? since you just want to know if there is a face or no face only and it it is hulf, full, or no face. i recomend using leave one out, for trainig and testing. choose features and classifiers that are inveriant to rotation scalling translation (try the trnsformed domain).

Since your data is imbalanced, I can detect two major mistakes without even further details:

- The SVM (no matter which kernel) is strongly dependent on class balance. To fix that you could either use something like SMOTE or optimize the class weights in the SVM. In general, you should definitely optimize the core hyperparameters of the Libsvm: regularization constant C, class weights and kernel hyperparameters, if a kernel is used.

- Even if your testing data is similar to your training data, you will get weird results. The reason is your chosen metric. You need to use a metric that is invariant to the class distribution. Accuracy is the worst metric that you could use in your case. It could be that everything is classified as one class and you get still 95%. However, when your test dataset is more balanced you would get results around 50%. For further reading about metrics, take a look at my paper: Article How to evaluate an agent's behavior to infrequent events?—Re...

Try Bag of Visual Words. Its give better classification accuracy.

You can use clean data for trainset and testset

The best way is to increase the number of training examples