I have a low accuracy(50%) on my image classification when using supplied test data but high accuracy (84%) when using 10-CV on my 900 images (observation data). From my reading, this happen due to overfitting. FYI, I am using LIBSVM with RBF kernel. I have done several things to overcome overfitting which includes:
a) increase data set & features (before this i got only 400 images)
b) Normalised the data
c) use a different images in test sets that have the same features in training
However, it still gives a low accuracy. Therefore in order to find the best accuracy I need to do some grid search to find the suitable setting for gamma and cost that hopefully will increase my accuracy.
After doing some grid search in Weka, I still got poor result in my accuracy as shown below. Am I wrongly set the grid search parameter?
=== Classifier model (full training set) ===
weka.classifiers.meta.GridSearch:
Classifier: weka.classifiers.functions.LibSVM -S 0 -K 2 -D 3 -G 1.0E10 -R 0.0 -N 0.5 -M 40.0 -C 1.0E18 -E 0.001 -P 0.1 -model "C:\\Program Files\\Weka-3-8" -seed 1
X property: Cost
Y property: gamma
Evaluation: Correlation coefficient
Coordinates: [18.0, 10.0]
Values: 1.0E18 (X coordinate), 1.0E10 (Y coordinate)
LibSVM wrapper, original code by Yasser EL-Manzalawy (= WLSVM)
Time taken to build model: 229.71 seconds
=== Stratified cross-validation ===
=== Summary ===
Correctly Classified Instances 422 49.7642 %
Incorrectly Classified Instances 426 50.2358 %
Kappa statistic -0.0047
Mean absolute error 0.5024
Root mean squared error 0.7088
Relative absolute error 100.4705 %
Root relative squared error 141.7527 %
Total Number of Instances 848
=== Detailed Accuracy By Class ===
TP Rate FP Rate Precision Recall F-Measure MCC ROC Area PRC Area Class
0.198 0.203 0.494 0.198 0.283 -0.006 0.498 0.499 DISENGAGED
0.797 0.802 0.499 0.797 0.613 -0.006 0.498 0.499 ENGAGED
Weighted Avg. 0.498 0.502 0.496 0.498 0.448 -0.006 0.498 0.499
=== Confusion Matrix ===
a b
Hi Siti,
In the result you posted, you are trying to optimize the Correlation coefficient (not the accuracy). You have to select the Accuracy as a metric to be optimized. Then, set XProperty = cost, and YProperty = gamma. You could also consider using the default kernel of LIBSVM.
NB: To understand the GridSearch result, I recommend you reading about it.
HTH.
Samer, PhD.
java -cp /users/software/weka-3-7-12/weka.jar weka.Run weka.classifiers.meta.GridSearch -t all47featuresNorm.csv -d all47TrainFeaturesMod.mod -c last -E CC -y-property kernel.sigma -y-min 1.0 -y-max 3.0 -y-step 1.0 -y-base 10.0 -y-expression pow\(BASE,I\) -x-property C -x-min 1.0 -x-max 3.0 -x-step 1.0 -x-base 10.0 -x-expression pow\(BASE,I\) -sample-size 100.0 -traversal ROW-WISE -log-file /Users/amita -num-slots 1 -S 1 -W weka.classifiers.functions.SMOreg -- -C 1.0 -N 0 -I "weka.classifiers.functions.supportVector.RegSMOImproved -T 0.001 -V -P 1.0E-12 -L 0.001 -W 1" -K "weka.classifiers.functions.supportVector.Puk -O 1.0 -S 1.0 -C 250007" Options: -E CC -y-property kernel.sigma -y-min 1.0 -y-max 3.0 -y-step 1.0 -y-base 10.0 -y-expression pow(BASE,I) -x-property C -x-min 1.0 -x-max 3.0 -x-step 1.0 -x-base 10.0 -x-expression pow(BASE,I) -sample-size 100.0 -traversal ROW-WISE -log-file /Users/amita -num-slots 1 -S 1 -W weka.classifiers.functions.SMOreg -- -C 1.0 -N 0 -I "weka.classifiers.functions.supportVector.RegSMOImproved -T 0.001 -V -P 1.0E-12 -L 0.001 -W 1" -K "weka.classifiers.functions.supportVector.Puk -O 1.0 -S 1.0 -C 250007" weka.classifiers.meta.GridSearch: Classifier: weka.classifiers.functions.SMOreg -C 10.0 -N 0 -I "weka.classifiers.functions.supportVector.RegSMOImproved -T 0.001 -V -P 1.0E-12 -L 0.001 -W 1" -K "weka.classifiers.functions.supportVector.Puk -O 1.0 -S 10.0 -C 250007" X property: C Y property: kernel.sigma Evaluation: Correlation coefficient Coordinates: [1.0, 1.0] Values: 10.0 (X coordinate), 10.0 (Y coordinate) Could someone please explain the meaning of Coordinates and Values and what were the final best parameter values. Thank You for the help. Amita -- Amita Misra Graduate Student Researcher Natural Language and Dialogue Systems Lab Baskin School of Engineering University of California Santa Cruz
- Badges
- Science topic