More specifically, Is there a clear correlation between feature values distribution among training samples and the weight assigned by the linear SVM model?
Say, I have a feature f1 that appears in both positive and negative classes in training samples, and for simplicity, let's assume the feature values are binary. If f1=1 appeared more frequently among positive training instances than among the negative ones. Would linear SVM learn a positive weight for this feature f1?
And does dimensionality influence such correlation? (I'm thinking maybe higher dimensionality will make it harder to correlate as there could be redundant features?)
Many thanks in advance!
In multivariate case, I would want to believe such feature appearing way more in the positive side may appear as noise in the negative side. Another case is that, it is redundant feature which is not a determinant for the target variable, in both situation.
Hey Gabby.
Unlike many ML and DL algorithms, SVM works with support vectors, and thus the weights are either \alpha or 0 for the features. If the weight of a feature is \alpha, it is said to be a support vector, and if 0 the model disregards the feature entirely.
In order to have a linear SVM with overlapping classes, a slack parameter must be given to the model (I believe the common notation is with \epsilon). The advantage of the slack parameter is that it allows misclassification. If so, the two most likely scenarios that I can think of are:
1) the conflicting feature will end up with a weight of 0 (i.e., not a SV) and therefore it does not affect your model.
2) If the optimization results in that feature being a SV, then it will most likely end in the class where its labels are repeated most. It will also depend on the classes of the features around it since, in the end, the SV of a linear model is just the "equilibrium" of the "center of mass" between the classes' SVs. So, as Stanley mentioned above, the feature will be just "noise" in your classification.
In the case of redundant or highly correlated features in a multidimensional dataset, it's likely that only one of the redundant features will be a SV and the rest will be disregarded. Unless these points are very close to the decision boundary.
I might be missing a couple of things here but I hope this helps.
Cheers
Gabby Xiong
If we talk about linear SVM , the result is a hyperplane and that help separates the classes . Weights represent this hyperplane by giving you the coordinates of a vector which is orthogonal to the hyperplane. These are the coefficients given by hyperplane.
Regards
Lakchchayam Divya Khare
Hi, thank you all so much for your insightful answers!
This is actually a follow-up question, I followed the answer from Sergio Cofre-Martel, and looked into the support vectors chosen by the trained linear SVC model. I found there are a lot of cases that are counter intuitive to me, I was wondering if you can provide me with more insights?
In my project, I have binary classification, and the features are with binary values, so I treat a sample contains feature f if I have f=1 in the sample, and vice versa. I looked at the feature occurrence among the chosen support vectors for the top 100 positively weighted features (I.e. top 100 largest positive weights in the model), and I found 45 out of the top 100 features are actually used more among the support vectors for the negative class than among those for the positive class. I was originally expecting more features to have the sign of their weight to be correlated with their usage among support vectors, as the model has high accuracy. I'm find this to be counter-intuitive and not sure what could be the reasons behind it, I would be really grateful if you can provide me some insights on this!
This also makes me think that it is not possible to predict whether a feature will be assigned with a positive / negative / zero weight in SVM, even when we know its usage among training samples, or usage among support vectors? Or can we?
Thank you fo your time in advance!
Best Regards
Hey Gabbi,
I'm glad that my answer was of help to you.
I'm not sure if I fully understand the concerns that you have. From what I can gather, after you trained your binary SVM classifier, you extracted or printed out the support vectors obtained by the model. Correct?
Depending on how you extracted this information I can see 2 possible scenarios:
1 - You extracted the lagrangian weight associated with the dual optimization (beta in the article linked below). It is my understanding that this value must be greater than zero for SVs and zero for non-SV. This doesn't seem to be your case since it appears that you have negative values (?).
2 - You extracted the actual vectors chosen as SV, i.e., now you basically have a subset of your original training input features (X). In this case, you should not worry about the balance between SV from one class or the other. It is not unlikely that one class might have many SVs and the other just a few, even with ratios like 1:10.
I don't know if this is of any help or not. If I have misunderstood your question please let me know.
All the best,
Sergio
https://www.analyticsvidhya.com/blog/2020/10/the-mathematics-behind-svm/
- Badges
- Science topic
- Similar topics
- Data Mining
- Classification