Recently, several studies on calibrating traffic flow models have been undertaken using support vector regression (SVR). For me, the method seems to be a bit counter-intuitive. Rather than the sum of the squared errors, the sum of squared prefactors of the model function is minimized. However, this seems to have nothing to do with the fit quality itself. The fit quality enters only indirectly in form of some constraints, and small deviations are not penalized at all. Furthermore, you obtain a sort of black-box model which is lengthy to write down explicitly and which cannot be understood intuitively. Under which circumstances, the SVR should nevertheless be preferred to an ordinary LSE minimization?
Support vector machines are typically used in contexts where the objective is to classify data. Namely, when the dependent variable takes values in a small countable set. For example, when your dependent variable is binary (representing, for instance, congested traffic and free-flow traffic). In this case, the optimization is geared towards finding a cutting plane through the data which separates the data into two regimes in a way that maximizes the distance from the cutting plane to both datasets (the margin). Unlike OLS, the plane that you find is not a model for predicting the dependent variable, but rather a plane above which you predict one outcome (e.g., 0) and below which you predict another (e.g., 1).
Hi Saif: my question did not relate to using SVM for classification (the dependent value is kategorially scaled and typically takes on a small number of integer dummy values) but for regression where the dependent variable y is, in fact, real valued, and is approximated by a function f(x) where x is the vector of exogeneous variables.
In my view, the main difference to ordinary regression is that the SVM regression function f(x) will not change when adding additional data points (y_i, x_i) as long as f(x_i) does not deviate more than a certain margin epsilon from y_i. Furthermore, further deviations are, in effect, penalized linearly instead of quadratically. As a third difference, the objective function also contains a quadratic term penalizing too high absolute prefactors of x. This is referred to as "flatness" and I do not understand the rationale behind it. .
In my view, SVM classification and (nonlinear) SVM regression are related by what I visualize as a "rubber sheet" or "rubber volume" in (x,y) hyperspace separating (i.e., classifying) the region "allowed" for data points (inside the sheet) from the forbidden zone outside. In y direction, the rubber sheet has a natural thickness of two times the margin epsilon. If new data points lie inside this natural thickness, everything is OK and the SVR regression result, i.e., the positioning, orientation and curvature of the sheet, are not changed. If new data points lie outside, however, and this cannot be resolved by repositioning or re-orienting the rubber sheet, its thickness has to be expanded to contain all data points once more, and this expansion is penalized linearly.
The question remains: When to use SVM regression rather than normal regression?
Hi Martin, thank you for the nice clarification. I now have a better appreciation for your question. Here's my take: consider the following estimation problem: Let x_i denote the a sample of exogenous variables and let w denote the vector of prefactors of x. For simplicity, let's say f(x_i) = w.x_i + w_0, that is, we are fitting a linear function and where w_0 is an intercept. A general estimation problem is given by
minimize [ \sum_i L( x_i, f(x_i) ) ] + \lambda ||w||_2,
where L(,) is a "loss"/distance function, \lambda is a constant weight (Lagrangian multiplier) and ||.||_2 is the l_2 norm.
The second term "\lambda ||w||_2" is interpreted as a rugularizer, which is usually included to mitigate the possibility of over-fitting and to promote a unique solution to the estimation problem.
For LSE minimization, one chooses an l_2 distance for the loss function, "L(,)". For SVM regression, one chooses an l_1 distance function (the absolute value). The difference between these two loss functions is that the l_1 distance function promotes solutions which are sparse, i.e., solution vectors with many zero entries. In the context of SVMs, this translates to only a small number of points in the dataset dictating the form of the fitted function f(); namely, those which lie on the boundary of the margin.
So, I guess your question boils down to: when is sparsity a preferred property? In general, I think the answer depends on context. Sparsity has its advantages when dealing with very large datasets and one is interested in understanding which small subset of the samples is most representative of the entire set. It is also a key attribute for compression.
I hope this provides some clues towards an answer to your question. This is indeed a very interesting question.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size