Soft-SVM Regression For Binary Classification

Given our characterization of the Soft-SVM exponential family distribution, performing Soft-SVM regression is equivalent to fitting a regularized generalized linear model. To this end, we follow a cyclic gradient ascent procedure, alternating between an iteratively re-weighted least squares (IRLS) step for Fisher scoring [7] to update the coefficients $\beta$ given the softness $\kappa$, and then updating $\kappa$ and $\alpha$ given the current values of $\beta$ via a Newton-Raphson scoring step.

For the IRLS step we need the link $g$ and the variance function $V$ in (5). The link is given by solving the composite function $f^{-1}_{\kappa,\delta}\circ[b^{{}^{\prime}}_{\kappa,\delta}(\mu)]^{-1}$ for $\eta$, where

with

and

Note that in all these expressions $\delta$ is always expressed in the product $\kappa\delta$, so it is immediate to use the alternative parameter $\alpha$ instead of $\delta$.

Our algorithm is summarized in Algorithm 1. For notation simplicity, we represent both bias and coefficients in a single vector $\beta$ and store all features in a design matrix $X$, including a column of ones for the bias (intercept).

Since $h_{\kappa,\delta}(\mu)=h_{\alpha}(\mu)=\log\cosh(2\alpha)+\log|\mu-1/2|/\sqrt{\mu(1-\mu)}$, we have that

and so the variance function in (5) can be written as $V_{\kappa,\alpha}(\mu)=\kappa\cdot r_{\alpha}(\mu)$ where, with $u_{\alpha}(\mu)\doteq\text{logit}(\mu)/2+\text{asinh}(\exp\{h_{\alpha}(\mu)\})$,

This way, parameters $\kappa$ and $\alpha$ have specific effects on the variance function: the softness parameter $\kappa$ acts as a dispersion, capturing the scale of $V(\mu)$, while the scaled separation $\alpha$ controls the shape of $V(\mu)$.

Figure 3 shows variance function plots for the ESL mixture dataset discussed in Section 2.4 when penalty parameter $\lambda$ and scale-parameter $\kappa$ are both fixed at $1$, and only allow the shape parameter $\alpha$ to increase from $0$ to $5$. We can see that as $\alpha$ increases, the overall variance decreases and the plot gradually splits from a unimodal shaped curve into a bimodal shaped curve. In practice, we observe that as $\alpha$ increases to achieve higher mode separation, most observations are pushed to have fitted $\mu$ values around $0.5$ and variance weights close to zero, caught in-between variance peaks.

We can then classify observations with respect to their mean and variance values: points with high variance weights have a large influence on the fit and thus on the decision boundary, behaving like “soft” support vectors; points with low variance weights and mean close to $0.5$ are close to the boundary but do not influence it, belonging thus to a “dead zone” for the fit; finally, points with low variance weights and large $\max\{\mu,1-\mu\}$ are inliers and have also low influence in the fit. This classification is illustrated in Figure 4.

As $\kappa$ increases we should anticipate numerical issues as $b_{\kappa}$ and $f_{\kappa}$ becomes non-differentiable up to machine precision. For this reason, we need to adopt numerically stable versions for $\text{asinh}(e^{x})$, $\log\cosh(x)$ and $\log\sinh(x)$, needed by $g_{\kappa}$ and thus also by $V_{\kappa}$ above.

The first step is a numerically stable version for the Bernoulli cumulant; with $\epsilon$ the machine precision, we adopt

This way, the soft plus can be reliably computed as $p_{\kappa}(x)=\text{log1pe}(\kappa x)/\kappa$. Again exploiting domain partitions and the parity of cosh, we can define

We branch twice for $\text{asinh}(e^{x})$: we first define

and then

Finally, we write

to make it stable whenever $x$ is greater or smaller than $0$.

To exam how the Soft-SVM regression works for classification in comparison with SVM and logistic regression, we carry out a detailed example using the simulated mixture data in [5], consisting of two features, $X_{1}$ and $X_{2}$. To estimate the complexity penalty $\lambda$ we use ten-fold cross-validation with 20 replications for all methods. The results are summarized in Figure 5. Because $\widehat{\kappa}$ is not much larger than unit, the fitted curve resembles a logistic curve; however, the softness effect can be seen in the right panel as points with the soft margin have estimated probabilities close to $0.5$, with fitted values quickly increasing as points are farther away from the decision boundary. To classify, we simply take a consensus rule, $\widehat{y}_{i}=I(\widehat{\mu}_{i}>0.5)$, with $I$ being the indicator function.

Figure 6 shows the effect of regularization in the cross-validation performance of Soft-SVM regression, as measured by Matthews correlation coefficient (MCC). We adopt MCC as a metric because it achieves a more balanced evaluation of all possible outcomes [3]. As can be seen in the right panel, comparing MCC with SVM and logistic regressions, the performances are quite comparable; Soft-SVM regression seems to have a slight advantage on average, however, and logistic regression has a more variable performance.

Prior works have shown that the performance of SVM can be severely affected when applied to unbalanced datasets since the margin obtained is biased towards the minority class [1, 2]. In this section, we conduct a simulation study to investigate these effects on Soft-SVM regression.

We consider a simple data simulation setup with two features. There are $n$ observations, partitioned into two sets with sizes $n_{1}=\lfloor\rho n\rfloor$ and $n_{2}=n-n_{1}$, for an imbalance parameter $0<\rho\leq 0.5$. We expect to see SVM performing worse as $\rho$ gets smaller and the sets become more unbalanced. Observations in the first set have labels $y_{i}=0$ and $x_{i}\sim N(\mu_{1},\sigma I_{2})$, while the $n_{2}$ observations in the second set have $y_{i}=1$ and $x_{i}\sim N(\mu_{2},\sigma I_{2})$. We set $\mu_{1}=(\sqrt{2},1)$ and $\mu_{2}=(0,1+\sqrt{2})$, so that the decision boundary is $X_{2}=X_{1}+1$ and the distance from either $\mu_{1}$ or $\mu_{2}$ to the boundary is $1$. Parameter $\sigma$ acts as a measure of separability: for $\sigma<1$ we should have a stronger performance from the classification methods, but as $\sigma$ becomes too small we expect to see complete separability challenging logistic regression.

For the simulation study we take $n=100$ and simulate $50$ replications using a factorial design with $\rho\in\{0.12,0.25,0.5\}$ and $\sigma\in\{0.5,1,1.5\}$. For the SVM and Soft-SVM classification methods we estimate complexity penalties $\lambda$ using ten-fold cross-validation and use standard, non-regularized logistic regression. We report performance using MCC and summarize the results in Figure 7.

The performance across methods is comparable, uniformly degrading as we boost confounding by decreasing $\rho$ and increasing $\sigma$, but differs in more pathological corner cases. The combination of small $\rho$ values (higher imbalance) and large $\sigma$ values (less separability) results in a poorer performance from SVM, as previously reported [2]. Small $\sigma$ values, on the other hand, result in lack of convergence and inflated coefficient estimates in logistic regression fits, even though predicted classification is not much affected. Soft-SVM classification, however, avoids both issues, exhibiting stable estimation and good performance in all cases.

To evaluate the empirical performance of Soft-SVM regression and classification, we select nine well-known datasets from the UCI machine learning repository [4] with a varied number of observations and features. Table 1 lists the datasets and their attributes. Not all datasets are originally meant for binary classification; dataset Abalone, for instance, has three classes—male, female and infant—so we only take male and female observations and exclude infants. The red/white wine quality datasets originally have ten quality levels, labeled from low to high. We dichotomize these levels into a low quality class (quality levels between $1$ and $5$, inclusive) and a high quality class (quality levels $6$ to $10$). All remaining datasets have two classes.

As in the simulation study, we assess performance using MCC from $50$ ten-fold cross-validation replications. For each replication, we estimate penalties $\lambda$ for all classification methods again using ten-fold cross-validations. Figure 8 shows the results.

In general, it can be seen that SVM classification performs worse on “tough” datasets with low MCC performance, such as Abalone and Haberman, probably due to poor selection of penalty parameters. On the other hand, SVM performs favorably when compared to Soft-SVM and logistic regressions in the Heart dataset, being more robust to cases where many features seem to be non-informative of classification labels. Overall, Soft-SVM regression performs comparably to the best of SVM and logistic regression even in hard datasets, as it was observed in the simulation study. Interestingly, it has a clear advantage over both SVM and logistic regressions in cases where a few features are very informative, such as in the Abalone and Australian credit datasets. It often has an edge over logistic regression in over-dispersed datasets such as Breast cancer and White wine datasets. It seems that Soft-SVM regression can better accommodate these cases via higher dispersion on moderate values of softness, i.e. $\kappa<2$, or via better use of regularization for increased robustness and generalizability.