Domain-Invariant Feature Alignment Using Variational Inference For Partial Domain Adaptation

The adversarial approach for domain adaptation usually centers around matching the source and target feature distributions through a two-player minimax game. The idea has resulted in developing a series of DANNs (domain adversarial neural networks) [19], which achieve high performance in a typical domain adaptation setup with shared label space across domains. In the proposed setup, the first player $dis(\cdot)$ is modeled as a domain discriminator and is trained to separate the $x_{s}$ from $x_{t}$. $enc(\cdot)$ poses as the second player trained to confuse the domain discriminator simultaneously by generating domain-invariant features. The encoder weights are learned by maximizing the loss of $dis(\cdot)$, whereas the discriminator weights are learned by reducing the loss of $dis(\cdot)$ to extract domain-transferable features $z$.

The overall objective of the Domain Adversarial Neural Network is realized by minimizing the following term:

$$L_{adv}=-\frac{1}{n_{s}}\sum\limits_{x_{s}\in D_{s}}w_{y_{s}}[log\hskip 5.69054ptdis(enc(x_{s}))]\\ -\frac{1}{n_{t}}\sum\limits_{x_{t}\in D_{t}}[1-log\hskip 5.69054ptdis(enc(x_{t}))]\text{, }w_{y_{s}}\in W$$

(1)

With the objective of eliminating negative transfer, we down-weight the contributions of all outlier source samples from the source label space $C_{s}-C_{t}$. This is achieved by multiplying $w_{y_{s}}$ to the log value of domain discriminator output over the source domain data ($y_{s}$ is the ground truth label of source sample $x_{s}$, $w_{y_{s}}$ represents the corresponding class weight in the class-importance weight vector $W$). The detailed process of estimating $W$ is presented in section III-B3.

As highlighted earlier, improving class conditional distribution alignment forms a salient task in pda besides attaining domain invariance. The underlying classification objective is better realized if samples with the same class labels are mapped to the same reference distribution while samples with different class labels are assigned to different distributions. We propose to address this by regularizing the latent space. In this work, the latent features are modeled as a mixture of Gaussian distributions

$$p(z|y)=\mathcal{N}(z|\mu_{y},\sigma^{2}_{y}I),\hskip 5.69054pt\forall y\in C_{s}$$

(2)

In the equation above, $I$, $\mu_{y}$, and $\sigma^{2}_{y}$ signify the identity matrix, mean, and variance parameters, respectively. Here, each $p(z|y)$ represents a reference feature distribution (prior) for a predicted class $y$ (${\mathrm{argmax}}\hskip 5.69054ptclass(\cdot)$).

The latent representations $z^{*}$ sampled from these distributions are subsequently processed for classification and data reconstruction to preserve class-discriminative and structural information in them. The reconstruction process is modelled with a Gaussian distribution $p(x|z^{*})=\mathcal{N}(x|g(z^{*}),cI)$. The mean of this distribution is represented by the output of a deterministic function $g(\cdot)$ on $z^{*}$, while the covariance matrix is defined as $c$ times the identity matrix $I$ ($c$ is a positive constant). We approximate this distribution using a decoder neural network $dec(\cdot)$ where:

$$p(x|z^{*})=\mathcal{N}(x|dec(z^{*}),I)$$

(3)

With the assumptions of latent features as samples drawn from a mixture of Gaussian distributions, we aim to estimate the posterior distribution $p(y,z^{*}|x)$. Citing the potential of variational inference for learning latent representations [16], we utilize it in our work to approximate $p(y,z^{*}|x)$ with $q(y,z^{*}|x)$. Assuming $x$ and $y$ are conditionally independent given $z^{*}$, we can expand the approximated distribution as $q(y,z^{*}|x)=q(y|z^{*})q(z^{*}|x)$. The product terms in the $r.h.s$ signify classification and encoding, respectively. To learn smooth latent features, $q(z^{*}|x)$ is formulated as a sample-wise Gaussian distribution.

$$q(z^{*}|x)=\mathcal{N}(z^{*}|\mu(enc(x)),\sigma^{2}(enc(x))I)$$

(4)

where $enc(.)$, $\mu(.)$, $\sigma^{2}(.)$ are neural networks. The classifier objective is realized by $q(y|z^{*})$, as:

$$q(y|z^{*})=class(z^{*})$$

(5)

Here, $class(.)$ represents the classifier neural network, followed by a $softmax$. A pseudo-labeling strategy using a non-parameterized classifier is employed to obtain labels $\hat{y}_{t}$ for samples $x_{t}$ in $D_{t}$. A subset $D_{t}^{\mathcal{T}}\subseteq D_{t}$ of target samples with above-average confidence predictions are filtered out for training $class(\cdot)$, (illustrated further in section III-B3). With the class information obtained for supervision, the adapted predictions are matched to the class predictions for capturing class semantic information, using cross-entropy loss $CL(\cdot||\cdot)$.

$$L_{class}=\frac{1}{n_{s}}\sum_{\mathclap{\begin{subarray}{c}(x_{s},y_{s})\in D_{s}\\ z^{*}_{s}\sim q(z|x=x_{s})\end{subarray}}}w_{y_{s}}CL(class(z^{*}_{s}),y_{s})\\ +\frac{1}{|D_{t}^{\mathcal{T}}|}\sum_{\mathclap{\begin{subarray}{c}(x_{t},\hat{y}^{p}_{t})\in D_{t}^{\mathcal{T}}\\ z^{*}_{t}\sim q(z|x=x_{t})\end{subarray}}}CL(class(z^{*}_{t}),\hat{y}^{p}_{t})$$

(6)

After establishing the reference (prior) and posterior distributions, we follow a variant of the distribution alignment strategy [15] inferred from maximizing the evidence lower bound and employ variational inference by aligning the encoded latent feature distributions $q(z^{*}|x)$ with the mixture of Gaussians $p(z^{*}|y)$. In addition, the adapted reconstructions are matched with the input sample using $l_{2}-norm$ to preserve the target information in the encoded representations. Using a similar strategy as utilized during adversarial domain alignment, we down-weight the contributions of all outlier source samples from the source label space using $W$ (presented in section III-B3). The combined objective encompassing the class distribution alignment ${L_{cda}}$ and data reconstruction $L_{recon}$, captured in $L_{var}$, is presented as:

$$L_{var}=L_{recon}+L_{cda},$$

(7)

$$\text{where, }L_{recon}=\frac{1}{n_{s}}\sum_{\mathclap{\begin{subarray}{c}(x_{s},y_{s})\in D_{s}\\ z^{*}_{s}\sim q(z|x=x_{s})\end{subarray}}}w_{y_{s}}||dec(z^{*}_{s})-x_{s}||\\ +\frac{1}{n_{t}}\sum_{\mathclap{\begin{subarray}{c}x_{t}\in D_{t}\\ z^{*}_{t}\sim q(z|x=x_{t})\end{subarray}}}||dec(z^{*}_{t})-x_{t}||,\text{ }w_{y_{s}}\in W$$

(8)

$$\text{and, }L_{cda}=\sum_{\mathclap{\begin{subarray}{c}y\in C_{s}\\ z^{*}\sim q(z|x)\\ x\in D_{s}\cup D_{t}^{\mathcal{T}}\end{subarray}}}\bigg{[}q(y|z^{*})log\bigg{(}\frac{q(z^{*}|x)}{p(z^{*}|y)}\bigg{)}\bigg{]}$$

(9)

The classification performance in a $pda$ setup is contingent upon the model’s capability to limit negative transfer from the private category samples in $s$. The extraneous information contained in these samples might confuse the classifier, resulting in an increase in classification error. Therefore, a filtration mechanism is necessary to limit their contribution to the learning process. Most of the existing solutions to the $pda$ problem [3, 14, 4, 5] attempt to address the negative transfer issue by re-weighting samples with their predicted classification probabilities or by performing a class-wise aggregation of all the target samples for estimating the shared classes. These are, however, not satisfactory strategies and may induce severe classification errors, thereby misleading the optimization process. The effect is especially drastic during the initial stages of training when the classifier is shallow-trained and does not generate predictions with high confidence.

This work utilizes a subset of target samples in the class-importance weight computation process by leveraging high-confidence target predictions. Inspired by the domain adaptation approaches presented in [18, 17, 31], we enable target domain supervision by incorporating pseudo-labels generated from a non-parametric classifier. The adopted pseudo-labeling strategy is as follows:

• •

  Step 1: For each input sample $x_{s}$ in $D_{s}$ and $x_{t}$ in $D_{t}$, we obtain their encoded latent representations $z_{s}^{*}$, $z_{t}^{*}$ respectively, using eq. 4.

• •

  Step 2: Using the encoded representation $z_{s}^{*}$ of a source sample $x_{s}$ and it’s corresponding category information $y_{s}$, we compute the cluster centers $\{\mu^{c_{s}}_{s}\}$ $\forall c_{s}\in C_{s}$, where:

  $$\begin{gathered}\mu^{c_{s}}_{s}=\frac{1}{n^{c_{s}}_{s}}\sum_{x^{c_{s}}_{s}\in D^{c_{s}}_{s}}z^{*{c_{s}}}_{s}\\ \text{where, }z^{*{c_{s}}}_{s}\sim\mathcal{N}(\mu(enc(x^{c_{s}}_{s})),\sigma^{2}(enc(x^{c_{s}}_{s}))I),\\ x^{c_{s}}_{s}\in D^{c_{s}}_{s}=\{x_{s}|(x_{s},y_{s})\in D_{s}\cap\hskip 2.84526pty_{s}=c_{s}\},\\ \text{and }n^{c_{s}}_{s}=|D^{c_{s}}_{s}|\end{gathered}$$

  (10)

• •

  Step 3: A similarity function $sim({\cdot})$ (returns a vector of size $|C_{s}|$) is computed for each $x_{t}$, that quantifies its closeness to the representative centers of each source class and is represented as:

  $$sim(x_{t})=\bigg{[}\frac{2-JS(z_{t}^{*}||\mu^{c_{s}}_{s})}{2}\bigg{]}_{c_{s}\in C_{s}}$$

  (11)

  We formulate a similarity measure using the Jensen-Shannon divergence metric ($JS(.)$) to measure the closeness between the latent-target vector and cluster centers of the latent-source representations. The final value for each entry is normalized in the range [0,1], with a higher value signifying greater similarity.

• •

  Step 4: Probability predictions are assigned for $x_{t}$ by computing softmax, ${\Phi}(.)$, over similarity values in $sim_{x_{t}}$. i.e.:

  $$\hat{p}_{t}=\hskip 5.69054pt{\Phi}(sim(x_{t}))$$

  (12)

  $$\hat{y}^{p}_{t}=\underset{c_{s}}{\mathrm{argmax}}\hskip 5.69054pt\big{[}{\Phi}(sim(x_{t}))\big{]}$$

  (13)

  In eq. 13, $\hat{y}^{p}_{t}$ represents the predicted pseudo-label for $x_{t}$, essential for classifier training on target samples. $\hat{p}_{t}$, in eq. 12, is a vector of softmax probability values, with $k^{th}$ entry representing the probability of $x_{t}$ belonging to class $k\in C_{s}$.

$\hat{p}_{t}$ is utilized for the estimation of class importance through the confidence probabilities, $max\big{(}\hat{p}_{t}\big{)}$. It gives us an idea of the degree of likeliness with which the target sample $x_{t}$ is mapped to its closest cluster center. A low value indicates that the model is still confused about mapping $x_{t}$ to a category in $C_{s}$. Using unreliable target samples with low confidence values for class-importance weight estimation might thwart classification by misleading the model optimization task. Citing this, we devise a voting strategy to compute the class-importance weight vector $W$ (of size $|C_{s}|$), where a subset of the target samples (ones with high confidence predictions) are allowed to participate. This is mathematically represented as:

$$D_{t}^{\mathcal{T}}=\{(x_{t},\hat{y}^{p}_{t})\hskip 2.84526pt|\hskip 2.84526ptx_{t}\in D_{t},\hskip 2.84526ptmax\big{(}\hat{p}_{t}\big{)}\geq\mathcal{T}\}$$

(14)

$$\begin{gathered}W=\frac{W^{\prime}}{max(W^{\prime})}\\ \text{where, }W^{\prime}=\frac{1}{|D_{t}^{\mathcal{T}}|}\sum_{x_{t}\in D_{t}^{\mathcal{T}}}\hat{p}_{t}\\ \end{gathered}$$

(15)

For a source sample $(x_{s},y_{s})\in D_{s}$, $w_{y_{s}}$ is represented as the corresponding class weight in the class-importance weight vector $W$.

The threshold parameter $\mathcal{T}$ is computed over the predicted outputs of the non-parametric classifier ${\Phi}(sim(\cdot))$ ($\Phi$ represents the softmax function) and measures the average probability of source domain samples belonging to the ground-truth class, i.e.:

$$\mathcal{T}=\frac{1}{n_{s}}\sum_{x_{s}\in D_{s}}max\big{(}{\Phi}(sim(x_{s}))\big{)}$$

(16)

In the early phases of a classification process, several side effects arise from adapting from one domain to another. They range from challenges with knowledge transfer caused by significant domain shifts to inducing uncertainty in the classifier. Entropy minimization on the predicted target samples forms a promising candidate to eliminate such adverse effects. In this work, we use an entropy minimization loss, which is described as:

$$L_{em}=-\frac{1}{n_{t}}\sum\limits_{x_{t}\in D_{t}}\sum\limits_{c_{s}\in C_{s}}\hat{y}^{c_{s}}_{t}log(\hat{y}^{c_{s}}_{t})$$

(17)

Where, $\hat{y}^{c_{s}}_{t}$ represents the probability of $x_{t}$ belonging to class $c_{s}$, as predicted by the classifier $class(\cdot)$.

To sum up, the overall objective function is modeled as follows:

$$L=L_{class}+\alpha L_{adv}+\beta L_{var}+\gamma L_{em},$$

(18)

with $\alpha,\beta$, and $\gamma$ as the trade-off hyper-parameters.