Autonomous Cross Domain Adaptation under Extreme Label Scarcity

LEOPARD is structured as a deep clustering network developed with a feature extraction layer extracting natural features $Z$ from raw input features $x$ by means of a mapping function $F_{W_{f}}(.)$ where $W_{f}$ stands for parameters of the feature extractor. The extracted features $Z$ are passed to a fully connected layer formed as a stacked autoencoder (SAE) with a tied-weight constraint. That is, the decoder parameters are the inverse mapping of the encoder parameters. The natural features $Z\in\Re^{u^{\prime}}$ are projected to a low dimensional latent space $h^{l}\in\Re^{R_{l}}$ where $u^{\prime},R_{l}$ are respectively the number of natural features and hidden nodes at the $l-th$ layer $R_{l}<<u^{\prime}$. The decoding and encoding mechanisms are expressed:

where $W_{enc}^{l}\in\Re^{R_{l}\times u_{l}},b^{l}\in\Re^{R_{l}}$ stand for the connective weights and biases of the $l-th$ layer of the encoder respectively while $W_{dec}^{l}\in\Re^{u_{l}\times R_{l}},c^{l}\in\Re^{u_{l}}$ denote the connective weights and biases of the $l-th$ layer of the decoder respectively. The tied-weight constraint $W_{dec}^{l}=(W_{enc}^{l})^{T}$ functions as a regularization mechanism preventing the issue of overfitting.

The clustering mechanism is carried out in each deep embedding space, each latent space. That is, it takes place in every hidden layer of SAE $h(.)^{l}$ creating different representations of data samples. The inference mechanism is performed by first calculating the similarity degree of a data sample and a hidden cluster[17]:

where $C_{j}^{l},h^{l}$ are the centroid of the $j-th$ cluster of the $l-th$ layer and the latent representation of a data sample $x$ of the $l-th$ layer while $Clus^{l}$ is the number of cluster created in the $l-th$ latent space, i.e., the $l-th$ layer of SAE. $\lambda=1$ is chosen here. The student t-distribution is adopted to model the similarity degree and $\phi_{j}^{l}$ is also regarded as the cluster posterior probability $P(C_{j}^{l}|X)$ [18] where $P(C_{j}|X)=1$ presents the case of perfect match between $h^{l}$ and $C_{j}^{l}$. The similarity degree $\phi_{j}^{l}$ is aggregated across $N_{m}$ prerecorded samples having true class labels $B_{0}^{S}=\{x_{i}^{S},y_{i}^{S}\}_{i=1}^{N_{m}}$. This operation produces the cluster’s allegiance [19] measuring cluster’s tendencies to a particular class. Suppose that $N_{o}$ stands for the number of prerecorded samples having the $o-th$ class as their labels, the cluster allegiance $Ale_{j,o}^{l}$ is calculated:

where $\phi_{j,o}^{n,l}$ measures the similarity degree of the cluster $C_{j}^{l}$ and the $n-th$ prerecorded sample $h_{o}^{l}$ falling into the $o-th$ class. (5) pinpoints the neighborhood degree of the $j-th$ cluster to the $o-th$ class and implies that an unclean cluster, occupied by data samples of mixed classes, possesses low cluster allegiance. The winner-takes-all principle $win=\arg\max_{j=1,...,Clus^{l}}\phi_{j}^{l}$ is adopted here, where a data sample is associated to the nearest cluster. The local score of the $l-th$ layer is defined as the allegiance of the winning cluster $Score^{l}=Ale_{win}^{l}$.The predicted class label $\hat{Y}$ is determined as a class label maximizing its global score. The global score is calculated as the summation of a local score across $L$ layers:

where the majority voting approach is implemented here. It is evident that LEOPARD merely benefits from the labelled prerecorded samples of the source stream $B_{0}^{S}$ to associate a cluster to a specific class. No label at all from both streams is solicited in the streaming phase where it confirms its applicability in the extreme label scarcity environments. Fig. 2 visualizes the network structure of LEOPARD. It is perceived that the clustering process occurs in every hidden layer of LEOPARD thus producing its local outputs. The final predicted class label is aggregated across all layers making use of a summation operation. $R_{l},L,Clus^{l}$ are self-evolved in respect to varying distributions.

The concept of adversarial domain adaptation can be implemented by deploying a domain classifier $\zeta_{W_{DC}}(F_{W_{F}}(.))$ working along with the feature extractor $F_{W_{f}}(.)$ and a classifier $\xi_{W_{C}}(F_{W_{F}}(.))$. The domain classifier predicts the origin of data samples whether they are generated by the source domain $D_{S}$ or the target domain $D_{T}$ while the classifier generates the final output of a network. The domain reversal layer is implemented in updating the feature extractor such that indistinguishable features of source and target domains are induced. That is, the overall loss function is written as follows:

where $L_{\xi,\zeta}(.)$ is implemented as the cross entropy loss function and $d_{n}$ is the domain identity, i.e., 1 for the source domain and 0 for the target domain. From (9), the gradient reversal layer inserts a negative constant confusing the domain classifier, i.e., generating indistinguishable samples. The parameter learning process is formulated as follows:

where the feature extractor is trained to produce similar features of the two domains seen in the negative sign of the gradient thereby leading to the domain invariant network.
Loss Function: the parameter learning strategy of LEOPARD is constructed using a joint loss function comprising two modules: clustering loss and cross-domain adaptation loss. The underlying goal is to produce domain-invariant parameters as well as clustering-friendly latent spaces such that the online cross domain adaptation can be solved under extreme label scarcity. The overall cost function is formalized as follows:

where $L_{cluster},L_{cd}$ respectively denote the clustering loss and the cross-domain adaptation loss while $\alpha_{1}$ is a trade-off constant controlling the influence of the cross-domain adaptation loss. It is an unconstrained optimization problem which can be optimized using the stochastic gradient descent approach with no epoch or epoch per batch to assure scalability in streaming environments. That is, a number of iteration is done per batch. A data batch is discarded once iterations across a number of epoch is completed to allow bounded complexity. The negative sign in (13) follows the gradient reversal strategy generating similar features across two domains. In other words, the gradient of clustering loss and the gradient of cross domain loss, the domain classifier loss, is subtracted [3].
Clustering-Friendly Latent Space: the clustering loss aims to achieve the clustering-friendly latent space via simultaneous feature learning and clustering. The clustering loss is formulated as the reconstruction loss and the KL divergence loss minimizing probabilistic distance of the latent space and the auxiliary target distribution [17]:

where $\Phi^{l}$ is the auxiliary target distribution of the $l-th$ latent space and $\alpha_{2}$ is a regularization constant controlling the strength of the KL divergence loss. $\phi^{l}$ is the similarity degree of the current sample to existing clusters. $L_{\xi}(.)$ is the reconstruction loss formed as the mean square error (MSE) loss function. It also performs nonlinear dimension reduction preventing the trivial solutions often happening in the case of linear mapping. It guarantees a data sample to be mapped back to its original representation. The key difference between the two loss functions lies in the adaptation mechanism in which $L_{1}$ is solved in the end-to-end fashion while $L_{2}$ is carried out in the layer-wise fashion.

The last term also known as the KL divergence loss minimizes the discrepancy of the distribution of a current data batch calculated via (4) and the auxiliary target distribution $KL(\phi^{l}|\Phi^{l})=\sum_{i}\sum_{j}\phi_{i,j}^{l}\log{\frac{\phi_{i,j}^{l}}{\Phi_{i,j}^{l}}}$. The auxiliary target distribution should satisfy three requirements [17]: 1) improve prediction; 2) emphasizes samples of high confidence; 3) normalize loss contribution of each cluster to avoid creations of large clusters. We adopt the same auxiliary distribution as in [17] where $\Phi_{i,j}^{l}$ is raised to the second power and normalized by frequency per cluster:

where $\zeta_{j}=\sum_{i=1}^{N}\phi_{i,j}^{l}$ is the frequency of a cluster. This strategy is understood as the soft-cluster assignment [17] where all clusters are updated and differs from the hard-cluster assignment only tuning the winning cluster. The clustering mechanism is hard to conduct in the high-dimensional space [21] thus calling for feature learning steps to be committed simultaneously. The clustering process takes place in every latent space $h(.)^{l}$ set as the common feature spaces between the source and target domain. That is, (14) is executed using samples of both source and target streams. This process also functions as an implicit domain adaptation strategy since the minimization of reconstruction loss across two streams with shared parameters ends up with an overlapping region of both domains [6]. The optimization procedure takes place simultaneously where the network parameters and the cluster parameters are adjusted concurrently with the SGD method.
Domain-Invariant Network: LEOPARD consists of three sub-modules: feature extractor $F(.)$, classifier $\xi(.)$ and domain classifier $\zeta(.)$ to achieve a domain invariant property as depicted in Fig. 3. The feature extractor is parameterized by $W_{F}$ and the classifier formed as the deep clustering module is parameterized by $W_{C}\in\{W_{enc}^{l},W_{dec}^{l},C^{l}\}$ while the domain classifier formed as a single hidden layer network is parameterized by $W_{DC}$. The feature extractor and the domain classifier play a minimax game via the gradient reversal layer where the feature extractor is trained to fool the domain classifier via production of similar features of source and target streams while the domain classifier is trained to identify the origin of data samples. The cross domain adaptation loss is thus formulated as the domain classifier loss as follows:

where $d_{n}$ stands for the origin of data samples, i.e., $1$ for the source stream and $0$ for the target stream. The domain classifier is tasked to solve a binary classification problem where $L_{\zeta}(.)$ is set as the cross entropy loss function. This leads to similar parameter learning processes for feature extractor, domain classifier and classifier respectively as in (10) - (12) except the presence of the clustering loss instead of the cross-entropy loss as defined in (14): $W_{f}=W_{f}-\mu(\frac{\partial L_{cluster}}{\partial W_{f}}-\alpha_{1}\frac{\partial L_{cd}}{\partial W_{f}});W_{C}=W_{C}-\mu\frac{\partial L_{cluster}}{\partial W_{C}};W_{DC}=W_{DC}-\mu\alpha_{1}\frac{\partial L_{cd}}{\partial W_{DC}}$ where $\mu$ denotes the learning rate. Note that the gradient reversal layer has no parameters and simply alters the sign of the gradients allowing maximization process to be carried out via the stochastic gradient descent approach. This only applies to the feature extractor as illustrated in Fig. 3.

The hidden unit growing and pruning steps are signalled by the statistical process control (SPC) approach [23] commonly used for anomaly detection tasks. The SPC method is applied here to detect the high bias or high variance condition and written as follows:

The SPC method is generalized here using $k_{2}=1.3\exp{(-Bias^{2})}+0.7$ and $k_{3}=1.3\exp{(-Var^{2})}+0.7$. This modification leads to dynamic confidence levels enabling for flexible growing and pruning phases. That is, the node growing process is likely performed in the case of a high bias while being strict in the case of a low bias. The same case also applies for the node pruning mechanism. $\mu_{bias}^{min,l}$ and $\sigma_{bias}^{min,l}$ are reset if the growing condition (19) is satisfied. On the other hand, if the pruning condition is met, $\mu_{var}^{min,l}$ and $\sigma_{var}^{min,l}$ are reset. The initialization of a new node is carried out using the Xavier initialization strategy. The least contributing node having the least statistical contribution is subject to the pruning step if (20) is observed. Since LEOPARD is constructed under a different-depth structure where every layer performs its own clustering mechanism and produces its local output, the growing and pruning steps are independently undertaken per layer. Furthermore, this mechanism occurs for both source and target streams to anticipate the asynchronous drift problem where the network structure is shared across two domains.

The classifier of LEOPARD is fitted with the hidden layer growing mechanism where it expands the network depth based on the drift detection mechanism [24]. The drift detection mechanism is designed from the concept of Hoeffding’s bound and analyzes the dynamic of latent features $Z$ to identify the change of marginal distribution. Note that no labelled samples are offered for model updates and the drift detection approach is executed for both source and target streams. The addition of a network layer is desired in practise because it is capable of substantiating network capacity significantly thus enhancing model’s generalization. The drift detection procedure starts by finding the cutting point, a point where population mean increases. A cutting point is declared by the following condition.

where $P\in\Re^{2N}$ is a data matrix containing two consecutive data batches $[B_{k-1},B_{k}]$, i.e., previous and current data batches while $Q\in\Re^{cut}$ is a data matrix with $cut$ as the hypothetical cutting point of interest, $cut<2N$. Two data batches are applied here to increase the sensitivity of cutting point identification because latent features are relatively stable compared to the original input space. The hypothetical cutting point is arranged as $cut=[25\%,50\%,75\%]\times 2N$ instead of every point to avoid false alarm. $\hat{P},\hat{Q}$ denote the statistics of data matrices $P,Q$. $\epsilon_{P,Q}$ stand for the error bound derived from the concept of Hoeffding’s bound as follows:

where $\alpha_{x}$ is the significance level being inversely proportional to the confidence level $1-\alpha_{x}$ while $size$ refers to the size of the data matrix of interest $P,Q$.

Once eliciting the cutting point of interest $cut$, a data matrix $R\in\Re^{2N-cut}$ is constructed. A drift is signalled if $|\hat{R}-\hat{Q}|\geq\epsilon_{D}$. Beside the drift condition, a warning condition is set and pinpoints a case where a drift needs to be confirmed by the next data batch. That is, $\epsilon_{W}\leq|\hat{R}-\hat{Q}|\leq\epsilon_{D}$ where $\alpha_{W}<\alpha_{D}$. The error bounds $\epsilon_{D,W}$ are defined as follows:

where $[a,b]$ denotes the range of the data matrix $P$. A new layer is created if a concept drift is found. That is, the number of nodes is set as the half of the network width of the previous layer $l-1$. This step enables the nonlinear feature reduction and avoids an over-complete network. The domain classifier and the feature extractor have a fixed structure because the structural learning of the classifier suffices to address the asynchronous drift problem.

Learning policy of LEOPARD is visualized in Fig. 3 and Algorithm 1 where LEOPARD is driven by the feature extractor, the classifier and the domain classifier. The forward pass procedure is done by feeding raw input attributes $x_{S,T}$ to the feature extractor $F(.)$ leading to latent input features $Z_{S,T}$. The latent features are passed to the classifier $\xi(.)$ implemented as the SAE and the clustering module. Note that the clustering module exists in every layer of SAE producing its own local output $Score^{l}$ where the majority voting is performed to generate a final predicted output. The learning process starts with a warm-up phase using $N_{init}$ unlabelled samples iterated across $E$ number of epochs to avoid the cold start problem. This process only involves the reconstruction loss $L_{\xi}(x_{S,T},\hat{x}_{S,T})$ and $L_{\xi}(h^{l}_{S,T},\hat{h}^{l}_{S,T})$ affecting only network parameters $W_{F}$ and $W_{enc}^{l},W_{dec}^{l}$. The main training loop is executed by minimizing $L_{cluster}(.)$ and is applied to $W_{F},W_{enc}^{l},W_{dec}^{l},C^{l}$. Minimization of clustering loss across the two domains can be also seen as the domain adaptation strategy because it leads to an overlapping region of source domain and target domain to be created, i.e., both the source stream and the target stream are used under shared parameters. The adversarial domain adaptation is carried out by minimizing $L_{cd}(.)$ afterward where the domain classifier $\zeta_{W_{DC}}(.)$ is updated as well as the feature extractor $F_{W_{f}}(.)$ using the cross domain loss. The gradient reversal strategy is adopted when adjusting the feature extractor thus converting the minimization problem to the maximization problem and in turn resulting in indistinguishable features of the source stream and the target stream. The cross domain adaptation strategy makes possible for the source streams and the target streams following different distributions to be mapped similarly, i.e., the covariate shift is addressed.

The structural learning process occurs in both the initialization phase and the main training phase in which it includes the cluster growing process, the hidden node growing and pruning processes and the hidden layer growing process. As with the warm-up phase, the initialization phase using $N_{init}$ prerecorded samples over $E$ epochs is implemented if a new layer is created. It is obvious that LEOPARD does not exploit any labelled samples for model updates except for labelled samples to be used to calculate the cluster allegiance (5). The structural learning mechanism addresses the issue of asynchronous drifts across both streams.

The numerical study is carried out using the prequential test-then-train protocol as per [23], A model is tested first before updating it with the same data stream. The numerical evaluation is independently undertaken per-batch where the numerical result is averaged across all batches. Our simulation is repeated 5 times to guarantee the consistency of numerical results where the final numerical results are averaged over 5 independent runs. The asynchronous drift problem is induced by applying the scaling hyper-plane strategy [27, 28] where a data stream is scaled to $x_{i}=\frac{d_{z}\times x_{i}}{||x||}$. $d_{z}$ is a randomly generated concept drift vector where $z$ is the number of concept drifts in the stream: $z=1$ for every source stream and $z=1$ for every target data stream. A fixed random seed is selected in setting $d_{z}$ to assure fair comparison. In realm of MN$\leftrightarrow$US, the concept drifts occurs at $k=35$ for source stream and $k=36$ for target stream whereas the concept drift takes place at $k=5$ for source stream and $k=6$ for target stream in the amazon@X and Office 31 problems. These configurations assure the asynchronous drift to be presented.

LEOPARD is compared with five algorithms: autonomous deep clustering network (ADCN) [29], deep clustering network (DCN) [30], autoencoder followed by K-Means (AE+KMeans), deep embedding clustering (DEC) [17] and domain adversarial neural networks (DANN) [3]. ADCN is a self-evolving deep clustering network where hidden clusters, nodes and layers are grown and pruned dynamically. The loss function is formulated with a combination of a clustering loss and a reconstruction loss. ADCN is not equipped by a specific domain adaptation loss function while it applies the hard cluster assignment approach as with [30], i.e., The $L_{2}$ distance loss of the winning cluster and the latent sample is put forward. DCN adopts a fixed network structure where the clustering mechanism only takes place at the bottleneck layer. It applies the same loss function as ADCN. AE+KMeans differs from DCN where the clustering mechanism is carried out after the training process. It does not utilize any clustering loss. DEC adopts the soft-assignment approach as with LEOPARD except that it relies on a static network structure and suffers from the absence of any domain adaptation loss. The reconstruction loss in the baseline algorithms are perceived as a domain adaptation procedure because they are carried out for both source and target streams under shared network parameters. DANN utilizes the adversarial domain adaptation as per LEOPARD without any clustering mechanism.

All of them work under the extreme label scarcity condition as with LEOPARD where access of true class labels is only provided for the prerecorded samples of the source stream while no label is offered during the process runs for both the source stream and the target stream. Comparison with ADCN is done by executing their published codes to assure fair comparisons. We utilize our own implementations of DCN, AE+KMeans, DEC and DANN.

The learning rate and momentum of LEOPARD are allocated as 0.01 and 0.95 while the regularization constant of the clustering loss $\alpha_{2}$ is set as 1 and the tradeoff constant of the cross-domain loss $\alpha_{1}$ is set as 0.1. LEOPARD also depends on labelled prerecorded samples $B_{0}^{S}$ of the source stream set as $10\%$ of source samples proportionally taken from each class $N_{m}=10\%N_{S}$. That is, each class contributes the same number of samples. The number of initial epochs are set as $E=100$ (amazon@X), $E=50$ (MN$\leftrightarrow$US), and $E=500$ (D$\leftrightarrow$W) respectively. The initialization phase is carried out using labelled prerecorded samples of the source stream. The parameters of the drift detector $\alpha_{x},\alpha_{D},\alpha_{W}$ are selected respectively as $0.001,0.001,0.005$. For amazon@X problems, LEOPARD runs in the one-pass learning procedure whereas for MN$\leftrightarrow$US experiments, the training process of the clustering loss adopts the epoch per batch strategy with $10$ (MN$\rightarrow$US, W$\leftrightarrow$D) epochs and $5$ epochs (US$\rightarrow$MN) respectively. The epoch per batch strategy satisfy the online learning requirement because a data batch is discarded after training over predetermined epochs. The same setting is also applied to the baseline algorithms assuring fair comparisons.

For MN$\leftrightarrow$US problem, the feature extractor is formed as convolutional neural networks. The encoder part is constructed as 2 convolutional layers using 16 and 4 filters respectively while having the max pooling layer in between. The decoder part is built upon two transposed convolutional layers with 4 and 16 filters respectively. For amazon@X sentiment analysis problems, the multi-layer perceptron feature extractor is put forward with two hidden layers where the number of nodes is fixed as $300$ and $100$. For the office31 problem, ResNet34 is applied as feature extractors. The initial nodes of fully connected layer are simply assigned as $96$ for the MNIST$\leftrightarrow$USPS problem, $30$ for amazon@X sentiment analysis problems and $500$ for D$\leftrightarrow$W. The ReLU activation function is applied for the intermediate layers while the decoder output utilizes the sigmoid activation function producing normalized reconstructed output. The network structures of baseline algorithms are set similarly to ensure fair comparison. Further details of our numerical studies are explained in the LEOPARD’s codes shared in https://github.com/wengweng001/LEOPARD.git

These parameters are fixed throughout all study cases to guarantee non ad-hoc performance of LEOPARD. The hyper-parameters of the baselines are selected as per the guidelines of their publications and hand-tuned if their performances are surprisingly compromised. The hyper-parameters of all consolidated algorithms are listed in the supplemental document.

From Table II, it is seen that LEOPARD outperforms other algorithms in 15 of 24 cases with noticeable margins. This aspect portrays the efficacy of the adversarial domain adaptation approach and the soft-cluster assignment mechanism where these two modules are absent in the baseline algorithms, i.e., ADCN, DCN, AE+KMeans adopt the hard-cluster assignment strategy and suffer from the absence of the adversarial domain adaptation approach. This finding also confirms the advantage of the adversarial domain adaptation over the feature reconstruction strategy with shared parameters across the two streams implemented in all baselines. The soft-cluster assignment approach where the cluster and network parameters are simultaneously optimized via the SGD method performs better than the hard cluster assignment strategy. It is demonstrated by the fact that LEOPARD beats ADCN in significant numbers of cases. There is no significant performance difference between AE+KMEANS and DEC. On the other hand, the importance of the structural learning component in handling data streams is clearly portrayed here where LEOPARD and ADCN are superior compared to other algorithms having static structures. Such mechanism allows timely reactions to the asynchronous drift problem across the source stream and the target stream. The performance of DANN implemented under conventional neural network structures is far inferior to LEOPARD under extreme label scarcity condition. This fact confirms the advantage of clustering approach compared to the conventional neural network structure to reduce label’s dependencies. The statistical test is undertaken using the t test $(P<0.05)$ confirming the advantage of LEOPARD where it beats other algorithms with statistically significant gaps in 13 of 24 cases.

This section studies the effect of LEOPARD’s learning modules by analyzing its performance when deactivating a particular learning module. LEOPARD is configured into four models: (A) the structural learning method is switched off leaving LEOPARD to have a static network structure. In addition, the parameter learning strategy is done with the absence of the cross domain adaptation loss and the KL divergence loss. In short, LEOPARD is driven by the reconstruction loss only; (B) the structural learning strategy of LEOPARD is deactivated while the parameter learning step utilizes both the cross domain adaptation loss and the KL divergence loss; (C) the structural learning mechanism of LEOPARD is activated but with the absence of the KL divergence loss and the cross domain adaptation loss; (D) BERT is applied as a feature extractor in lieu of a word2vec model. Our ablation study is carried out using four study cases: AM1$\rightarrow$AM3, AM1$\rightarrow$AM4, AM5$\rightarrow$AM1 and AM5$\rightarrow$AM4.

From table III, LEOPARD suffers from major performance degradation for all configurations (A)-(D) confirming the efficacy of its current version. For AM5$\rightarrow$AM1, configuration (A) where both the structural learning strategy, the KL divergence loss and the cross domain adaptation loss are absent produces poor results. The performance improves slightly with the activation of the KL divergence loss and the cross domain adaptation loss as per configuration (B). Although the KL divergence loss and the cross domain adaptation loss are deactivated, the structural learning mechanism improves the accuracy as per configuration (C) but is not yet on par to the LEOPARD. An interesting observation presents in the case of AM5$\rightarrow$AM4 where configuration (C) produces poor results. The performance betters in configuration (A) and (B) without the structural learning mechanism. Nevertheless, configuration (A)-(C) are not comparable to the LEOPARD where all modules are engaged. We note that the size of source stream is much less than the size of target stream in the AM5$\rightarrow$AM4 case. This leads to very few labelled samples provided in the warm-up phase. For AM1$\rightarrow$AM4, the absence of structural evolution, configuration (B), drops the LEOPARD’s accuracy slightly. More severe degradation than configuration (B) occurs in configuration (A) and configuration (C) where the KL divergence loss and the cross domain adaptation loss are deactivated. The same pattern is demonstrated in the case of AM1$\rightarrow$AM3. This finding clearly confirms the advantage of the KL divergence loss and the cross domain adaptation loss for LEOPARD. In addition, the structural evolution boosts the performance of LEOPARD especially in the presence of drifts. The use of BERT as feature extractor as shown in Configuration (D) worsens predictive performances of LEOPARD significantly. This finding confirms the compatibility of the word2vec model over the BERT model as a feature extractor of LEOPARD most likely due to the absence of self-attention mechanism or recurrent connection in LEOPARD.

Fig. 4 illustrates the t-SNE plots of LEOPARD [18] for USPS$\rightarrow$MNIST case on the target stream before and after the training process. It is observed that there does not exist any cluster structures initially in this problem but such cluster structures are clearly present after the training process showing the effectiveness of (14). These facts confirm that LEOPARD does not call for the existence of any cluster structures beforehand and the clustering loss $L_{cluster}$ is capable of establishing the clustering-friendly latent space.

This paper has successfully developed an algorithmic solution of multistrean classification problems under extreme label shortages, LEOPARD. That is, given two different but related streaming processes, LEOPARD properly functions with few prerecorded samples of the source domain and the absence of any labels when the streaming processes run. This benefit goes one step ahead of existing multistream classifiers or unsupervised domain adaptation methods calling for fully labelled source streams. Nonetheless, the problem of multistream classification remain at the infant stages leaving several open issues for future works.

The problem of open set domain adaptation presents a case where the source and target domains do not share the same target classes [31]. Such setting is also seen as a way to reduce the labelling cost and is beneficial in realm of multistream classifications. [31] proposes a feature transformation strategy associating target classes of target domain to those of source domain. [32] puts forward the concept of openness and unknown classes in the open set domain adaptation problem. The theoretical bound of the open set domain adaptation is derived in [33]. The application of theoretical bound for deep learning is demontrated in [34]. These approaches are limited to the offline case calling for extensions for the multistream classification setting. Gradual Domain Adaptation [35] is highly relevant to the multistream classification context because the multistream classification problem still considers different but related streams and constant discrepancies. Also, the asynchronous drifts usually appear suddenly. (1) assumes a small and fixed combined risk and is unrealistic because the combined risk may increase during the training process [36]. This issue is still ignored in the multistream classification problems. Few-shot Hypothesis adaptation [37] is another interesting direction for the multistream classification topic and extends the few-shot domain adaptation problem without any source domain data.