Rethinking Multi-domain Generalization with A General Learning Objective

Learning of $\mathcal{D}$-independent conditional features under prior. The generalization alignment upper bound (PUB), a novel GJSD variational upper bound that is tied to domain generalization alignment, is derived based on the generalized Jensen-Shannon divergence (GJSD) [31].

Our method addresses the standard scenario in which the weights are evenly distributed across domains: $w_{1}=...=w_{N}=1/N$. To achieve $\phi({\mathbf{X}}),\psi({\mathbf{Y}})\perp\!\!\!\perp\mathcal{D}$, minimizing domain gap between $P({\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})})$ can be converted to minimizing GJSD across all domains:

$$\begin{split}&\min\nolimits_{\phi,\psi}GJSD(\{P({\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})})\}_{n=1}^{N})\\ \equiv&\min\nolimits_{\phi,\psi}H(P({\phi({\mathbf{X}}),\psi({\mathbf{Y}})}\mid\mathcal{D}))\\ &-\mathbb{E}[H(P({\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})}))].\end{split}$$

(5)

We further involve a prior knowledge distribution $\mathcal{O}$ under the consideration of a variational density model class $\mathcal{Q}$. Drawing upon [9], we have a variational upper bound:

$$\begin{split}&GJSD(\{P({\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})})\}_{n=1}^{N})\\ \leq&H_{c}(\mathbb{E}[P({\phi({\mathbf{X}}),\psi({\mathbf{Y}})]\mid\mathcal{D}}),\mathcal{O})-a,\end{split}$$

(6)

where $a\triangleq\sum\nolimits_{n=1}^{N}H(P({\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})}))$ is constant w.r.t $\phi,\psi$, hence ignored during optimization. The novel PUB is derived from Eq. LABEL:eq:bound2, is:

$$\begin{split}&\min\nolimits_{\phi,\psi}PUB(\{P({\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})})\}_{n=1}^{N})\\ \triangleq&\min\nolimits_{\phi,\psi}{P({\phi({\mathbf{X}}),\psi({\mathbf{Y}})}\mid\mathcal{D}))}\\ &+{D_{\mathrm{KL}}(P({\phi({\mathbf{X}}),\psi({\mathbf{Y}})})\|\mathcal{O}))}-{a}.\end{split}$$

(7)

Minimizing PUB is the proposed objective for GAim1 and GReg1. This implies that methods like MIRO, solely minimizing GReg1, might result in substantial suboptimality, leaving the domain gap unresolved. We discuss two situations of $\mathcal{O}$ in Section 4.

Suppressing invalid causality. The relaxation of $P({\mathbf{Y}}|\mathcal{D})$ static assumption may lead to unexpected causality while learning the invariance. GAim1 is reformed as:

$$\begin{split}\text{{GAim1}}=&H(P(\phi({\mathbf{X}})\!\mid\!\mathcal{D}))\!\!+\!\!H(P(\psi({\mathbf{Y}})\!\mid\!\phi({\mathbf{X}}),\mathcal{D}))\\ =&H(P(\psi({\mathbf{Y}})\!\mid\!\mathcal{D}))\!\!+\!\!H(P(\phi({\mathbf{X}})\!\mid\!\psi({\mathbf{Y}}),\mathcal{D})),\end{split}$$

(8)

where minimizing GAim1 with relaxation of $P({\mathbf{Y}}|\mathcal{D})$ static assumption may lead to $\psi({\mathbf{Y}})\!\to\!\phi({\mathbf{X}})$ since the term $H(P(\psi({\mathbf{Y}})|\phi({\mathbf{X}}),\mathcal{D}))$ and $\phi({\mathbf{X}})\!\to\!\psi({\mathbf{Y}})$ since the term $H(P(\phi({\mathbf{X}})|\psi({\mathbf{Y}}),\mathcal{D}))$.

Figure 2 graphically demonstrates the causal diagram under this scenario. Since the prediction relationship from $\phi({\mathbf{X}})\to{\mathbf{Y}}$ and the casual path $\psi({\mathbf{Y}})\to{\mathbf{Y}}$, $\phi({\mathbf{X}})\to\psi({\mathbf{Y}})$ should be preserved for prediction. However, the casual path $\phi({\mathbf{X}})\to\psi({\mathbf{Y}})$ may compromise the predication from $\phi({\mathbf{X}})\to{\mathbf{Y}}$ when $\psi({\mathbf{Y}})$ is unknown during the inference, leading to generalization degradation. This unveils that the invalid causality from $\psi({\mathbf{X}})\to\phi({\mathbf{Y}})$ that may happen during the learning invariance needs to be suppressed as $\max_{\phi,\psi}H(P(\phi({\mathbf{X}})|\psi({\mathbf{Y}})),\mathcal{D})$ while $\min_{\phi,\psi}H(P(\psi({\mathbf{Y}})|\phi({\mathbf{X}})),\mathcal{D})$, which can be simplified as:

$$\min_{\phi,\psi}H(P(\phi({\mathbf{X}})))-H(P(\phi({\mathbf{X}}))|P(\psi({\mathbf{Y}}))),$$

(9)

where is GReg2. See more mathematical details in Supplementary 7. Our experiments also unveil the phenomenon of invalid causality within invariant feature learning, where suppressing it could improve generalizability. The investigation of previous objectives also discloses that, in addressing the varying $P({\mathbf{Y}}|\mathcal{D})$, constructs akin to GReg2 are often implicitly included (see Table 1), though their efficacy was not explicitly stated. Moreover, their efficacy may be compromised due to the lack of $\psi$ and other objective terms.

Then, we assume that $\phi(X),\psi(Y)$ in the RKSH follow Multivariate Gaussian-like Distributions which are denoted as $\mathcal{N}(\phi({\mathbf{X}});\mu_{\mathbf{X}},\Sigma_{{\mathbf{X}}{\mathbf{X}}}),\mathcal{N}(\psi({\mathbf{Y}});\mu_{\mathbf{Y}},\Sigma_{{\mathbf{Y}}{\mathbf{Y}}})$. $P(\phi({\mathbf{X}})|\psi({\mathbf{Y}}))$ follows $\mathcal{N}(\phi({\mathbf{X}})|\psi({\mathbf{Y}});\mu_{{\mathbf{X}}|{\mathbf{Y}}},\Sigma_{{\mathbf{X}}{\mathbf{X}}|{\mathbf{Y}}})$. GReg2 can be simplified as:

$$\begin{split}&H(\mathcal{N}(\phi({\mathbf{X}});\mu_{\mathbf{X}},\Sigma_{{\mathbf{X}}{\mathbf{X}}}))\\ &-H(\mathcal{N}(\phi({\mathbf{X}})\mid\psi({\mathbf{Y}});\mu_{{\mathbf{X}}|{\mathbf{Y}}},\Sigma_{{\mathbf{X}}{\mathbf{X}}|{\mathbf{Y}}}))\\ =&\frac{1}{2}\ln(\frac{|\Sigma_{{\mathbf{X}}{\mathbf{X}}}|}{|\Sigma_{{\mathbf{X}}{\mathbf{X}}|{\mathbf{Y}}}|})\geq 0,\end{split}$$

(10)

where the inequality stands owing to the Condition Reducing Entropy. This implies $H(\mathcal{N}(\phi({\mathbf{X}});\mu_{\mathbf{X}},\Sigma_{{\mathbf{X}}{\mathbf{X}}}))\!\!\geq\!\!H(\mathcal{N}(\phi({\mathbf{X}})\mid\psi({\mathbf{Y}});\mu_{{\mathbf{X}}|{\mathbf{Y}}},\Sigma_{{\mathbf{X}}{\mathbf{X}}|{\mathbf{Y}}}))$, deduced from $|\Sigma_{{\mathbf{X}}{\mathbf{X}}}|\!\!\geq\!\!|\Sigma_{{\mathbf{X}}{\mathbf{X}}|{\mathbf{Y}}}|\!\!\geq\!\!0$, considering they are positive semi-definite. Distinct from the [52, 18] which decompose causal effects through extra networks, our method is based on transfer entropy (TE) by ensuring $TE(\phi({\mathbf{X}})\!\!\!\to\!\!\!\psi(Y))\!\!\!\geq\!\!\!TE(\phi({\mathbf{X}})\!\!\!\to\!\!\!\psi(Y))$, i.e., $H(\phi({\mathbf{X}})|\psi(Y))\!\!\!\geq\!\!\!H(\psi(Y)|\phi({\mathbf{X}})).$ Thus, minimization of Eq. LABEL:eq:cfs_detial occurs iff $|\Sigma_{{\mathbf{X}}{\mathbf{X}}}|=|\Sigma_{{\mathbf{X}}{\mathbf{X}}|{\mathbf{Y}}}|$, reformulating the task as ​​$min_{\phi,\psi}|\Sigma_{{\mathbf{X}}{\mathbf{X}}}|\!\!\!-\!\!\!|\Sigma_{{\mathbf{X}}{\mathbf{X}}|{\mathbf{Y}}}|$, where $\Sigma_{{\mathbf{X}}{\mathbf{X}}\mid{\mathbf{Y}}}=\Sigma_{{\mathbf{X}}{\mathbf{X}}}\!\!-\!\!\Sigma_{{\mathbf{X}}{\mathbf{Y}}}\Sigma_{{\mathbf{Y}}{\mathbf{Y}}}^{-1}\Sigma_{{\mathbf{Y}}{\mathbf{X}}}$, per [21]. Therefore, GReg2 is simplified as minimizing Conditional Feature Shift (CFS):

$\displaystyle\min_{\phi,\psi}|\Sigma_{{\mathbf{X}}{\mathbf{Y}}}\Sigma_{{\mathbf{Y}}{\mathbf{Y}}}^{-1}\Sigma_{{\mathbf{Y}}{\mathbf{X}}}|.$

(11)

This section presents the empirical losses used to implement Eq. 3. More detailed derivation can be referred to in Supplementary 7. We introduce the mapping $\psi$ to relax the static target distribution. The implementation of $\psi$ varies across tasks, utilizing MLPs for classification and regression, and ResNet-50 for segmentation. To promote a consistent latent space, the mapped $\psi({\mathbf{Y}})$ retains the same dimension as that of $\phi({\mathbf{X}})$. $\psi({\mathbf{Y}})$ and $\phi({\mathbf{X}})$ are separately fed into $\mathcal{C}$ for making predictions and obtaining $\mathcal{L}_{A2}$ for posterior maximization:

$\displaystyle\mathcal{L}_{A2}(\mathcal{C},\phi,\psi)=H_{c}(\phi({\mathbf{X}}),{\mathbf{Y}})+H_{c}(\psi({\mathbf{Y}}),{\mathbf{Y}}).$

(12)

To mitigate domain shifts and learn domain invariance, we minimize cross-domain conditional feature distribution discrepancies. Specifically, the mean and variance of the joint distribution of $(\phi({\mathbf{X}}),\psi({\mathbf{Y}}))$ in each domain are estimated using VAE encoders. Consider $n$-pairs means and variance of $n$ domains, we derive a joint Gaussian distribution expression $P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n}))\triangleq\mathcal{N}({\bm{x}}_{n},{\bm{y}}_{n};\mu_{n},\Sigma_{n})$. Accordingly, we establish $\mathbb{E}[P({\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})})]\triangleq\mathcal{N}(\bar{{\bm{x}}},\bar{{\bm{y}}};\bar{\mu},\bar{\Sigma})$ where $\bar{\mu}=\mathbb{E}[\mu_{n}],\bar{\Sigma}=\mathbb{E}[\Sigma_{n}]$. Base on PUB in Eq. LABEL:eq:PUB, we introduce $\mathcal{L}_{A1}$ to minimize the conditional feature gap across domains:

$\displaystyle\mathcal{L}_{A1}(\phi)=\sum\nolimits_{i=1}^{n}(\log|\Sigma_{i}|+||\bar{\mu}-\mu_{i}||^{2}_{\Sigma_{i}^{-1}}).$

(13)

To integrate prior information, similar to MIRO, we utilize VAE encoders to capture the means and variances of ${\mathbf{X}}$: $P(\phi({\mathbf{X}}))\triangleq\mathcal{N}({\bm{x}};\mu_{x},\Sigma_{x})$ and the output features $x_{\mathcal{O}}$ form $\mathcal{O}$. Given that $\mathcal{O}$ preserves the correlation between ${\mathbf{X}}$ ($\phi({\mathbf{X}})$) and ${\mathbf{Y}}$ ($\psi({\mathbf{Y}})$), and is frozen during training, ${\mathbf{Y}},\psi({\mathbf{Y}})$ is omitted in empirical loss. We propose $\mathcal{L}_{R1}$ to minimize the divergence between features and $\mathcal{O}$:

$\displaystyle\mathcal{L}_{R1}(\phi)=\log|\Sigma_{x}|+||x_{\mathcal{O}}-\mu_{x}||^{2}_{\Sigma_{x}^{-1}}.$

(14)

For suppressing the invalid causality, derived from Eq. 11, the loss is designed to minimize the CFS:

$\displaystyle\mathcal{L}_{R2}(\phi)=||\Sigma_{{\mathbf{X}}{\mathbf{Y}}}\Sigma_{{\mathbf{Y}}{\mathbf{Y}}}^{-1}\Sigma_{{\mathbf{Y}}{\mathbf{X}}}||_{2},$

(15)

where $\Sigma_{{\mathbf{X}}{\mathbf{Y}}}=\mathbb{E}[(\phi({\mathbf{X}})-\mathbb{E}[\phi({\mathbf{X}})])^{\top}(\phi({\mathbf{Y}})-\mathbb{E}[\phi({\mathbf{Y}})])]$, and a similar calculation process is done for $\Sigma_{{\mathbf{Y}}{\mathbf{Y}}}$ and $\Sigma_{{\mathbf{Y}}{\mathbf{X}}}$. The final loss is a weighted combination of the above losses¹¹1See detailed hyper-parameters settings in Supplementary 10.:

$\displaystyle\mathcal{L}(\mathcal{C},\phi,\psi)\!\!=\!\!v_{A1}\mathcal{L}_{A1}\!\!+\!\!v_{A2}\mathcal{L}_{A2}\!\!+\!\!v_{R1}\mathcal{L}_{R1}\!\!+\!\!v_{R2}\mathcal{L}_{R2}.$

(16)

We perform a regression task on synthetic data to illustrate the impact of using $\psi$, showcasing its potential for superior results if $\psi$ is not bijective.

Synthetic data. Supplementary Figure 3 illustrates the construction of synthetic data, built on ${\mathbf{X}}$-${\mathbf{Y}}$ pair latent features with a linear relationship, ensuring invariant existence. To better explore this issue, we created four distinct data groups: without and with distribution shift, used affine or squared and cubed transformations as domain-conditioned transformations, and their cross combinations. More description can be seen in Supplementary 10.

Experimental setup. We use two of three constructed domains for training and validation and the last one for testing. Validation and test losses are calculated by MSE. To maintain fairness, all experiments adopt the same network which is selected by the best validation results. Learning aims to find invariant hidden features of $X,Y$ while preserving predictive ability from unseen $X$ to $Y$.

Results. Toy experiment results are reported in Table 3, which are also visualized in Figure 4. It is observed that across all settings, employing $\psi$ with $\mathcal{L}_{A1}$ yields superior results, outperforming ERM and ERM+$\mathcal{L}_{A1}$($\phi$) without $\psi$, validating the enhanced generalization effect brought by utilizing $\psi$ whenever $Y$ varies per domain, supporting Eq. 17. Supplementary Figure 4 shows that $\psi$ does learn the invariant representations for $Y$ to relax previous $Y$-invariant assumption. Specifically, learning the invariance of $Y$ with $\psi$ results in superior invariant representations as the latent representations of $X,Y$ are primarily linear, aligning with $X$ and $Y$’s linear relationship during data construction. The bottom-left figures reveal that though ERM has learned the most invariant $\phi(X)$, it suffers the worst test loss, indicating that a well-learned invariant $\phi(X)$ is not sufficient when $Y$ also has domain-dependent traits. The results also suggest that assuming that $Y$ vary across domains, using $\mathcal{L}_{A1}$ without $\psi$ may not yield superior results.

We conduct the Monocular Depth Estimation task as the real-world regression task to further verify GMDG.

Experimental setup. We employ VA-DepthNet [32] with Swin-L [33] backbone as the baseline and follow their hyperparameter settings. Experiments are conducted on NYU Depth V2 [46]. To construct multiple domains, we split the dataset into four categories: ‘School’ (S), ‘Office’ (O), ‘Home’ (H), and ‘Commercial’ (Co). We conduct the standard leave-one-out cross-validation as an evaluation method. We use the best checkpoint on the seen domains for the evaluation. Note that all models are trained on the newly constructed dataset. Statistical results on popular evaluation metrics such as the square root of the Scale Invariant Logarithmic error (SILog), Relative Squared error (Sq Rel), Relative Absolute Error (Abs Rel), Root Mean Squared error (RMS), and threshold accuracy ($\delta_{1},\delta_{2}$) are used as evaluation metrics. See more experimental details in the Supplementary 10.

Results. The Monocular Depth Estimation results are exhibited in Table 5. It can be seen that using terms that are proposed in GMDG leads to better generalization on unseen domains, and using the full GMDG leads to the best results in most metrics. The improvements suggest the feasibility of our GMDG in real-world regression tasks. Specifically, using GReg2 with other terms significantly improves the results when Home is the unseen domain, which is the most difficult domain to be generalized for the VA-DepthNet, and barely compromises the performances while using other domains as the unseen. This reveals that suppressing the causality can improve the generalization of the model (refer to Figure 2 (b) for visual results). Note that, except SILog, all metrics results are scaled by $100$ for readability; due to the lack of objective targeting on the mDG problem, VA-DepthNet performs worse than its baseline. See more visualizations in Supplementary 11.

Experimental setup. We follow the experimental setup of RobustNet [10] for mDG segmentation experiments, particularly using DeepLabV3+ [8] as the semantic segmentation model architecture, with ResNet-50 backbone and SGD optimizer. As shown in Table 1, RobustNet’s objective is equivalent to using GAim2 and GReg2. Consistent with previous methods, mIoU serves as our evaluation metric. Datasets comprise real-world datasets (Cityscapes [11] (Ci), BDD-100K [53] (B), Mapillary [35] (M)) and synthetic datasets (GTAV [41], SYNTHIA [42]). Specifically, we train a model on GTAV and Cityscapes, testing on other datasets. We compare our results to DeepLabv3+[8], IBN-NET[36] and RobustNet [10]. We use Intersection over Union (mIoU) as the evaluation metric. See Supplementary 10 for more experimental details.

Results. Table 6 shows the efficacy of our proposed objective in segmentation tasks upon introducing $\psi$. Ablation results highlight that using $\psi$ alongside GAim1 can enhance baseline performance, experimentally substantiating that the introduction of $\psi$, in relaxing assumptions, boosts performance for better generalization. Using GReg1 alone also improves average mIoU. Importantly, the most enhancement in average mIoU is observed when GReg1 and GAim1 are used together, which finds validation in the PUB derivation in Eq. LABEL:eq:PUB. See more results and visualizations in Supplementary 11.

Experimental setup. We operate on the DomainBed suite [14] and leverage standard leave-one-out cross-validation as an evaluation method. We experiment on $5$ real-world benchmark datasets, including PACS [25], VLCS [12], OfficeHome [50], TerraIncognita [2], and DomainNet [38]. The results are the averages from three trials of each experiment. Following MIRO, two backbones are used for the training (ResNet-50 [15] pre-trained in the ImageNet [15] and RegNetY-16GF backbone with SWAG pre-training [47]). The backbones are trained with our proposed objective barely and further with SWAD [6], respectively. See Supplementary 10 for more experimental details.

Results. Table 7 displays the results of non-ensemble algorithms and ensemble algorithms that employ pre-trained models as oracle models. Specifically, our proposed objectives demonstrate more substantial improvements when a higher-quality pre-trained oracle model ($\mathcal{O}$) is applied. When employing the ResNet-50 model, our approach yields average improvements of approximately $0.3\%$ and $0.1\%$ without and with SWAD, respectively, compared to MIRO. In contrast, when RegNetY-16GF serves as an oracle, GMDG results in significant average improvements of $1.1\%$ and $0.9\%$ without and with SWAD, respectively. Remarkably, our approach outperforms $0.1\%$ more than the SOTA method, SIMPLE++, which relies on an ensemble of 283 pre-trained models as oracle models, whereas ours only engages a single pre-trained model. Overall, these results strongly support GMDG’s effectiveness in classification tasks. See more results in Supplementary 11.

To better compare our objective with previous objectives, we conduct a systematic ablation study on the classification task since most previous objectives are only available for classification due to the lack of $\psi$. Experimental setup. In the ablation studies, we test varied terms (see Table 8) combinations on the HomeOffice dataset using SWAG pre-training [47] and SWAD [6]. Every experiment is repeated in three trials, sharing the same hyper-parameter settings for evaluation. See Supplementary 10 for more details.

Results. Table 8 presents ablation study results. The first column denotes previous methods equivalent to term combinations. The main findings are as follows. See Supplementary 11.1 for more other findings.

1). Previous methods that partially utilize our proposed objectives often yield suboptimal results. Note that iAim1 is the unconditional version of GAim1. By eliminating other factors, it can be seen that employing our proposed full objectives offers the most significant improvements, while previous objectives may lead to inferior results.

2). The effectiveness of using conditions. By conducting uniform implementation and testing, it can be observed that the use of conditions yields superior results compared to the unconditional approach. This observation aligns with Eq. 19, suggesting that minimizing the gap between conditional features across domains leads to improved generalization. The disparity in performance between CDANN and DANN might be attributed to differences in their implementation details.

3). Learning invariance is crucial, regardless of whether integrating prior knowledge. Evidently, learning invariance facilitates improvement whether prior is applied or not, as validated in the PUB derivation in Eq. LABEL:eq:PUB. This contradicts MIRO’s argument that achieving similar representations to a prior can replace the need for learning invariance.

4). Impacts of using prior. The significant improvement owes to the use of a pre-trained oracle model ($\mathcal{O}$) preserving correlations between ${\mathbf{X}}$ and ${\mathbf{Y}}$ - a concept validated by MIRO and SIMPLE. However, utilizing our full set of objectives can further enhance this improvement by an additional $0.7\%$. Notably, the invalid causality may not work when using prior knowledge, while the invariance across domains is not permitted. We hypothesize that such invalid causality is inherently eliminated within a ‘good’ feature space obtained by $\mathcal{O}$, but may be reintroduced when we minimize the domain gap with $\mathcal{O}$. Thus, using the full objective can synergistically produce optimal results.

5). Constraining only the covariance shifts of features across domains (iReg2) does not guarantee better results when prior knowledge is available. We find that using the objectives of CORAL performs better than DANN, CDANN, and CIDG. The results suggest that considering the covariance shifts of features does lead to improvements, which we hypothesize are primarily driven by $H(P(\phi({\mathbf{X}})))$. However, when a large pre-trained oracle model ($\mathcal{O}$) is provided, the performance actually degrades. This implies that the use of $\mathcal{O}$ implicitly minimizes the covariance shifts of features across domains. Under this scenario, the unexpected effect of $-H(P({\phi({\mathbf{X}}|\mathcal{D})}))$ hinders improvement, while the benefits brought by $H(P(\phi({\mathbf{X}})))$ are diminished by the use of prior knowledge. In contrast, GReg2 continues to yield improvements. This suggests that GMDG is more versatile and suitable for various situations.

Details for Eq. LABEL:eq:bound1.

We denote $P_{mix}\triangleq\sum_{n}w_{n}P({\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})})$. Therefore for GJSD, we have:

$$\begin{split}&GJSD(\{P({\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})})\}_{n=1}^{N})\\ =&\sum\nolimits_{n}w_{n}KL(P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n}))\|P_{mix})\\ =&\sum\nolimits_{n}w_{n}[H_{c}(P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})),P_{mix})\\ &-H(P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})))]\\ =&\sum\nolimits_{n}w_{n}H_{c}(P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})),P_{mix})\\ &-\sum\nolimits_{n}w_{n}H(P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})))\\ =&\sum\nolimits_{n}w_{n}\int_{\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})}-P(x,y)\ln P_{mix}(x,y)d(x,y)\\ &-\sum\nolimits_{n}w_{n}H(P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})))\\ =&\int_{\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})}-\sum\nolimits_{n}w_{n}P(x,y)\ln P_{mix}(x,y)d(x,y)\\ &-\sum\nolimits_{n}w_{n}H(P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})))\\ =&\int_{\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})}-P_{mix}(x,y)\ln P_{mix}(x,y)d(x,y)\\ &-\sum\nolimits_{n}w_{n}H(P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n})))\\ =&H(P_{mix})-\sum\nolimits_{n}w_{n}H(P(\phi({\mathbf{X}}_{n}),\psi({\mathbf{Y}}_{n}))).\end{split}$$

(20)