Spectrum-Guided Adversarial Disparity Learning

Following the notations in Kingma et al.’s work (Kingma and Welling, 2013), we denote by $p_{data}(x)$ the observed data distribution, $p(x)$ the model distribution of reconstructing data, $q(z|x)$ the encoding distribution, and $p(x|z)$ the decoding distribution. The distribution $q(z|x)$ encodes the original data into latent codes $z$, which denotes the latent representation for data reconstruction. Suppose that the intraclass disparity $\delta$ confuses pure information $\gamma$ in a linear relationship, we define that the latent representation $z$ is composed by

where $q_{\varphi}(\gamma|x),q_{\eta}(\delta|x)$ denote the pure information and disparity encoding distributions, respectively.

In this phase, we use the denoising autoencoder to denoise disparity and to reconstruct the signal waveform. We consider the signal waveform reconstruction with the following requirements: (i) the decoder should be able to reconstruct original data by $z=\gamma+\delta$; (ii) $\gamma$ should be able to learn the representative and invariant waveform information; (iii) $\delta$ should be the confusion factor in waveform only related to class information.

Given the optimal disparity $\delta$, the reconstruction can be demonstrated as a denoising autoencoder from such disparity. The autoencoder fuses the learned disparity $\delta$ with raw data as inputs and then trains autoencoder to recover the original data. Thus, the model will be less sensitive to the disparity components. In conventional denoising autoencoders (Vincent et al., 2008), the disparity part in original data is vague; thus, they fuse the noise in raw data (e.g., Gaussian noise). Different from the meaningless noise, the proposed disparity encoder $q_{\eta}(\delta|x)$ can extract the disparity components, which will be proved to be the disparity with class information in the regularization phase by the adversarial training. Therefore, we move the fusion process after encoding.

The requirement (i)-(iii) can be explained with the following definitions. Let $\gamma$ be pure latent codes and fix the optimal disparity distribution to let $\delta$ be the constant noise. Then, the purified latent code $z=\gamma$ and the confused code $z^{{}^{\prime}}=\gamma+\delta$. The condition probability distribution of $z^{{}^{\prime}}$ on x is $q_{\varphi}(z^{{}^{\prime}}|x)$. Consider an approximate posterior distribution between $x$ and $z$:

where $c(z|z^{{}^{\prime}})$ is the observed probability distribution between confused codes and pure codes. Then, given the decoding distribution $p_{\theta}(x,z)=p_{\theta}(x|z)p(z)$, the lower bound of the autoencoder may be formed in the following way (Im et al., 2017) by Jensen’s inequality:

Therefore, the lower bound of autoencoder can be sorted into minimizing the negative weighted likelihood $L_{rec}$:

In this phase, we consider the encoding distribution regularization. The conventional AAE (Balabka, 2019) only applies binary discriminator to regularize generation distribution as a fixed prior distribution by predicting ’real’ or ’fake’, which fails to portray the class-conditioned intraclass disparity. Therefore, we propose a multi-label classifier and a learnable prior distribution to improve the conventional AAE by concluding the following requirements: (i) discriminator should be able to distinguish $\gamma$ and $\delta$ based on the information validity in prediction; (ii) $\gamma$ represents the generalized class information without subject variation; (iii) $\delta$ represents the subject variation conditioned on activity.

As mentioned above, some intraclass disparity exists due to the motion and body shape variance. The similarity distribution regularization will keep this disparity in the representations. Thus, we derive the original binary classifier to a multi-label classifier to distinguish the validity of learned representations. The representations without disparity should be more robust in predicting different subjects, while the disparity representations will impair the performance. Note that, we only consider the case that the dataset contains the class information; otherwise, the prediction validity cannot be measured. Then, we can explain requirement (ii)-(iii) by denoting the low-validity distribution in regularization as the disparity distribution and high-validity distribution as the pure information in class prediction. The requirements (i) can be modified to let discriminator predict whether the latent codes are ‘valid’ or ‘invalid’ according to the likelihood of correct predictions.

First, we discuss the Categorical Cross Entropy (CCE) for multi-label classification:

where $y$ denotes a one-hot probability distribution of ground truth; $\hat{y}_{c}$ represents the discriminator probability of true label $c$.

We can find that CCE loss is equivalent to the negative log-likelihood of the true prediction probability. Thus, minimizing the CCE loss equals maximizing the validity (i.e., the likelihood of correct prediction). We utilize the adversarial regularization to optimize requirements (ii)-(iii) and then conclude three requirements in an adversarial format.

Given data $(X,Y)$, the competitive distribution (i.e., the encoder of $\gamma$) enables the generator distribution (i.e., the encoder of $\delta$) to learn the class conditioned intraclass disparity by optimizing the loss function $L_{reg}$:

where $d$ denotes the parameters of the multi-label discriminator $D$; $D_{c}$ denotes the probability of correct predictions, respectively.

As mentioned in proof, the optimization $D_{d^{{}^{\prime}}}$ converges to precisely predicting $q_{\varphi}(\gamma|x)$ and $q_{\eta^{{}^{\prime}}}(\delta|x)$, and $q_{\varphi^{{}^{\prime}}}(\gamma|x)$ is optimized to be distinct from $q_{\eta^{{}^{\prime}}}(\delta|x)$. In other words, $q_{\varphi^{{}^{\prime}}}(\gamma|x)$ is invisible to discriminator $D_{d^{{}^{\prime}}}$. Thus, only if $q_{\varphi^{{}^{\prime}}}(\gamma|x)$ learns the purified data will $D_{d^{{}^{\prime}}}$ predict the accurate classification. To further analyze the loss functions of proposed disparity learning, we specify the details of two competitive encoding training and name the two steps of the min-max optimization of $L_{reg}$ as pure loss and disparity loss according to the corresponding encoding distributions:

We exhibit curves of $L_{pur},L_{dis}$ to display the convergence of two competitive encoding losses and the training classification performance of using $D_{d^{{}^{\prime}}}$ predicting the purified representations $q_{\varphi^{{}^{\prime}}}(\delta|x)$ in Section 3.4 Convergence Analysis, which proves intraclass disparity learning and denoising.

The spectrum analysis focuses on the patterns of signals in the frequency domain, which is transformed from the time domain. Since sensors are only able to record signals in a certain sampling rate, let $x[n]$ be the observed signals in such discrete-time domain. We can obtain the corresponding frequency domain data $X$ and amplitude spectrum $A$ that records the signal strength of each frequency via Fourier transform(Smith et al., 1997):

where $A[n]$ denotes the amplitude of frequency $n$; $N$ denotes the discrete time point number of the sampled signal; $i,e$ represent the imaginary number and the mathematical constant. The amplitude spectrum is perfectly symmetric, so we only analyze the positive frequency part, $k\in[0,N-1]$.

Our goal is to design an amplitude spectrum guide function $S_{\zeta}(A[i])\rightarrow\mathbb{R}$, which can measure the importance of each frequency according to the signal strength. We define the mean score of the amplitude spectrum to represent the signal information amount (i.e., the signal importance) in the numerical format. To ease our illustration, we express the mean score of an arbitrary frequency set $T$ evaluated by spectrum guide function $S_{\zeta}$ as

where $\zeta$ denotes the function parameters; $|T|$ represents the element number of $T$. Therefore, the signal importance can be measured by its amplitude spectrum $S_{\zeta}(A)$.

To automate the spectrum analysis, we analyze amplitude spectrum in terms of the frequency set. According to (Viterbi and Omura, 2013), the signal information is mainly composed of some frequencies with the highest signal strengths. We let $U$ be the frequency set of the highest amplitudes (i.e., information) and $I$ be the frequency set of the lowest amplitudes (i.e., gap and noise):

where $U,I$ are two subsets of $A$.

The spectrum analysis principles are based on intra-relationship and inter-relationship in signals, which describe the relationship of frequencies within a signal and between signals, respectively. We illustrate the proposed principles by recalling the phenomenon in Fig. 2: differing from time-segments, given two arbitrary data segments clipped from a series of noisy continuous signals with gaps, we can easily tell the main composition (i.e., orange and green peak points) and the class of two segments in the frequency domain. Also, we can easily see that the green segment contains more information, because its curve has more strengths than orange segment’s, especially around peak points. The conventional band-pass filter analysis (Viterbi and Omura, 2013) manually selects the frequencies with the highest strengths, i.e., the peak points. To automate the spectrum analysis, we transform the frequency selection to weigh frequencies based on frequency set analysis and propose two principles:

1. (1)

   Given an amplitude spectrum $A$, $\forall i\in U,\forall j\in I$, function $S_{\zeta}$ should satisfy

   $$S_{\zeta}(A[i])-S_{\zeta}(A[j])\propto A[i]-A[j]$$

2. (2)

   Given the certain frequency’s amplitudes from two arbitrary different spectra $A_{a},A_{b}$, $\forall i\in A$, function $S_{\zeta}$ should satisfy

   $$S_{\zeta}(A_{a}[i])-S_{\zeta}(A_{b}[i])\propto A_{a}[i]-A_{b}[i]$$

The principle (i) describes the traditional band-pass filter theory (Viterbi and Omura, 2013): if the frequency $A[i]$ is the most significant frequency (i.e., the information frequency), $A[i]$ should have much higher amplitudes than others’. Since principle (i) only considers the intra-amplitude difference relationship, we further derive the principle (ii) to complement the inter-relationship.

Beginning with principle (i), refer to Eq. (6), we consider that the information set $U$ and the noise set $I$ (including noise and gaps) meet the significant amplitude difference requirement. To replace the traditional case-specific band selection, we utilize the information set $U$ to represent the frequencies where information may exist. Then, we can maximize the below loss function $L_{N}$ to let $S_{\zeta}$ satisfy principle (i):

Further, we utilize principle (ii) to distinguish different signal weights. Similar to principle (i), we can easily conjecture the amplitude difference relationship between the particular frequencies of two spectra in the frequency-set format:

Then, for arbitrary spectra $\overline{A_{a}}>\overline{A_{b}}$, we can minimize the following loss function $L_{O}$ to meet principle (ii):

where $\alpha$ denotes the proportionality constant.

To put the proposed principles into practice, we further pose a value range in the function $S_{\zeta}(A)\rightarrow(0,1)$ (0 means extremely insignificant and 1 means extremely significant), and thus the optimal maximum of $S_{\zeta}(U)-S_{\zeta}(I)$ should be 1. Maximizing $L_{N}$ is equivalent to minimising $1-L_{N}$. The empirical proportionality constant holds $\alpha=1$.

With the above empirical setting, the expected range of $A$ should be $(0,1)$. To match the different analysis perspectives, we propose to rescale the amplitudes by min-max normalization in two folds. Assuming that the spectrum set $\mathcal{A}^{n\times m}:=\{A_{1},A_{2},\dots,A_{n}\}$ contains $n$ spectra with $m$ frequency channels. We normalize each spectrum $A_{i}$ in frequency-level and sample-level to be consistent with two principles (Section 2.2). Denote $A_{i}^{N}$ and $A_{i}^{O}$ to represent the normalized data of intra- and inter-relationships for an arbitrary spectrum $A_{i}$, respectively:

where $\mathcal{A}_{i},\mathcal{A}_{j}$ denote a row and column vector of $\mathcal{A}$, respectively; $\mathcal{A}_{i}^{min},\mathcal{A}_{i}^{max},\mathcal{A}_{j}^{min},$ and $\mathcal{A}_{j}^{max}$ denote the minimum and maximum of each row or column. Then, the score function evaluates the $i$th frequency using both intra and inter normalized features by

To ensure $U$ and $I$ satisfy our definitions, we empirically take the 20% of frequencies with the highest amplitudes as $U$ and the lowest 50% as $I$ in $A^{N}$. With the min-max normalization, the mean amplitude difference between $U$ and $I$ will be large enough to distinguish the noise and information.

The optimization of the score function $S_{\zeta}$ can be demonstrated by two steps: (i) for any amplitude spectrum $A$, use $A^{N}$ to update $\zeta$ by minimizing $1-L_{N}$; (ii) for any amplitude spectrum pair $(A_{a},A_{b})$, use $(A^{O}_{a},A^{O}_{b})$ to update $\zeta$ by minimizing $L_{O}$. We can then unify the steps to optimize $\zeta$ by minimizing a pairwise loss function $L_{S}(A_{a},A_{b})$:

where the first line denotes $1-L_{N}$ for $A^{N}_{a},A^{N}_{b}$ and the second line is $L_{O}$ for spectrum pair. Note, we adopt a Fully Connected (FC) layer followed by Sigmoid as $S_{\zeta}$ to match the value range in the definition, and then the optimization will be a traditional regression problem, which can be converged (Kingma and Ba, 2014).

Given a set of signals with labels $(X,Y)$, we calculate the spectrum weighted loss function for an arbitrary signal $(x_{i},y_{i})$ as follows:

where $A_{x_{i}}$ denotes the amplitude spectrum of $x_{i}$; $w_{c}$ denotes the weight of $x_{i}$’s class; $|X_{c}|$ denotes the number of the samples in class $c$. Due to the activity type and the subject variance, the information amounts carried by signals are varying from each other and thus we apply the mean class spectrum weights of the ground truth to leverage the discriminator training.

We apply two weighing methods in SAAE. Since the reconstruction phase focuses on a single signal waveform, we leverage different signals by its own spectrum. The regularization phase focuses on the class conditioned disparity, so we leverage signals by overall class information amount distribution. The optimization algorithm is shown in Algorithm 1.