Disentangling semantics in language through VAEs and a certain architectural choice

The model differs from classical VAE encoders in that it will encode a sentence into $n_{3}$ latent variables, where $n_{3}$ is a fixed integer (regardless of the sentence length).
Our choice was to use a transformer Vaswani et al. (2017). More specifically, we will use the transformer encoder-decoder architecture that is mostly used for machine translation. Contrary to an encoder only transformer , this architecture allows for obtaining a number of output elements that is different from that of the input sequence (as is needed for translation). It has been established that transformers can store sentence-level statistics in artificially introduced tokens (e.g.SEP in Devlin et al. (2019)). In a similar manner, we will feed a set of fixed $n_{3}$ learnable vectors to the decoder in place of it s targets. The transformer encoder-decoder’s architecture and the decoding process are explicited in figure 3, where the ”Previous Latent variable value” placeholder (light blue) is empty.We will apply $n_{3}$ distinct linear transformations (resp. $n_{3}$ MLPs with softplus activations) to obtain the means (resp. the standard deviations) of the posterior distribution of $z_{3}$.

Similarly to the way we obtain $z_{1}$, we use a Transformer encoder-decoder architecture. The latent variables that we condition on are introduced here by concatenating them to the input sentence after positional-encoding and Transformer-encoding, as depicted in figure 3. These latent variables are viewed as additional elements of the sentence with no specific positioning.

As explained in 3.1, we use a learnable structured prior $p_{\theta}(z_{1},z_{2},z_{3})=p(z1)p_{\theta}(z_{2}|z_{1})p_{\theta}(z_{3}|z_{1},z_{2})$. To obtain the parameters of $z_{2}$ and $z_{3}$ in the generative model, we use the same architecture we used in the encoder without inputting text (i.e we use the model in figure 3 while dropping the green part).

One must notice that, given our training procedure (c.f. ELBo in section 4), all latent variables in $z_{1}$ are enforced to follow the same prior. In the generation step, we will be sampling a set of similarly distributed random variables with no means for the decoding network to distinguish between them. As our objective is to encode $n_{1}$ different types of information in $z_{1}$, and to have the decoder identify and leverage this information, we will concatenate the vectors corresponding to the value of each latent variable in $z_{1}$ to a latent-variable-specific trainable vector before having it decoded. The same will be done for $z_{2}$, and $z_{3}$ even though their trainable priors enable better distinguishability .

Sequence to sequence models (Seq2seq Sutskever et al. (2014)) that do not use attention were always found to be lacking in comparison to those that do. As a side effect to our architectural choices, we will be able to use attention based decoders and thus benefit from their higher expressiveness. In the same spirit as that of the previous section, we will use sequence transduction components that were originally designed to be used in machine translation to simultaneously translate and align.
We chose to use here the same transformer encoder-decoder architecture used in the encoding stage, but with different inputs. It will closely follow machine translation in this step by receiving the latent variable values as source inputs, and the previously generated tokens as target inputs. Contrary to what is done in the sequence encoder, the transformer applied to targets will use an attention mask that enforces the current generated word to depend only on previous words. A Latent variable identifier coupled with a transformer decoder are depicted in figure 4.

ROOT-DEP-APPEAR, DEP-APPEAR, and DEP-ALTER are lists of dependency labels. We found it interesting to look at the latent variables that causes each of the dependency labels to appear in each of the 3 lists.
Influencing ROOT-DEP-APPEAR means having the corresponding dependency label appear/disappear from the ROOT children. Influencing DEP-APPEAR means having the corresponding dependency label appear/disappear from the whole dependency tree. Influencing DEP-ALTER means having the text behind a certain dependency label change in a static length sentence.
We report the latent variables with the highest influence on each dependency label for these three statistics in figure 5. A first look at the results shows that a major part of the variability in the generated sentences is expressed by latent variable (LV) 10. Figure 5 shows that is it responsible for the content of the ROOT node, which explains how its influence propagates to the major part of the sentence. LV 30 seems to influence nominal subjects (passive or active), auxiliaries (possibly for conjugation) in terms of content, and numeral modifiers, and expletives when it comes to structure. These cues clearly point to LV 30 being responsible for subject related information. Other highly influential LVs are 35, and 43. 35 is responsible for the appearance of conjunctions, as well as the content of direct objects. LV 43 controls the content in prepositional objects, and is structurally related to the appearance of compounds, adverbial clause modifiers, and markers. Consequently, we expect these last two LVs to control most of the information past verbs and subjects.

We plot the influence of each latent variable on the appearance of OpenIE arguments (OIE-APPEAR) and on their content (OIE-ALTER) as heat maps. OIE-APPEAR showed no evidence of the presence of variables that control structure while disregarding content. Therefore the heat map for OIE-APPEAR is reported in Appendix B, while the heat map for OIE-ALTER is in figure 6.
As was expected from the dependency parsing analysis, LV 10 is the most influential on the verb. It can also be seen that LV 30 has the highest influence on ARG0 (i.e. the subject). Along with the information from the dependency analysis, figure 6 further stresses the roles of LV 35 and LV 43. LV 35 seems to specialize in the direct object (ARG1), while LV 43 partly describes the direct object as well as secondary arguments (ARG2 often corresponds to prepositional objects), and contextual arguments (ARGM-DIR, ARGM-LOC, and ARGM-MNR correspond to direction, location, and manner). One should notice that LV 10 is in the root level of latent variables ($10<16$), LV 30 in the middle level ($16<30<32$) and LVs 35 and 43 in the leaf level ($32<35<43<48$). The disentangled information is consequently also arranged as dictated by a linguistic dependency structure. This further confirms the ability of machine learning models to align with our conception of linguistic structure.

As was pointed out by the quantitative analyses, this LV is the most influential (overall) on the sentence. But it seems to specialize, to a certain extent, in specifying the verb. In table 1, We can see that varying LV 10 keeps the same subject for the sentence, but varies the verb and the object (which is highly dependent on the verb). As LV 10 is a low level variable, changing it results in an incompatibility with the higher level variables, and a radical change in the sentence is observed. The fact that verbs can’t be changed independently from their subject is also observed in table 2. In fact, swapping LV 10 clearly results in unexpected changes.

Despite some negative examples(table 1, 5th row, 5th column), Tables 1 and 2 clearly demonstrates that LV 30 contains the information on the subject. We can see, nevertheless, that changing it results in co-adaptation of the rest of the sentence, such as the conjugation of a verb (table 1, 5th row, 3rd column). A surprising observation can be made in table 2, 6th row: a change of subject from plural to singular resulted in the same co-adaptation of the verb on 3 examples. It is unclear whether ”sitting” has been reinterpreted as ”laying”, or the latent code stores the action for a group in a different area than the action for a single individual.

These two LVs, most often encode low level information (i.e. leaf information in the dependency parsing sense). To generate table 2, we had to try both LVs and see which contained the information for the verb at hand. Another constraint for results in table 2 to be coherent, was for the sentences to feature the same verb. As these LVs are leaf LVs, it is only natural that they can only be swapped between sentences where they remain a high probability sample when conditioned on the root LVs. As can be seen in table 1 (rows 8 and 10) when varying these LVs for the same sentence, they change different aspects of the object. We could also confirm that LV 35 most often controls direct objects (table 1 lines 7, 8, and table 2 lines 7, and 9), while LV 43 holds the information on prepositional objects (table 1 lines 11 and 12, and table 2 line 8). LV 35 also seems to control some intransitive verbs (table 1 line 9). Given that most sentences are in the past, these verbs may be perceived by the model as objects to the auxiliary.

An inherent short-coming of VAEs is the fact their objective (ELBo) is only a lower bound to the exact marginal log-likelihood of the data. In fact, the positive term quantifying the gap between ELBo and $log(x)$ is $KL[q(z|x)||p(x|z)]$. This difference results in a discrepancy between encoder and decoder. We study this discrepancy under the light of the attention values between input tokens, and latent variables. Example heat maps of these attention values for our latent variables of interest (10, 30, 35, and 43) are provided in Appendix C. The most striking observation that can be made is that our latent variables attend to positions where the information they need is expected to appear, with little reliance on the tokens present in the attended positions. In fact, LV 10, LV 30, LV 43, and LV 35 almost exclusively attend to positions $\left[3-5\right]$, $\left[1-3\right]$, $\left[5-6\right]$, and $\left[6-7\right]$. Fortunately, the latent variables in the generator do not seem to only influence the same predefined positions as is illustrated by the examples in tables 1 and 2 (e.g. LV 43 influences tokens at position 9 in the last example of table 1). A secondary observation that can be made is that LV 10 and LV 30 attend a lot to latent variables from previous layers. This establishes that the encoder is actively using its latent variable structure (i.e it successfully learned a structured posterior).