Quantum Language Model with Entanglement Embedding for Question Answering

Neural Network Language Model (NNLM) [1] is widely used in Natural Language Processing (NLP) and information retrieval [2]. With the rapid development of deep learning models, NNLMs have achieved unparalleled success on a wide range of tasks [3, 4, 5, 6, 7, 8, 9, 10]. It becomes a common practice for the NNLMs to use word embedding [11, 4] to obtain the representations of words in a feature space. While NNLM has been very successful at knowledge representation and reasoning [7], its interpretability is often in question, making it inapplicable to critical areas such as the credit scoring system [12]. Two important factors have been summarized in [13] for evaluating the interpretability of a machine learning model, namely, Transparency and Post-hoc Interpretability. The model transparency relates to the forward modelling process, while the post-hoc interpretability is the ability to unearth useful and explainable knowledge from a learned model.

Another emerging area is quantum information and quantum computation where quantum theory can be utilized to develop more powerful computers and more secure quantum communication systems than their classical counterparts [14]. The interaction between quantum theory and machine learning has also been extensively explored in recent years. On one hand, many advanced machine learning algorithms have been applied to quantum control, quantum error correction and quantum experiment design [15, 16, 17, 18]. On the other hand, many novel quantum machine learning algorithms such as quantum neural networks and quantum reinforcement learning have been developed by taking advantage of the unique characteristics of quantum theory [19, 20, 21, 22, 23, 24, 25, 26]. Recently, Quantum Language Models (QLMs) inspired by quantum theory (especially quantum probability theory) have been proposed [27, 28, 29, 30, 31, 32, 33, 34] and demonstrated considerable performance improvement in model accuracy and interpretability on information retrieval and NLP tasks. QLM is a quantum heuristic Neural Network (NN) defined on a Hilbert space which models language units, e.g., words and phrases, as quantum states. By embedding the words as quantum states, QLM tries to provide a quantum probabilistic interpretation of the multiple meanings of words within the context of a sentence. Compared with the classical NNLM, the word states in QLM are defined on a Hilbert space which is different from the classical probability space. In addition, modelling the process of feature extraction as quantum measurement which collapses the superposed state to a definite meaning within the context of a sentence could increase the transparency of the model.

Despite the fact that previous QLMs have achieved good performance and transparency, the state-of-the-art designs still have limitations. For example, the mixed-state representation of the word sequence in [32, 34] is just a classical ensemble of the word states. As shown in the left of Fig. 1, a classical probabilistic mixture of quantum states is not able to fully capture the complex interaction among subsystems. Although Quantum Many-body Wave Function (QMWF) [33] method has been applied to model the entire word sequence as the combination of subsystems, it is still based on a strong premise that the states of the word sequences are separable, as shown in the middle of Fig. 1. A general quantum state has the ability to describe distributions that cannot be split into independent states of subsystems like in the right of Fig. 1. In quantum physics, the state for a group of particles can be generated as an inseparable whole, leading to a non-classical phenomenon called quantum entanglement [35]. Quantum entanglement can be understood as correlations (between subsystems) in superposition, and this type of parallelized correlations can be observed in human language system as well. A word can have different meanings when combined with other words. For example, the verb turn has four meanings {move, change, start doing, shape on a lathe}. If we combine it with on to get the phrase turn on, the meaning of turn will be in the superposition of change and start doing. However, this kind of correlation has not been explicitly generated in the present NN-based QLMs. Besides, a statistical method has been proposed in [36] to characterize the entanglement within the text in a post-measurement configuration, which still lacks the transparency in the forward modelling process.

In this paper, we propose a novel quantum-inspired Entanglement Embedding (EE) module which can be conveniently incorporated into the present NN-based QLMs to learn the inseparable association among the word states. To be more specific, each word is firstly embedded as a quantum pure state and described by a unit complex-valued vector corresponding to the superposition of sememes. The Word Embedding neural network module is adopted from [34]. Word sequences (phrases, $N$-grams, etc.) are initially given as the tensor product of the individual word states, and then transformed to a general entangled state as the output of the EE module. The EE module is realized by a complex-valued neural network, which is essentially approximating the unitary operation that converts the initial product state to an entangled state. After the entanglement embedding, high-level features of the word sequences are extracted by inner products between the entangled state vector and virtual quantum measurement vectors [14]. All the parameters of the complex-valued neural network are trainable with respect to a cost function defined on the extracted features. Entanglement measures for quantifying and visualizing the entanglement among the word states can be directly applied on the output of the learned model. We conduct experiments on two benchmark Question Answering (QA) datasets to show the superior performance and post-hoc interpretability of the proposed QLM with EE (QLM-EE). In addition, the word embedding dimension can be greatly reduced when compared with previous QLMs, due to the composition of word embedding and EE modules in the hierarchical structure of the neural network. Note that the current QLM-EE is proposed to implement on classical computing devices with a quantum-inspired neural network structure.

The main contributions of this paper are summarized as follows.

• •

  A novel EE neural network module is proposed. The output of the EE module represents the correlations among the word states with a quantum probabilistic model which explores the entire Hilbert space of quantum pure states. The embedded states can reveal the possible entanglement between the words, which is an indication of parallelized correlations. The entanglement can be quantified to promote the transparency and post-hoc interpretability of QLMs to an unprecedented level.

• •

  A QLM-EE framework is presented by cascading the word embedding and EE modules. The word embedding module captures the superposed meanings of individual words, while the EE modules encode the correlations between the words at a higher level. The resulting cascaded deep neural network is more expressive and efficient than the shallow networks used by previous QLMs.

• •

  The superior performance of QLM-EE is demonstrated over the state-of-the-art classical neural network models and other QLMs on QA datasets. In addition, the word embedding dimension in QLM-EE is greatly reduced and the semantic similarity of the embedded states of word sequences can be studied using the tools from quantum information theory. The entanglement between the words can be quantified and visualized using analytical methods under the QLM-EE framework.

This paper is organized as follows. Section II provides a brief introduction to preliminaries and related work. Entanglement embedding is presented in Section III. Section IV proposes the QLM-EE model. Experimental results are presented in Section V and the results show that QLM-EE achieves superior performance over five classical models and five quantum-inspired models on two datasets. Post-hoc interpretability is discussed in Section VI and concluding remarks are given in Section VII.

Quantum probabilistic superposition of states models the polysemy of words and sequences. Quantum states span the entire complex Hilbert space, while the amplitudes and complex phases of the states give the probability distribution of the multiple meanings of words. An $N$-gram is the composition of $N$ words whose joint state is defined on the tensor product of $N$ Hilbert spaces. However, the joint state itself is not necessarily the tensor product of $N$ states of the Hilbert spaces. The set of entangled state is significantly larger than that of the tensor-product states, and compound semantic meaning lying within the $N$-gram is mainly captured by the entanglement representation. Then the parameters of the entangled states is trained by the EE module, which characterizes the features of entanglement between the word states within the sequence.

The EE module is trained to capture high-level features between the words in the $N$-gram, while the word embedding module can focus on encoding the basic semantic meanings of individual words with reduced dimension. The clear separation of duties in the hierarchical structure of the neural network increases the transparency of the model to an unprecedented level, which allows an accurate interpretation of the intermediate states using the tools borrowed from quantum information theory.

In line with the previous works, the complex-valued word embedding module is used to transform the one-hot representation of the word into a quantum pure state in the Hilbert space of word states $\mathcal{H}_{w}$, expressed as a state vector $|\psi\rangle=[\alpha_{0}\ \cdots\ \alpha_{i}\ \cdots]^{T}$, with $\sum_{i}|\alpha_{i}|^{2}=1$. The general quantum pure state for describing a word sequence is defined on the tensor-product Hilbert space $\mathcal{H}_{s}:=\otimes_{i}(\mathcal{H}_{w})_{i}$, and can be formulated as

where $\sum_{i_{1},\dots,i_{N}}|\beta_{i_{1}\dots i_{N}}|^{2}=1$. $\arrowvert\psi_{s}\rangle$ is used to characterize the probability distribution of the sememes. If the word embedding dimension is $D$, then a general pure state vector defined on the tensor product of $N$ Hilbert spaces contains $D^{N}$ elements.

The EE module starts with composing the contiguous sequences of words as $N$-grams [42]; see Fig. 3 for an example. For each $N$-gram, a separable pure state is generated as the tensor product of its word states taking the following form

The word states are defined on Hilbert spaces which are isomorphic to each other, with the same semantic basis states. For this reason, $N$-grams are defined on the same tensor-product space before entering the EE module. The complex-valued EE module is then connected to transform the initial separable state to an unnormalized vector $\arrowvert\tilde{\psi}_{s}\rangle=[\tilde{\beta}_{i_{1}\dots i_{N}}]$ which will then be normalized to determine a general pure state $\arrowvert\psi_{s}\rangle=[\beta_{i_{1}\dots i_{N}}]$ in the form of (17). The transformation induced by the NN can be formally written as

where $\mathcal{W}$ is the weight matrix. Then the output vector must be normalized by

to be consistent with quantum theory. Note that the operations (19)-(20) induce an endomorphism of this Hilbert space and the EE module transfers one pure state to another. The pure states are unit vectors in the tensor-product Hilbert space. Any unit vectors in the same Hilbert space can be connected by a unitary matrix which induces a rotation. It is known that for any two quantum pure states $|a\rangle$ and $|b\rangle$ of an $N$-qubit register, there exists a gate sequence to physically realize the unitary matrix $U$ such that $|b\rangle=U|a\rangle$ [43]. Since the model proposed in this paper is only quantum-inspired, we are using a linear matrix multiplication and a normalization layer to approximate the effect of the transformation matrix by the classical way of training. In principle, if one layer of linear matrix multiplication is not sufficient for learning the transformation, it is always possible to stack several layers of linear operations to accurately approximate any transformation, which is consistent with the traditional neural network theory.

We take Fig. 3 as an example to illustrate the working mechanism of the EE module. We denote the state vectors for the first two words as $[\alpha_{1}\ \alpha_{2}]^{T}$ and $[\alpha_{3}\ \alpha_{4}]^{T}$. The two word states form the separable state as the input to the NN layer by the following tensor product

The output of the NN layer is an unnormalized vector $[\tilde{\beta}_{00}\ \tilde{\beta}_{01}\ \tilde{\beta}_{10}\ \tilde{\beta}_{11}]^{T}$. After normalization, we obtain

which is in the most general form of the state vector on the joint Hilbert space. If $[\beta_{00}\ \beta_{01}\ \beta_{10}\ \beta_{11}]^{T}$ cannot be written in a decomposable form just like the RHS of (21), then the output vector is a representation for an entangled state which captures the non-classical correlations between the word states.

The structure of QLM-EE for QA is shown in Fig. 4. It is a complex-valued, end-to-end neural network optimized by back-propagation on classical computing devices. The QLM-EE can be divided into three major steps.

• •

  Word embedding and entanglement embedding. The word embedding module generates the word states, and the entanglement embedding module generates the complete quantum probabilistic description of the sequence of word states. The complete probabilistic description is given by a generic quantum pure state vector, which may be entangled to encode the information on the non-classical correlations between the word states. Entanglement embedding and word embedding modules encode the states as feature vectors at different levels, which improves the transparency of the quantum probabilistic modelling process. In particular, the distance between the feature vectors in the embedding space can be calculated based on the well-established measures from the quantum information theory for comparing quantum states, which could be used to reveal the relations between the words and phrases on a deeper semantic level. For example, in the classical Word2Vec, the cosine similarity is defined on real-valued embedded vectors as

  $$\mathcal{C}(\vec{a},\vec{b})=\frac{\vec{a}\cdot\vec{b}}{||\vec{a}||\cdot||\vec{b}||}.$$

  (23)

  In our case, the cosine similarity is defined on the complex-valued unit vectors as

  $$\mathcal{C}(\vec{a},\vec{b})=\frac{\vec{a}^{\dagger}\vec{b}}{||\vec{a}||\cdot||\vec{b}||}=\vec{a}^{\dagger}\vec{b}.$$

  (24)

  In general, $\mathcal{C}(\vec{a},\vec{b})$ is a complex number, which means that the state representations could differ by a complex phase factor. Since complex phases are hard to visualize, we employ the quantum fidelity measure to compare the words and word sequences.

  In Fig. 5, we visualize the process how the state of the phrase solar system evolves in our model using the quantum fidelity measure. After the word embedding layer, the state vector that represents the word solar is close to $\{$sun, crazy, dominated$\}$, which reflects how the model comprehends the meaning of the input words. The word state vector of system is embedded close to $\{$party, committee, ordered$\}$. After the EE module, the phrase solar system can be linked to other high-level phrases, e.g., united nations, while its state vector is still very close to the phrase the sun and the comet which share a similar meaning.

• •

  Measurement operation. Projective measurement operation is equivalent to the calculation of fidelity between quantum states. That is, the output of the measurement can be seen as a distance between the measured state and a measuring state, corresponding to the distance between the semantic entangled state and a measuring sememe which is used for comparison. Therefore, performing the same set of measurement operations provides a way to compare the semantic similarity of the question and answering word sequences. After entanglement embedding, a series of parameterized measurements $\{\arrowvert m_{i}\rangle\}$ are performed on the state $\arrowvert\psi_{s}\rangle$ via the formula

  $$p_{i}(\arrowvert\psi_{s}\rangle)=|\langle m_{i}\arrowvert\psi_{s}\rangle|^{2}.$$

  (25)

  Here $p_{i}$ is defined as the measurement output, which indicates the probability of the state $\arrowvert\psi_{s}\rangle$ possessing the semantic meaning represented by the measurement vector $\arrowvert m_{i}\rangle$. Note that the unit vectors $\{\arrowvert m_{i}\rangle\}$ are optimized in a data-driven way. By using pure states as measurement basis, the computation cost for measuring a quantum state virtually is reduced from $O(n^{3})$ to $O(n)$ compared to CNM [34], in which density matrices were used as the measurement basis. For a sentence described by the concatenation of $L$ word sequences, the feature matrix which stores all the measurement results has $L\times M$ entries if $M$ measurement vectors are used.

• •

  Similarity Measure. A max-pooling layer is applied on each row of the feature matrix for down-sampling. To be more specific, the down-sampling takes the maximum value of each row of the matrix to form a reduced feature vector. Then a vector-based similarity metric can be employed in the matching layer for evaluating the distance between the pair of feature vectors for question and answer. The answer with the highest matching score is chosen as the predicted result among all candidate answers.

von Neumann entanglement entropy $\mathcal{S}$ [55] is an accurate measure of the degree of quantum entanglement for a bipartite quantum pure state. The entanglement entropy is calculated as follows

where $\lambda_{i}$ is the Schmidt coefficient of the composite pure state and $K$ is the minimal dimension of the subsystems, i.e., $K={\rm min}({\rm dim}(\mathcal{H}_{A}),{\rm dim}(\mathcal{H}_{B}))$. Apart from the analytical entanglement measure (27) for bipartite states, it is also worth mentioning that efficient numerical methods are available for quantifying quantum entanglement for multipartite states [56].

TABLE V shows the selected most and least 2-grams of words, ranked by the von Neumann entanglement entropy. The most entangled pairs are mostly set phrase or some well-known combinations of words, e.g., how long. However, is cataracts is clearly not a set phrase or well-known combination of words. We found that is cataracts appears many times, and cataracts is always next to is in this particular training dataset. This may be the reason why the learned model takes is cataracts as a fixed combination of words. The least entangled pairs consist of words with fixed semantic meaning such as names $\{$Quarry, China$\}$ and interrogatives $\{$what, who$\}$, some of which appear only once in the dataset. In other words, there is not so much semantic ambiguity or superposition with these combinations that demands a quantum probabilistic interpretation.

TABLE VI shows the selected most and least entangled 3-grams, whose von Neumann entanglement entropies are calculated to indicate the entanglement between the first two words, and the remaining word (the third word). Similar to 2-grams, the most entangled 3-grams are fixed collocations and combinations that often appear together in the training set, such as a set phrase. Interestingly, punctuation marks also appear in the most entangled 3-grams. For example, entanglement in at times . is large, which implies that the phrase at times is often at the end of the answers. However, there worldwide ? is among the least entangled combinations, since there worldwide is not in any of the question sentences for training. The third word in the least entangled 3-grams are mainly names and numbers, which cannot combine with the first two words to form a fixed phrase.

In Fig. 7, we also visualize the entanglement entropy in some selected sentences, where the degree of darkness indicates the level of entanglement. It can be seen that words with multiple meanings in different contexts, e.g., $\{$is, does, the, film$\}$, have greater capabilities to entangle with their neighboring words.

In this paper, we proposed an interpretable quantum-inspired language model with a novel EE module in the neural network architecture. The EE enables the modelling of the word sequence by a general quantum pure state, which is capable of capturing all the classical and non-classical correlations between the word states. The expressivity of the neural network is greatly enhanced by cascading the word embedding and EE modules. The complex-valued model has demonstrated superior performance on the QA datasets, with much smaller word embedding dimensions compared to previous QLMs. In addition, the non-classical correlations between the word states can be quantified and visualized by appropriate entanglement measures, which improves the post-hoc interpretability of the learned model.

The future plan is to apply the adaptive methods [57, 58] for optimizing the virtual measurement operations to increase the efficiency in feature extraction. The QLM-EE model is expected to be more powerful on huge datasets in which the semantic meanings of words and their correlations are far more complex. With a larger dataset and richer semantic superpositions between the words, several entanglement embedding modules can be cascaded to form a deeper neural network, which could encode the multipartite correlations within the text at different scales.