Building Dialogue Understanding Models for Low-resource Language Indonesian from Scratch

ID-WOZ is constructed with the goal of obtaining highly natural conversations between a customer and an agent or a query information center focusing on daily life. We consider various possible dialogue scenarios ranging from basic requests like hotel, restaurant, to a few urgent situations such as hospital or police. Our dataset consists of nine domains, namely plane, taxi, wear, restaurant, movie, hotel, attraction, hospital, police, most of which are extended domains including the sub-task Booking (with the exception of police).

We adopt the Wizard-of-Oz (Kelley, 1984) dialogue-collecting approach, which has been shown effective for obtaining a high-quality corpus at relatively low costs and with a small-time effort. Following the successful experience of MultiWOZ (Budzianowski et al., 2018), we create a large-scale corpus of natural human-human conversations on a similar scale. Based on the given templates for several domains, the users and wizards generate conversations using heuristic-based rules to prevent the overflow of information. We design and develop a collection-annotation pipeline platform with a user-friendly structure for building up the dataset, shown in Experiment Section.

In order to accelerate and optimize the process of collection and annotating, we design and develop a pipeline platform. Our platform consists of three stages, “collection - annotation - statistics & analysis”, which are executed synchronously after the initialization process. We split a number of well-trained annotators (i.e., 80 local people, 70 of them whose mother language are ID, 10 of them are bilingual citizens, plus 2 main organizers) into two groups to produce dialogue and annotation. A quarter of annotators (i.e., 20) are trained following the templates we provide to play the wizard role. After collecting 1k dialogues initially (about one week), while the collecting conversation is still ongoing, the second group of annotators (i.e., 62) join in to work towards the detailed full-labeled corpus, including domains, actions, intents, and slots.

The quality is assured in three processes, namely “scripts checking”, “cross-checking”, and “supervisor-checking”. In particular, the scripts can filter the hypothesis cases which have potential faults such as vacant labels or malformed. For cross-checking process, the annotators are assigned not only to fresh unlabeled annotation tasks but also a few sampling labeled cases (i.e., 20%) from their peers, in a double-blind process. The cases passing the cross-checking procedure will be sampled and handed over to the supervisors (i.e., the two organizers), who are more familiar with the details of the entire annotation task to control the overall consistency and accuracy of the annotation. We adopt the inter-annotator agreement (IAA) (Fleiss et al., 1969) to measure how well our recruited annotators can make the same annotation decision for a certain category further, as follows:

where $p_{o}$ and $p_{e}$ denote the relative observed agreement among raters, and hypothetical probability of chance agreement, respectively. The average score of our dataset is 0.834.

Table 1 compares our dataset with existing datasets in Indonesian, and the English dialogue dataset MultiWOZ (Budzianowski et al., 2018). ID Chat (Koto, 2016) is the first publicly available Indonesian chat corpora, and draw a few related research on the Indonesian Language dialogue (Chowanda and Chowanda, 2017). Dyadic Chat (Tho et al., 2018) is another public chat corpus on Indonesian, which focuses on the dyadic term. Dyadic is a term that describes the relationship between two people, for instance, the romantic relationship between a couple. Compared with these small-scale datasets, ID-WOZ is the first to contain large-scale (about ten thousand dialogues in multi-domains) corpus focusing on general task-oriented chat.

MultiWOZ (Budzianowski et al., 2018) is a large-scale multi-domain task-oriented English dialogue dataset, including seven distinct domains (taxi, restaurant, hotel, attraction, hospital, police, train) and fine-labeled actions and slots in the spoken language understanding stage. Considering the regional cultural background, our collected dataset contains a few more general domains (i.e., wear, movie, plane) and more corresponding slots type, such as clothes type, movie genre, movie synopsis.

The first stage of our framework is “Bi-confidence-frequency Mixing” (BiCF Mixing). As shown in Fig. 2, we use the source language data in two steps. The first is to generate the frequency-word set ($\textbf{W}_{freq}$) of source data. The second is to obtain the word alignment with the translating-confidence ($\lambda_{conf}$) of each word and generate confidence-word set ($\textbf{W}_{conf}$). The goal of this stage is to select both frequent and high-confidence word pairs for English and Indonesian, and yield mixed data $\mathcal{T}_{mix}$.

Given the set of source sentences $\mathcal{S}=\{s_{1},s_{2},...,s_{n}\}$, we calculate TF-IDF (Salton et al., 1982; Ramos et al., 2003) for each word in the source dialogue corpus, as shown in Eq. 2:

where $\mathcal{N}(w_{i}^{s},s_{j})$ is the number of occurrences of the word $w_{i}^{s}$ in the source language sentence $s_{j}$, and the denominator ($\sum_{k}\mathcal{N}(w_{k}^{s},s_{j})$) is the sum of occurrences of all terms $w_{k}^{s}$ in the sentence $s_{j}\in\mathcal{S}$. $\left|S\right|$ represents the number of sentences and $\left|j:w_{i}^{s}\in d_{j}\right|$ denotes the number of sentences containing the word $w_{i}^{s}$. The frequency-word set $\mathbf{W}_{freq}=\langle(w_{i}^{s},r_{i}),...,(w_{j}^{s},r_{j})\rangle$ are obtained by sorting the output ($tf\texttt{-}idf_{(i,j)}$) from the TF-IDF algorithm, where $r_{i}$ denotes the frequency score.

Then we adopt small-scale high-quality parallel sentences (i.e., 1K), translated by skilled bilingual translators, to generate the alignments of words by using fast_align (Dyer et al., 2013). Given a few English sentences and their corresponding confidently translated sentences, the fast_align model uses a log-linear reparameterization of IBM Model 2 (Collins, 2011) to generate a set of confidence-word pairs $\mathbf{W}_{conf}={\langle(w_{i}^{s},w_{i}^{t}),p_{i}\rangle,...,\langle(w_{j}^{s},w_{j}^{t}),p_{j}\rangle}$ with Indonesian word and confidence score, denoted by $w_{i}^{t}$ and $p_{i}$, respectively.

As shown in Algorithm 1, after selecting the words both over the frequency threshold $\lambda_{freq}$ and the confidence threshold $\lambda_{conf}$, we then fusing words to generate the substitute words set $\mathbf{W}_{sub}$. $Thresh$ function of line 1 and 2 in Algorithm 1 are designed as Eq. 3:

where $\mathbf{W}(\cdot)$ denotes frequency-word set ($\mathbf{W}_{freq}$) or confidence-word set ($\mathbf{W}_{conf}$). $\mathcal{P}(\cdot)$ denotes frequency scores $r_{i}$ or confidence score $p_{i}$. $\odot$ is selecting the top subset operation. And $Fusion$ function in line 3 can be implemented as Eq. 4:

where $\theta$ is the hyper-parameter to adjust the ratio of two branch of word sets. Lines 4 to 13 in Algorithm 1 illustrate the mixing procedure. We incrementally substitute the source word $w^{s}$ of a temporarily copied sentence $\widehat{s}$ with the corresponding target word $w^{t}$. In this way, the mixed corpus $\mathcal{T}_{mix}$ is generated, consisting of both English words and Indonesian words.

We train and refine the initially pre-trained multilingual model (i.e., Multilingual-BERT) on the mixed corpus $\mathcal{T}_{mix}$ with annotations from the source English dataset. This operation could update the embeddings of English words as well as the Indonesian words. Therefore this stage allows our model to make use of English corpora and obtain a refined latent space to improve semantic representations. The multilingual latent space can be updated with the discriminative training process as Eq. 5:

where $\eta^{l}$ denotes the learning rate of the $l$-th layer. $\Theta_{i}^{l}$ represents the parameters of the model at $l$-th layer in $i$ step. $\nabla_{\Theta^{l}}J(\Theta)$ is the gradient of parameters $\Theta_{i}^{l}$ at $l$-th layer with regard to the model’s objective function, i.e., supervised by intent classification and slot-filling annotations in our model.

When the performance is stable on the training set (around 25 epochs in our experiments), we save the model that performs best on the validation set as the mixed refined embedding model, denoted in blue embedding space in the middle of Fig. 2. Then we feed fine-labeled Indonesian data into the mixed refined embedding model and transfer one more time to obtain a refined target-specific embedding model. In this way, by utilizing the English dataset, we generate a better representation latent space for Indonesian, i.e., encoding each sentence into $\mathbb{R}^{1\times 768}$ representation feature vector.

The decoder of our framework performs two tasks, i.e., intent classification and slot-filling sequence labeler, respectively. We make use of deep bi-directional long short-term memory (Bi-LSTM) networks and a CRF layer, as shown in Eq. 6, to predict the classifications for the input words (Chen et al., 2017; Wang et al., 2017; Chen et al., 2016; Dozat and Manning, 2016).

where $f_{l}$ is the hidden state of backward propagation and $f_{r}$ is the hidden state of forward feeding in BiLSTM, respectively. And then CRF layer is appended to decode slot classes further and generate results of the framework finally.

We take MultiWOZ (Budzianowski et al., 2018) as the English dataset and our collected ID-WOZ as the target language Indonesian dataset. As the hospital and police domains in MultiWOZ contain very few dialogues (5% of total dialogues), and only appear in the training dataset, we choose to ignore them in our experiments, following (Wu et al., 2019). The train domain is invalid in Indonesian data because it reflects the cultural difference between English and Indonesia. Therefore, we only adopt four domains as the main experiment restaurant, hotel, taxi, attraction shared by MultiWOZ and ID-WOZ. Statistics of them are shown in Table 2. In order to suit the testing set, we have to merge the annotations of English data with Indonesian dataset, thereby abandoning a few types of labels, such as reference, choice in MultiWOZ. After processing, the statistics for the four domains in two datasets are reported in Table 2. Contrastive study of differences between our dataset and similar well-known datasets are shown in Table 3 All of the experiments are evaluated on the same test set from ID-WOZ (1K dialogues, 250 dialogues for each domain), which suits the local cultural background. We use the F1 score as the evaluation metric, which is calculated by the precision rate and recall rate.

There are three branches of methods to utilize English dataset and pre-trained models, i.e., Machine Translation based (MT); Multilingual pre-trained embedding model with English corpus (MLEn); and our proposed BiCF.

1) MT. We adopt the machine translation preprocessing method and extract word embeddings (i.e., $\mathbb{R}^{1\times 768}$) by random initiation, pre-trained multilingual-BERT (ML-BERT) and ID-BERT. We also take Indonesian-fastText (ID-fastText) (Joulin et al., 2016), Transformer (Vaswani et al., 2017) and our pre-trained Indonesian-Word2vec (ID-Word2vec) into comparison. ($\mathbb{R}^{1\times 300}$)

2) MLEn. We adopt three pre-trained multilingual word embedding models in this baseline, namely multilingual fastText (ML-fastText) (Joulin et al., 2016), multilingual Word2vec (ML-Word2vec) (de Melo, 2017), and multilingual-BERT (ML-BERT) (Devlin et al., 2018). By extracting the embeddings of MultiWOZ and ID-WOZ, we encode each sentence into $\mathbb{R}^{1\times 300}$, $\mathbb{R}^{1\times 300}$ and $\mathbb{R}^{1\times 768}$ dimensions, respectively.

3) BiCF. We generate about 1.5K confident word pairs from MultiWOZ and 1K translated parallel sentences. For our method BiCF, the training process converges after 20 epochs. It reaches 91.13, 87.84/ 90.17, 82.09/ 93.37, 82.98/ 89.55, 85.54 for the F1 score of intent classification and slot-filling on the MultiWOZ validation set of restaurant, hotel, taxi, attraction domains, respectively. And then the Indonesian training data of ID-WOZ is fed to refine the Indonesian embedding model.

We feed 16K Indonesian sentences of ID-WOZ to each method and validate their performance on same test set of ID-WOZ. In our implementation, five-fold cross-validation is employed to investigate the optimal parameter setting within training datasets ($learning\_rate=e^{-3},batch\_size=64,dropout\_rate=0.1,optimizer=sgd$). To verify the stability of the proposed method, we run the experiments five times for each set of parameter settings and compare their mean performance, reported in Table 4.

We also conduct a series of experiments by feeding batches of annotated Indonesian data (i.e., 1K sentences, 2K sentences, 4K sentences, …, full-scale). We pick the results of restaurant, hotel, taxi, and attraction domains in Fig. 3 and Fig. 4, as they are widely usable domains and have the most scale of dialogue data and annotations both in MultiWOZ and ID-WOZ. The entire annotated dataset, experiment results and codes are detailedly reported in Table 5 and Code 1. Also, we conduct a comparison experiment for Multilingual-BERT (ML-BERT) and ID-BERT on all domains of full-scale ID-WOZ, as reported in Table 6.

The results of the method in Section 6.2 are shown in Table 5, with English data of MultiWOZ and 16k Indonesian data of ID-WOZ. The method of machine translation based methods (MT + ML-BERT/ ID-BERT) surpass multilingual model with English data (MLEn + ML-BERT) on the intent classification task, outperforming by about 1.21%, 1.95%; 0.51%, 1.77%; 0.73%, 1.29% and 0.59%, 0.89% on F1 score for restaurant, hotel, taxi, attraction, respectively. The main reason is that the machine translation methods enjoy much more Indonesian sentences with corresponding intention labels. However, on the slot-filling task, the machine translation methods are weaker. Because the machine translation methods suffer from invalid or mismatching labels after translation. Overall, our proposed framework (BiCF + ML-BERT / ID-BERT) performs better than others in both tasks, as we are capable of utilizing the English intention labels and correct slot-filling annotations effectively. And from Table 6, we can see that ID-BERT outperforms ML-BERT across all domains, demonstrating Indonesian-specific word-embedding model (ID-BERT) is capable of representing more information and semantic knowledge than the general multilingual model (ML-BERT) in all of domains.

The statistics line chart is shown in Fig. 3, where the four upmost sub-graphs denote intent classification, the four downmost sub-graphs denote slot-filling and the red line is the performance of ID-BERT baseline. The detailed results and all of the line charts of rest domains are in Figure 4.

1). MT methods rely heavily on the quality of translation. We conduct the BLEU (Papineni et al., 2002) test for the entire MultiWOZ, and the performance of translation is 28.46 (BLEU-5) on 30k sentences. However, during the translation of dialogue, one incorrect word would cause misunderstanding. Several examples are shown in Fig. 7. Different sentences in English may be translated from the same source sentence in Indonesian. In the first case, the true meaning is requesting “how much” but the model may misunderstand the customer’s intent into requesting the type of plane ticket. And in the second, the customer is wondering “how” to order a ticket, but the translator gives the result that the customer’s request is“request location”. Based on Fig. 3 and Fig. 4, when the scale of ID-WOZ is negligible, the machine translation has large advantage on intent classification but performs badly on slot-filling. The reason is that the MT method has the ability to adjust or reset the grammar and syntactic structure to the target language, whose characteristic leads to the bad consequences that make the English slots labels dislocated, invalid and wrong.

2). MLEn methods only learn semantic information from English data in the beginning, which causes low accuracy on intent classification than others. When feeding this model with ID-WOZ, it has weakness coming from the English data because the large-scale English data shrinks the feeding ID-WOZ data. This method has strength on slot-filling when the comparison is under small scale of ID-WOZ. Because labels of slot-filling in the English data are accurate and complete. But the performance does not improve when more ID-WOZ data is further used, which shows ML-BERT has a limitation on reaching higher performance. Overall, this method is not recommended for building stable low-resource language dialogue understanding models even with gold annotated data.

3). BiCF does not outperform machine translation when the scale of fed Indonesian data is negligible on the intent classification. When the scale of ID-WOZ data gets larger, its strength of BiCF becomes more obvious. It starts to outperform significantly better than the other methods while the Indonesian data grows. It is capable of avoiding misunderstanding caused by translation and mitigating the shrink effect of the English corpus, which makes it achieve the best performance and even better than the baseline ID-BERT, when the ID-WOZ data reaches around 16k for restaurant, hotel, taxi, attraction domains on the intent classification, i.e., 92.92%, 94.30%, 92.23%, 94.80% on F1 score, respectively. This method outperforms other methods on slot-filling when the ID-WOZ data fed is negligible. Not only it makes use of correct slot-filling annotations from the English dataset, but it can also reduce the bad effects of large-scale English corpus. The accuracy reaches 82.84%, 76.95%, 90.45%, 90.44% on F1 score for restaurant, hotel, taxi, attraction on the slot-filling, respectively. Fig. 4 reports the results of three methods trained by 16K of ID-WOZ. It shows that the cross-lingual method performs better than others when the slots need more words to describe.