Automatic Recognition of the General-Purpose Communicative Functions defined by the ISO 24617-2 Standard for Dialog Act Annotation

ISO 24617-2, the ISO standard for dialog act annotation (?, ?) aims at setting the ground for more comparable research in the area. According to it, dialog act annotations should not be performed on turns or utterances, but rather on functional segments (?). Furthermore, the annotation of each segment does not consist of a single label or set of labels, but rather of a complex structure containing information about the participants, relations with other functional segments, the semantic dimension of the dialog act, its communicative function, and optional qualifiers concerning certainty, conditionality, partiality, and sentiment.

The standard defines nine semantic dimensions – Task, Auto-Feedback, Allo-Feedback, Turn Management, Time Management, Discourse Structuring, Own Communication Management, Partner Communication Management, and Social Obligations Management – in which different communicative functions may occur. These communicative functions are the standard equivalent of the dialog act labels used to annotate dialogs before the introduction of the standard. They are divided into general-purpose and dimension-specific functions. The former can occur in any semantic dimension and are organized hierarchically as shown in Figure 1. The latter can only occur in the corresponding dimension. The latter can only occur in the corresponding dimension and are distributed as shown in Table 1. Of the nine semantic dimensions, only the Task dimension does not have specific functions. This means that only general-purpose communicative functions occur in that dimension. Furthermore, with the exception of the functions specific to the Social Obligations Management dimension, which can be split into their initial and return counterparts, dimension-specific functions are all at the same level.

Annotating all of the aspects defined by the standard is an exhaustive process and, consequently, the amount of available data is still reduced and, in many cases, not all of the aspects are considered (e.g. ?, ?, ?, ?). To the best of our knowledge, the DialogBank (?, ?) is the only publicly available source of dialogs fully annotated according to the standard. It features (re)-annotated dialogs from several corpora, but, currently, there are only 15 dialogs in English and 9 in Dutch, which amount to less than 3,000 segments.

Given a turn, utterance, or functional segment in a dialog, to which we will refer generically as segment in the remainder of the article, automatic dialog act recognition aims at identifying the intention behind that segment. It is a task that has been widely explored over the years, using both classical machine learning and deep learning approaches. In both cases, the approaches differ mainly on how the representation of a segment is generated from the representations of its tokens and how they are able to weigh context information in the decision process. The article by ? (?) provides a comprehensive overview on classical machine learning approaches on the task, except for the more recent Support Vector Machine (SVM)-based approaches (?, ?).

Regarding deep learning approaches, both Recurrent Neural Networks (e.g ?, ?, ?, ?) and Convolutional Neural Networks (e.g. ?, ?, ?) have been used to generate segment representations by combining the embedding representations of their words. While the first focus on capturing information from relevant sequences of tokens, the latter focus on the context surrounding each token and, thus, on capturing relevant token patterns independently of where they occur in the segment. We have compared different RNN- and CNN-based segment representation approaches and achieved higher performance on the task using a set of parallel CNNs with different window sizes, while also consuming less resources than when using RNNs (?). Still, most of the more recent studies on the task rely on bidirectional RNNs for segment representation (e.g. ?, ?, ?, ?).

Regarding the representation of the segment’s tokens, most approaches on dialog act recognition using deep learning have relied on pre-trained word embedding representations generated by Word2Vec (?) or GloVe (?). However, similarly to what happens on most NLP tasks, using contextualized word representations, especially those generated by BERT (?), leads to higher performance (?). Additionally, there are studies which also rely on character-level information (e.g. ?, ?, ?, ?). In our studies on that matter, we observed that there are complementary cues for intention at a sub-word level which cannot be captured when relying solely on word-level tokenization (?, ?). Finally, functional-level tokenization, mainly in the form of Part-of-Speech (POS) tags, has also been explored for the task (?, ?). However, it seems to be less relevant.

In terms of context information, previous studies have observed performance improvements when using information regarding turn-taking by the speakers (?, ?) and the topics covered by the dialog (?). Still, the most important source of context information is the surrounding segments. Studies dedicated to that matter have shown that the influence of preceding segments decreases with the distance and that their dialog act classification is more informative than their words, even when the classifications are obtained automatically (?, ?). Furthermore, we have shown that sequentiality information and long distance dependencies between the preceding segments can be captured by using a RNN to generate a summary of their classifications (?).

In fact, considering the dependencies between the multiple segments in a dialog, several studies attempted to predict the sequence of dialog acts in a complete dialog by approaching the task as a sequence labeling problem (e.g. ?, ?). These cases rely on a hierarchical approach in which the representations of the segments are provided to a conversation-level RNN that models the whole dialog. Furthermore, the performance can be improved by including attention mechanisms that identify the information in the surrounding segments that is most relevant for predicting the dialog acts (e.g. ?, ?, ?, ?). ? (?) also observed that propagating uncertainty information concerning the previous predictions can lead to the prediction of better dialog act sequences. Finally, most of the studies that approach the task as a sequence labeling problem also rely on Conditional Random Fields (e.g. ?, ?, ?, ?) or generative models (?) as the final layer to predict the sequence of dialog acts. This way, the prediction of the dialog act for a segment is further conditioned to the previous predictions. Overall, this kind of approach achieves the highest performance on the task. However, the conversation-level RNN is typically bidirectional. This means that the prediction of the dialog act for a segment relies not only on the information from previous segments, but also from future ones, which are not available to a dialog system.

Additional studies on dialog act recognition explore alternative approaches or focus on specific applications. For instance, ? (?) approached the task as a Question Answering (QA) problem and employed adversarial training. On the other hand, ? (?) focused on developing dialog act recognition models that can be used in mobile applications.

Regarding the recognition of the communicative functions defined by the ISO 24617-2 standard, to the best of our knowledge, besides us, only ? (?) have addressed that problem. More specifically, they have explored the recognition of a compressed set of eight communicative functions on the TRADR corpus (?). This compressed set merges functions in the Task, Turn Management, and Feedback dimensions and does not consider the hierarchical nature of general-purpose functions. Thus, the task was approached as a flat classification problem. The authors compared the performance of several Deep Neural Network (DNN) architectures and uncontextualized embedding approaches and observed the highest performance when the representation of the segment was generated by passing GloVe embeddings through a Long Short-Term Memory Unit (LSTM). However, the use of parallel CNNs was not explored and only large window sizes were considered. Furthermore, similarly to what was observed in previous studies on dialog act recognition, using context information regarding the dialog history led to improved performance. However, in this case, it was summarized as the average of the embedding representations of all the words in the dialog history.

Given the reduced amount of dialogs in the DialogBank, we have mapped the dialog act labels of the LEGO corpus (?) into the communicative functions defined by the ISO 24617-2 standard. To assess the utility of these partially annotated dialogs, we have performed preliminary experiments on the automatic recognition of general-purpose communicative functions in the DialogBank (?). In those experiments, we flattened the hierarchy and addressed the task as a flat dialog act recognition problem using the same approach we used to predict the original dialog act labels of LEGO corpus (?). Then, we observed performance improvements when the LEGO dialogs were included in the training phase, in comparison to when the classifier was trained solely on DialogBank dialogs.

As discussed in Section 2.1, to the best of our knowledge, the DialogBank (?, ?) is the only publicly available source of dialogs annotated fully according to the ISO 24617-2 standard guidelines. Thus, in our experiments, we use it as gold standard for evaluating the performance of the different approaches. However, given the scarcity of annotated dialogs, we also rely on the LEGO-ISO dataset (?), which features dialogs that were partially annotated with the communicative functions defined by the ISO 24617-2 standard through label mapping processes. Finally, we also rely on Switchboard Dialog Act Corpus (?), which is the most explored in dialog act recognition studies, to train models for transfer learning purposes. These datasets are described in further detail below.

In this study, we focus on the recognition of ISO 24617-2 general-purpose communicative functions in the DialogBank. Below, we describe the evaluation scenarios, the evaluation approach, and the metrics.

As discussed in Section 3, our approach to deal with the hierarchical problem posed by the general-purpose communicative functions defined by the ISO 24617-2 standard can be applied on top of any approach to generate segment representations. In our experiments, we relied on the same approach used in our study that explored the multiple aspects that contribute to dialog act recognition (?). This way, we know that the segment representation approach is able to capture information regarding intention and we can use the models trained during that study in transfer learning processes.

Figure 4 shows the complete architecture of the network used in our experiments. Two representations of the segment are generated in parallel, one based on its characters and another on contextualized embedding representations of its words, generated by BERT (?). In both cases, the representation of the segment is generated by concatenating the outputs of three parallel CNNs with different window sizes followed by a max-pooling operation. At the character level, we use windows of size three, five, and seven, in order to focus on affixes, lemmas, and inter-word relations. At the word level, we use windows of size one, two, and three, in order to focus on independent words and short word patterns. The two representations are then concatenated and decorated with context information.

Regarding context information, considering that the number of dialogs in the DialogBank is small, we do not rely on a summary of the whole dialog history as in the original approach, because it is prone to overfitting. Instead, we use a flattened sequence of classifications and turn-taking information of the three preceding segments, which have been proved the most important in previous studies (?, ?). The classification of each preceding segment is represented as a concatenation of the one-hot representations of the communicative functions at each level of the hierarchy. Turn-taking information is provided as flags stating whether the speaker changed.

The specialization and output layers are as described in Section 3. That is, there are dedicated specialization and output layers per each level in the hierarchy, the specialization layer of each level also considers the output at the upper levels, and each output layer considers an additional class representing the lack of communicative function at that level.

In order to take advantage of the intention information captured by a dialog act recognition model trained on a large corpus, we apply a transfer learning process. More specifically, we preset the weights of the segment representation layers of our hierarchical model using the corresponding weights of the flat dialog act recognition model. Consequently, only the specialization and output layers are trained on the dialogs annotated according to the ISO 24617-2 standard. This way, the segment representations provide generic information regarding intention that is then specialized for the distinction among general-purpose communicative functions at each level.

To implement our classifiers, we used Keras (?) with the TensorFlow (?) backend. To update the weights during the training phase, we used the Adam optimizer (?) with the default parameterization and mini-batches with size 512. To decide when to stop training, we used early stopping with 10 epochs of patience. That is, the training phase of each classifier stopped after ten epochs without improvement on the validation set composed of the corresponding fine-tuning dialog(s).

To obtain contextualized word representations, we used the output of the last layer of the large uncased BERT model (?). When using character-level tokenization, embedding representations of the characters were trained together with the network to capture relations between them. To generate the segment representations, we used 100 filters in each CNN and aggregated the results using the max-pooling operation. Finally, the specialization layers were implemented as Rectified Linear Units (?) with 200 neurons and 50% dropout probability (?).

For transfer learning purposes, we relied on the model that achieved the top performance on the Switchboard Dialog Act Corpus in our study on dialog act recognition (?). Thus, the weights of the CNNs and the character-level embedding layer were set to those of that model and fixed during the training phase.

Table 5 shows the results of our experiments on the automatic recognition of general-purpose communicative functions in segments that have a communicative function in the Task dimension. When evaluating using leave-one-dialog-out cross-validation, we can see that, on average, the hierarchical classifier outperformed the flat one by 3.49 percentage points in terms of exact match ratio and 2.74 in terms of hierarchical F-measure. This suggests that the per-level specialization layers are able to capture the information that is most important for distinguishing the communicative functions at a given level and that the output cascade is able to capture information regarding the hierarchical dependencies between the communicative functions at different levels.

When evaluating using leave-one-corpus-out cross-validation, we can observe similar patterns. However, the average exact match ratio and F-measure of the best approach decrease by 18.98 and 11.52 percentage points, respectively. In addition to the lower number of segments used to train the classifiers, part of this drop in performance is explained by the fact that, as discussed in Section 4.1, the corpora featured in the DialogBank have different characteristics and, thus, some communicative functions mostly or only occur in one of the corpora. Consequently, the drop is slightly higher in terms of recall (11.67 percentage points) than precision (11.33 percentage points). This reveals the importance of having a representative amount of dialogs for training, whic covers all the communicative functions.

Overall, by looking at the different metrics, we can see that there is a gap of at least 18.21 percentage points between the results in terms of exact match ratio and hierarchical F-measure. This shows that even when the classifiers fail to predict the correct communicative function of a segment, they still predict part of the path correctly. Furthermore, the higher precision than recall suggests that the classifiers avoid predicting communicative functions that are deeper in the hierarchy. To confirm this, we randomly selected one of the runs of the hierarchical approach and looked into the full communicative function paths predicted for the segments. This way, we were able to assess the performance of the classifier on each level of the hierarchy independently, as shown in Table 6. We can see that, when all segments are considered, the lowest exact match ratio is on Level 4, which is the one with the highest number of possible communicative functions. However, if we do not include segments that should be classified as None in the computation of the exact match ratio, we can see that it decreases as the depth increases. Furthermore, in the levels in which there are segments misclassified as None, this kind of misclassification is predominant in relation to confusion among the actual communicative functions of the level. On the other hand, when only considering the segments that should be classified as None, the recall is higher than precision and increases with depth. This confirms the bias of the classifier towards the prediction of shallower functions. Overall, this can be justified by the reduction in number of labeled examples with depth, which biases the deeper output layers towards the prediction of the None label. Still, as previously discussed, the impact of this issue can be attenuated by training the approach on set of dialogs with more extensive coverage of the communicative functions that are deeper in the hierarchy.

Table 7 shows the results of our experiments which also considered the automatic recognition of the segments that have a communicative function in the Task dimension. We can see that, in this scenario, the hierarchical approach still outperforms the flat one in terms of hierarchical F-measure by 2.24 and 1.99 percentage points when evaluating using leave-one-dialog-out and leave-one-corpus-out cross-validation, respectively. However, the performance of both approaches is similar in terms of exact match ratio. Furthermore, the wide gap between exact match ratio and hierarchical F-measure no longer exists, with a maximum difference of 2.22 percentage points. In fact, when using the flat approach, the exact match ratio surpasses the hierarchical F-measure.

Overall, the exact match ratio is higher than when the segments that do not have a communicative function in the Task dimension are not considered because, in most cases, they are easy to identify. On the other hand, the F-measure is lower since misclassifying a segment as not having communicative functions in the Task dimension means that all the functions in the correct path are missed. Additionally, since the segments without communicative functions in the Task dimension have the None label in every level of the hierarchy, it is expected that the classifiers become even more biased towards the prediction of shallower communicative functions.

Looking at the results, we can see that the performance drop is higher in terms of recall than precision, which is suggestive of the bias towards the prediction of shallower communicative functions. To confirm it, we also looked into the per-level results of one of the runs of the hierarchical approach. These results are shown in Table 8. In comparison with the results in Table 6, we can see that the exact match ratio on segments that have a communicative function on the Task dimension is lower and the percentage of error due to misclassifications with the None label is higher. On the other hand, the performance on the detection of segments that do not have general-purpose communicative functions at each of the levels is higher not only in terms of recall, but also in terms of precision. While the former is explained by the bias of the classifiers, the latter is explained simply by the addition of the segments with no communicative function in the Task dimension, which account for over half of the segments in the DialogBank.

As discussed in Section 5.1, we believe that the bias towards the prediction of shallower communicative functions can be attenuated by training the hierarchical approach on a sufficiently large amount of dialogs with representative coverage of the communicative functions that are deeper in the hierarchy. Still, in an attempt to attenuate the bias without relying on additional annotated data, we explored the use of a two-step approach which uses a binary classifier to identify the segments which have a communicative function in the Task dimension before applying the hierarchical approach to those segments.

In Table 7, we can see that the two-step approach achieves the highest performance in terms of exact match ratio. However, it is outperformed by the hierarchical approach in terms of hierarchical F-measure. The average performance of the binary classifier in terms of exact match ratio is of 85.52 percentage points when evaluating using leave-one-dialog-out cross validation and of 84.05 percentage points when evaluating using leave-one-corpus-out cross-validation. These values are in line with those achieved by the hierarchical approach on the top level. Thus, the differences between the two approaches can be explained by two factors. On the one hand, the hierarchical part of the two-step approach is trained solely on the segments that have communicative functions in the Task dimension. Thus, it is less biased towards the prediction of the None label in every level, which improves the performance in terms of exact match ratio on those segments. On the other hand, an incorrect decision of the binary classifier cannot be corrected by the predictions on the lower levels using MAP prediction. Thus, these misclassifications at the top level have a more prominent impact on the performance of the approach in terms of hierarchical F-measure.

When evaluating using leave-one-dialog-out cross-validation, the difference between the hierarchical and two-step approaches is of just 0.22 and 0.15 percentage points in terms of average exact match ratio and hierarchical F-measure, respectively. This suggests that the overall performance of both approaches is actually similar. Still, when evaluating using leave-one-corpus-out cross-validation, the differences between the two approaches are more noticeable, with a 0.70 percentage-point difference in favor of the two-step approach in terms of exact match ratio and a 0.50 percentage-point difference in favor of the hierarchical approach in terms of hierarchical F-measure. Thus, the selection of the best approach for the automatic recognition of general-purpose communicative functions is dependent on the metric that is most relevant for subsequent tasks.

In order to assess the importance of the multiple components of our hierarchical approach to the automatic recognition of ISO 24617-2 general-purpose communicative functions, we performed a set of ablation studies, in which one of the components was removed and the performance was compared with that of the full approach. These studies can be split into two categories: those regarding the architecture of the approach and those regarding data and transfer learning aspects. The experiments were performed in the scenario that targets the recognition of general-purpose communicative functions in segments that have a communicative function in the Task dimension and were evaluated using leave-one-dialog-out cross-validation. Table 9 shows the results of these experiments, with the first block showing the performance of the full approach for comparison.

The second block of Table 9 shows the results achieved when one of the components of the architecture is removed. We can see that the performance is negatively impacted in every case. By removing the connections between the output at each level and those below it, the average performance decreased by 3.33 percentage points in terms of exact match ratio and 2.12 percentage points in terms of hierarchical F-measure. Furthermore, while the performance drops in terms of both precision and recall, the drop is higher in terms of the latter. This suggests that the cascading outputs help attenuating the bias towards the prediction of shallower communicative functions.

By removing the specialization layers, the impact on performance is higher than that of removing the cascading outputs. More specifically, the average performance drops 5.60, 2.94, 3.38, and 3.18 percentage points in terms of exact match ratio and hierarchical precision, recall, and F-measure, respectively. This means that the specialization layers are able to capture the information present in the representation of the segment decorated with context information that is most relevant to distinguish among the communicative functions at the corresponding level of the hierarchy.

Finally, by replacing the MAP prediction approach with a top-down iterative one that limits the choices at each level of the hierarchy by only considering the communicative functions that are children of that selected for the level above it, the performance decreased 2.35 percentage points in terms of exact match ratio and 1.19 in terms of F-measure. This confirms that the MAP prediction approach can attenuate the impact of misclassifications in the top levels by relying on the distributional outputs obtained using softmax to obtain a joint prediction for all levels of the hierarchy.

The last block in Table 9 shows the results achieved without pre-training the segment representation layers for dialog act recognition on the Switchboard Dialog Act Corpus and without considering the dialogs of the LEGO corpus during the training phase. First of all, we can see that using transfer learning improves the performance in terms of every metric, especially exact match ratio, which improves by 6.70 percentage points. This shows that, given the reduced amount of data, training the layers that generate the segment representations solely on the dialogs annotated according to the ISO 24617-2 standard leads to overfitting. On the other hand, since the Switchboard Dialog Act Corpus is sufficiently large, a model trained on its dialogs generates representations that capture information regarding generic intention that is not specific to a single set of labels. The specificities of different sets are then captured by the specialization and output layers, leading to the generation of models that have higher generalization potential.

On the other hand, the performance increased when the LEGO dialogs were not considered during the training phase, leading to an average exact match ratio of 70.21% and an average hierarchical F-measure of 86.73%. This is contrary to what we first observed when studying the mapping of the original dialog labels of the LEGO corpus into the communicative functions defined by the ISO 24617-2 standard and the use of those dialogs to help in the recognition of the communicative functions in the DialogBank (?). This can be explained by several factors. First, on that study, we did not rely on BERT word embeddings nor on transfer learning processes to obtain segment representations. Thus, in that case, the LEGO dialogs were important for generating more generic segment representations and avoiding overfitting. On the other hand, when those dialogs only contribute for the training of the specialization and output layers as in this study, they have a negative impact because the LEGO dialogs have different characteristics than those in DialogBank, especially in terms of the user utterances. Second, considering that the hierarchical approach has one pair of specialization and output layers per level, it has more trainable parameters than the flat one. Consequently, it is more prone to overfitting. That is relevant in this context since the LEGO dialogs are in larger number, they are highly repetitive, and are mostly covered by a small set of general-purpose communicative functions. Last, we were not focusing on the generalization ability of the classifiers in that study. Thus, the cross-validation process fine-tuned the classifiers to achieve the highest performance on the test dialog of the corresponding fold. Given this fine-tuning of the classifiers, they were less prone to overfit to the specificities of the LEGO dialogs.