RheFrameDetect: A Text Classification System for Automatic Detection of Rhetorical Frames in AI from Open Sources

Advances in Artificial Intelligence (AI) have at times been described as an AI development race among major nations in the world. World leaders, such as Russian President Vladimir Putin, Chinese President Xi Jinping and Chief Technology Officer of The Trump Administration have issued statements in the past calling for greater prioritization, swift action and increased investment in order to gain strategic advantage in this AI development race. This paper focuses on how AI development is framed as a competition, either between nations or private companies. Our goal is to capture the Rhetorical Frames of Competition between governments or companies in the field of Artificial Intelligence (AI). This Frame may be expressed in military terms (such as, arms race, Cold War, battle for supremacy, etc.) or non-military (such as, win, versus, race, compete with, etc.). In Table 1, we show some examples and counter-examples of Rhetorical Frames as well as the rationale behind them. For a more detailed understanding of Rhetorical Frames, please refer to Imbrie et al. (Imbrie et al., 2020)

With massive online availability of open sources (online news media, twitter, blogs), identifying Rhetorical Frames using human annotators can be very limited in scope as well as difficult to scale in real-time. Detecting Rhetorical Frames from open sources in (near) real-time can assist policy makers or analysts in assessing whether attitudes of world governments or private companies towards AI development are becoming more cooperative or competitive over time. However, these textual open sources are unstructured (noisy) and therefore, pose challenges in identifying the occurrence of Frames.

The availability of massive textual open sources coincides with recent developments in language modeling, such as Word2Vec (Mikolov et al., 2013a; Mikolov et al., 2013b), GloVe (Pennington et al., 2014), Doc2Vec (Le and Mikolov, 2014), Sentence-BERT (SBERT) (Reimers and Gurevych, 2019) and others. These neural network based language models when trained over a representative corpus can be used to convert words as well as any piece of text (documents, paragraphs and sentences) to dense low-dimensional vector representations, most popularly known as embeddings. These embeddings are then provided as input to downstream classifiers for classifying a piece of text.

Motivated by these techniques, we develop RheFrameDetect, a text classification system for automatic identification of Rhetorical Frames from textual open sources. Our main contributions are as follows:

• •

  Automated: RheFrameDetect is fully automatic. Given an input news article, it will automatically locate all the occurrences of Frames mentioned in the context of AI.

• •

  Novelty: To the best of our knowledge, there has been no prior systematic efforts at automatic detection of Rhetorical Frames from open sources.

• •

  Real-time: RheFrameDetect is an end-to-end system and can be deployed in a (near) real-time setting.

• •

  Evaluation: We present a detailed and prospective analysis of RheFrameDetect by evaluating the accuracies of text classifiers employed at document as well as paragraph level against manually annotated Frames at the corresponding level.

• •

  Case Studies: We also present multiple case studies providing insights into how Frames located by RheFrameDetect in news articles compare against the human annotated ones.

In this paper, we intend to focus on detecting as well as locating Rhetorical Frames mentioned in the context of AI within an incoming news article. RheFrameDetect aims to solve both these problems by performing multiple tasks in a hierarchical fashion as given below.

• •

  DocContainsAI: If the incoming article contains a discussion of AI, then RheFrameDetect assigns a value of Yes to DocContainsAI. Otherwise, RheFrameDetect assigns the value of No and do not proceed for the document.

• •

  DocContainsFrame: If the incoming article contains at least one occurrence of Frames, then RheFrameDetect assigns a value of Yes to DocContainsFrame. Otherwise, RheFrameDetect assigns the value of No and do not proceed further.

• •

  ParContainsFrame: If the incoming document contains both a discussion of AI and at least one occurrence of Frames, RheFrameDetect aims to identify the paragraphs of the article which contain the Frames. Therefore, for each paragraph, RheFrameDetect assigns a value of Yes to ParContainsFrame if the paragraph contains at least one occurrence of Frames. Otherwise, RheFrameDetect assigns No to the paragraph.

• •

  FrameSpan: For the paragraphs assigned Yes for ParContainsFrame, RheFrameDetect identifies all occurrences of the Frames mentioned in the paragraph.

A flowchart depicting the tasks performed by RheFrameDetect for an incoming article is shown in Figure 1.

In this section, we provide a brief description of the techniques used by RheFrameDetect for solving the tasks shown in Figure 1.

In this section, we provide a brief description of our experimental setup, including the dataset of news articles, the annotation process and the classifiers used by RheFrameDetect for the tasks DocContainsFrame and ParContainsFrame.

In this section, we try to ascertain the efficacy and applicability of RheFrameDetect by investigating some of the pertinent questions related to the problem of automated detection of Rhetorical Frames.

Doc2Vec based Classifiers vs SBERT based classifiers - Which is a better method for detecting Rhetorical Frames at the document level?

In Tables 4 and 5, we evaluated Doc2Vec and SBERT based classifiers for the binary classification task DocContainsFrame. For DocContainsFrame, we are mostly interested in the performance metrics (Precision, Recall and F1-score) for Yes class as it is the minority class and therefore, hard to predict. Moreover, performance metrics for the Yes class will inform us whether classifiers can detect presence of Frames at the document level. We are also interested in the macro-average metrics across the classes. Overall, in terms of F1-score for Yes class and macro-average, Multi-layer Perceptron with embedding DV-DBOW-NEG and SVM with embedding SBERT emerge out to be the best performing classifiers. In terms of Recall, Logistic Regression consistently emerges out to be the best classifier but performs worse in terms of Precision leading to low F1-score than SVM and Multi-layer Perceptron. Random Forest outperforms other classifiers in terms of Precision but performs worse in terms of Recall leading to very poor F1-score compared to SVM and Multi-layer Perceptron.

If we compare Doc2Vec vs SBERT in terms of the F1-score, we do not notice any significant performance difference. However, in terms of Recall, Logistic Regression performs better with Doc2Vec embeddings than SBERT. Similarly, for Precision, Random Forest performs better with Doc2Vec embeddings than SBERT. This depicts that Doc2Vec generates slightly better embeddings at the document level than SBERT. Since SBERT technique is trained over a data set of nearly billion sentences, it may not be the ideal technique for generating embeddings at the document level.

Paragraph Vector and SBERT based Classifiers vs Self-Attention based Classifiers - Which is a better method for detecting Rhetorical Frames at the paragraph level?

In Tables 6,  7 and  8, we evaluated Paragraph Vector and SBERT based classifiers against the Self-Attention based classifiers for the binary classification task ParContainsFrame.

Firstly, we compare Paragraph Vector based classifiers against SBERT based classifiers for this task. Similar to DocContainsFrame, we are mostly interested in the performance metrics (Precision, Recall and F1-score) for Yes class as it is the minority class and therefore, hard to predict. Moreover, performance metrics for the Yes class will inform us whether classifiers can detect presence of Frames at the paragraph level. As can be clearly seen, SVM with SBERT embeddings outperform all the other techniques in terms of F1-Score for the Yes class as well as macro-average. Overall, with both Doc2Vec and SBERT embeddings, Multi-layer Perceptron emerges out to be the best performing classifier in terms of F1-Score followed by SVM. Similar to DocContainsFrame, Logistic Regression is the best performing classifier in terms of Recall for ParContainsFrame and Random Forest outperforms other classifiers in terms of Precision.

When we compared Paragraph Vector against SBERT across the four classifiers, SBERT based classifiers provide better F1-score than the corresponding Paragraph Vector based classifiers. SBERT model is trained over a dataset of billion sentences in comparison to Paragraph Vector trained over our corpus dataset of nearly 1 million sentences and therefore, SBERT model is expected to generate better quality embeddings for sentences/paragraphs than Paragraph Vector.

Finally, we compared the Self-Attention based classifiers (SelfAttention and GuidedSelfAttention) against the Paragraph Vector and SBERT based classifiers. As expected, Self-Attention based classifiers (specifically GuidedSelfAttention) outperform Paragraph Vector and SBERT based classifiers in terms of Recall and F1-Score due to the powerful Self-Attention mechanism guided by the FrameSpan annotations. For the Yes class of ParContainsFrame, GuidedSelfAttention achieves the highest Recall (0.87) and F1-Score (0.81) among all the classifiers. Superior performance of Self-Attention based classifiers is expected as they are sequence based models. Therefore, they learn the sequential patterns of Frames in paragraphs more efficiently compared to the Paragraph Vector based classifiers. Moreover, Self-Attention mechanism provides dual benefits to the classifier: not only does it result in better classification metrics, but it also drives the classifier to locate the most relevant words in paragraphs/sentences which contribute to the classification decision.

Finally, we also depict five case studies related to FrameSpan identification after ParContainsFrame classification. For each case study, we chose a paragraph from the test dataset of ParContainsFrame classification and then, we visualized the attention weights of tokens in each paragraph assigned by SelfAttention and GuidedSelfAttention models. In Table 3, we show the selected paragraphs and the FrameSpan mentioned within each paragraph. In Figure 2, we show the visualization of the attention weights for both SelfAttention and GuidedSelfAttention. As can be seen clearly in Figure 2, GuidedSelfAttention is able to locate the tokens within FrameSpan more prominently in comparison to SelfAttention. Specifically, in paragraphs corresponding to case studies 2 and 5, GuidedSelfAttention captures accurately the FrameSpan ‘compete’ and ‘competition’ respectively. This is expected as the attention weights in GuidedSelfAttention are supervised based on the FrameSpan annotations. This motivates GuidedSelfAttention to apply more attention to the portions in paragraphs which may contain FrameSpan. On the other hand, attention weights in SelfAttention model are unsupervised, i.e. it does not have any information about the FrameSpan location in paragraphs. Therefore, GuidedSelfAttention not only outperforms SelfAttention in terms of the metrics (Recall and F1-Score) for the Yes class in ParContainsFrame but also has the capability to accurately locate FrameSpan information within the paragraph leading to enhanced intelligibility from a human analyst perspective.

In this manuscript, we have introduced RheFrameDetect, the first automated end-to-end system for detecting and locating Rhetorical Frames in AI from Open Sources. RheFrameDetect takes an incoming news article as input and performs multiple tasks (Text Pre-processing, Keyword matching and binary classifications) in a hierarchical manner. Binary classification tasks performed by RheFrameDetect were used to detect Frames at both document level (DocContainsFrame) and paragraph level (ParContainsFrame). We evaluated a wide variety of classifiers for both these tasks. For DocContainsFrame classification task, we found that Multi-layer Perceptron and SVM emerge out to be the best performing classifiers in terms of F1-Score for the Yes class and macro-average. In terms of Recall and Precision for DocContainsFrame, Doc2Vec based classifiers perform slightly better than SBERT based classifiers indicating better quality of embeddings generated at the document level by Doc2Vec than SBERT. For the ParContainsFrame classification task, GuidedSelfAttention classifier outperforms all other classifiers achieving a macro-average F1-Score of 0.90. For the Yes class, GuidedSelfAttention achieves the highest Recall of 0.87 and F1-Score of 0.81. We showed in Figure 2 that GuidedSelfAttention also has the capability to accurately locate FrameSpan information within the paragraph compared to SelfAttention. Superior performance of GuidedSelfAttention can be attributed to the fact that it directs the Self-Attention mechanism to put more emphasis (higher weights) on words/tokens within FrameSpan during training, ignoring the other words/tokens in paragraphs.

Our future work will focus on expanding RheFrameDetect to detect the purpose of Rhetorical Frames in AI, such as Motivation, Critique, Structured and Explanation. We also aim to quantify the FrameSpan identification task in terms of Precision, Recall and F1-Score using rankings based on attention weights of words/tokens in each paragraph. Finally, we also aim to extract named entities mentioned in the context of Frames within a paragraph so that we can provide human analysts more context around the detected Frames.

This work was performed under contract with the Center for Security and Emerging Technology (CSET) at Georgetown University, contract number CON-0011700. We would also like to thank SPi Global for providing us AI Rhetorical Frame annotations at the document and paragraph level.