Ranking Computer Vision Service Issues using Emotion

Recent advances in artificial intelligence have provided software engineers with new opportunities to incorporate complex machine learning capabilities, such as computer vision, through cloud based ‘intelligent’ web services. These new set of services, typically offered as API calls are marketed as a way to reduce the complexity involved in integrating AI-components. However, recent work shows that software engineers struggle to use these intelligent services (Cummaudo et al., 2019a). Furthermore, the accompanying documentation fails to address common issues experienced by software engineers and, often, engineers resort to online communication channels, such as, JIRA and Stack Overflow (SO) to seek advice from their peers (Cummaudo et al., 2019a). While seeking advice on the issues, software engineers tend to express their emotions (such as frustration or confusion) within the questions. Recognising the value of considering emotions, other researchers have investigated emotions expressed by software developers within communication channels (Ortu et al., 2016) including Stack Overflow (SO) (Novielli et al., 2018; Calefato et al., 2018); the broad motivation of these works is to generally understand the emotional landscape and improve developer productivity (Murgia et al., 2014; Ortu et al., 2016; Gachechiladze et al., 2017). However, previous works have not directly focused on the nature of emotions expressed in questions related to intelligent web services. We also do not know if certain types of questions express stronger emotions. The machine-learnt behaviour of these cloud intelligent services is typically non-deterministic and, given the dimensions of data used, their internal inference process is hard to reason about (Cummaudo et al., 2019b). Compounding the issue, documentation of these cloud systems does not explain the limits, nor how they were created (esp. data sets used to train them). This lack of transparency makes it difficult for even senior developers to properly reason about these systems, so their prior experience and anchors do not offer sufficient support (Cummaudo et al., 2019a). In addition, adding machine learned behaviour to a system incurs ongoing maintenance concerns (Sculley et al., 2015). There is a need to better understand emotions expressed by developers to inform cloud vendors and help them improve their documentation and error messages provided by their services. This work builds on top of recent work that explored what pain-points developers face when using intelligent services through a general analysis of 1,425 SO posts (questions) (Cummaudo et al., 2019a) using an existing SO issue classification taxonomy (Beyer et al., 2018). In this work, we consider the emotional state expressed within these pain-points, using the same data set of 1,425 SO posts. We identify the emotions in each SO question, and investigate if the distribution of these emotions is similar across the various types of questions. In order to classify emotions from SO posts, we use EmoTxt, a recently proposed toolkit for emotion recognition from text (Novielli et al., 2018; Calefato et al., 2017; Calefato et al., 2018). EmoTxt has been trained and built on SO posts using the emotion classification model proposed by Shaver et al. (1987). The category of issue was manually determined in our prior work.

The key findings of our study are:

• •

  The distribution of emotions is different across the taxonomy of issues.

• •

  A deeper analysis of the results, obtained from the EmoTxt classifier, suggests that the classification model needs further refinement. Love and joy, the least expected emotions when discussing API issues, are visible across all categories.

In order to promote future research and permit replication, we make our data set publicly available.¹¹1See http://bit.ly/2RiULgW. The paper structure is as follows: section 2 provides an overview on prior work surrounding the classification of emotions from text; section 3 describes our research methodology; section 4 presents the results from the EmoTxt classifier; section 5 provides a discussion of the results obtained; section 6 outlines the threats to validity; section 7 presents the concluding remarks.

Several studies have investigated the role of emotions generally in software development (Wrobel, 2013; Shaw, 2003; Ortu et al., 2016; Gachechiladze et al., 2017). Work in the area of behavioural software engineering established the link between software developer’s happiness and productivity (Graziotin et al., 2017). Wrobel (Wrobel, 2013) investigated the impact that software developers’ emotion has on the development process and found that frustration and anger were amongst the emotions that posed the highest risk to developer’s productivity. Recent studies focused on emotion mining from text within communication channels used by software engineers to communicate with their peers (Murgia et al., 2014; Ortu et al., 2016; Gachechiladze et al., 2017; Novielli et al., 2018). Murgia et al. (2014) and Ortu et al. (2016) investigated the emotions expressed by developers within an issue tracking system, such as JIRA, by labelling issue comments and sentences written by developers using Parrott’s framework. Gachechiladze et al. (2017) applied the Shaver framework to detect anger expressed in comments written by developers in JIRA. The Collab team (Calefato et al., 2017; Novielli et al., 2018) extended the work done by Ortu et al. (2016) and developed a gold standard data set collected from SO posts consisting of questions, comments and feedback. This data set was manually annotated using the Shaver’s emotion model. The Shaver’s model consists of a tree-structured, three level, hierarchical classification of emotions. The top level consists of six basic emotions namely, love, joy, anger, sadness, fear and surprise (Shaver et al., 1987). The subsequent levels further refines the granularity of the previous level. One of their recent work (Novielli et al., 2018) involved 12 raters to manually annotate 4,800 posts (where each post included the question, answer and comments) from SO. The same question was assigned to three raters to reduce bias and subjectivity. Each coder was requested to indicate the presence/absence of each of the six basic emotions from the Shaver framework. As part of their work they developed an emotion mining toolkit, EmoTxt (Calefato et al., 2017). The work conducted by the Collab team is most relevant to our study since their focus is on identifying emotion from SO posts and their toolkit is trained on a large data set of SO posts.

As mentioned in our introduction, this paper uses the data set reported in Cummaudo et al.’s ICSE 2020 paper (Cummaudo et al., 2019a). As this paper is in press, we reproduce a summary of the methodology used in constructing this data set methodology below. For full details, we refer to the original paper. Supplementary materials used for this work are provided for replication.¹¹footnotemark: 1 Our research methodology consisted of the following steps: (i) data extraction from Stack Overflow resulting in 1,425 questions about intelligent computer vision services; (ii) question classification using the taxonomy presented by Beyer et al. (2018); (iii) automatic emotion classification using EmoTxt based on Shaver et al.’s emotion taxonomy (Shaver et al., 1987); and (iv) manual classification of 25 posts to better understand developers emotion. We calculated the inter-rater reliability between EmoTxt and our manually classified questions in two ways: (i) to see the overall agreement between the three raters in applying the Shaver et al. emotions taxonomy, and (ii) to see the overall agreement with EmoTxt’s classifications. Further details are provided below.

Figure 1 displays the overall distribution of question types from the 1,245 posts classified in (Cummaudo et al., 2019a), when adjusted for majority ruling as per section 3.2.1. It is evident that developers ask issues predominantly related to API errors when using computer vision services and, additionally, how they can use the API to implement specific functionality. There are few questions related to version issues or self-learning.

Table 2 displays the frequency of questions that were classified by EmoTxt when compared to our assignment of question types, while fig. 2 presents the emotion data proportionally across each type of question. No Emotion was the most prevalent across all question types, which is consistent with the findings of the Collab group during the training of the EmoTxt classifier. Interestingly, API Change questions had a distinct distribution of emotions, where 31.25% of questions had No Emotion compared to the average of 42.01%. This is likely due to the low sample size of API Change questions, with only 12 assignments, however the next highest set of emotive questions are found in the second largest sample (API Usage, at 59.21%) and so greater emotion detected is not necessarily proportional to sample size. Unsurprisingly, Discrepancy questions had the highest proportion of the Anger emotion, at 7.97%, compared to the mean of 4.74%, which is indicative of the frustrations developers face when the API does something unexpected. Love, an emotion which we expected least by software developers when encountering issues, was present across the different question types. The two highest emotions, by average, were Fear (16.67%) and Surprise (14.90%), while the two lowest emotions were Sadness (4.47%) and Anger (4.74%). Joy and Love were roughly the same and fell in between the two proportion ends, with means of 8.96% and 8.16%, respectively. Results from our reliability analysis showed largely poor results. Guidelines of indicative strengths of agreement are provided by Landis and Koch (1977), where $\kappa\leq 0.000$ is poor agreement, $0.000<\kappa\leq 0.200$ is slight agreement and $0.200<\kappa\leq 0.400$ is fair agreement. Our readings were indicative of poor agreement between raters ($C_{\kappa}$$=-0.003$) and slight agreement with EmoTxt ($L_{\kappa}$$=0.155$). The strongest agreements found were for No Emotion both between each of our three raters ($L_{\kappa}$$=0.292$) and each rater and EmoTxt ($C_{\kappa}$$=0.086$), with fair and slight agreement respectively.

Our findings from the comparison between the manually annotated SO posts and the automatic classification revealed substantial discrepancies. Table 3 provide some sample questions from our data set and the emotion identified by EmoTxt within the text. A subset of questions analysed by our three raters do not indicate the automatic (EmoTxt) emotion, and upon manual inspection of the text after poor results from our reliability analysis, an introspection of the data set sheds some light to the discrepancy. For example, question 55113529 shows no indication of Joy, rather the developer is expressing a state of confusion. The phrase “Thanks for your help” could be the reason why the miss-classification occurred if words like “thanks” were associated with joy. However, in this case, it seems unlikely that the developer is expressing joy as the developer has followed a tutorial but is still encountering an error. Similarly, question 39797164, classified as Love and question 50500341, classified as Sadness express a state of confusion and the urge to know more about the product; upon inspecting the entire question in context, it is difficult to consistently agree with the emotions as determined by EmoTxt, and further exploration into the behaviour and limitations of the model is necessary. Our results indicate further work is needed to refine the ML classifiers that mine emotions in the SO context. The question that arises is whether the classification model is truly reflective of real-world emotions expressed by software developers. As highlighted by Curumsing (2017), the divergence of opinions with regards to the emotion classification model proposed by theorists raises doubts to the foundations of basic emotions. Most of the studies conducted in the area of emotion mining from text is based on an existing general purpose emotion framework from psychology (Bruna et al., 2016; Ortu et al., 2016; Novielli et al., 2018) – none of which are tuned for software engineering domain. In our our study, we note the emotions expressed by software developers within SO posts are quite narrow and specific. In particular, emotions such as frustration and confusion would be more appropriate over love and joy.

In this paper we analysed SO posts for emotions using an automated tool and cross-checked it manually. We found that the distribution of emotion differs across the taxonomy of issues, and that the current emotion model typically used in recent works is not appropriate for emotions expressed within SO questions. Consistent with prior work (Lin et al., 2018), our results demonstrate that machine learning classifiers for emotion are insufficient; human assessment is required.