CADV: A software visualization approach for code annotations distribution

The term “metadata” is used in various contexts in Computer Science. In all of them, it means data referring to the data itself. When discussing databases, while “data” refers to the domain information persisted, “metadata” refers to their description, i.e., the table’s structure. In the object-oriented context, the data are the instances, and the metadata is their description, i.e., information describing the class. As such, fields, methods, super-classes, and interfaces are all metadata of a class instance. A class field, in turn, has its type, access modifiers, and name as its metadata. When a developer uses reflection, it is manipulating the metadata of a program and using it to work with previously unknown classes [12].

Some programming languages provide features that allow custom metadata to be defined and included directly on programming elements. This feature is supported in languages like Java and C#, which are referred to as code annotations and attributes. They were integrated into the Java language in version 1.5, becoming popular in the development community. Some well-known and established APIs and frameworks such as EJB (Enterprise Java Beans), JPA (Java Persistence API), JUnit, and Spring Boot use annotations extensively. This native annotation support encourages many Java frameworks and API developers to adopt the metadata-based approach in their solutions.

Two recent studies that measured their occurrence in the source code highlight the importance of studying code annotations. One of them evaluated the top-ranked Java projects hosted on GitHub and observed that, on average, 76% of classes have at least one annotation [1]. The other study included 1,094 projects in the analysis and found that the median value of annotations per project is 1,707 [2]. Both studies detected possible abuses in the usage of annotations, finding a high number of annotations in individual classes and projects. This evidence shows that annotations are a popular feature in Java projects and consequently can influence several aspects of the code and software projects as a whole.

When discussing annotation-based APIs in the context of metadata-based frameworks, such as JPA, JUnit, and Spring, an important concept is “annotation schema”²²2in this work we also refer to it simply as “schema”. Since one annotation rarely is enough to represent the information needed in a metadata-based API [13], usually a group of annotations is combined in the same package to fulfill that requirement. So, “annotation schema” can be defined as the group of annotations used to represent the metadata structure used by a metadata-based API.

Since these annotations, which are part of the same metadata structure, are often part of the same package, this information is used as a heuristic to identify an annotation schema [1]. In other words, annotations from the same package are considered part of the same schema. Consequently, a single metadata-based framework may contain several schemas, usually representing distinct data structures responsible for representing metadata for different framework features.

To better illustrate the definition of “schema”, consider the code in Figure 1 that presents a simple class responsible for executing unit tests using the JUnit 4 framework. The annotations @Test, @After, @Before that belong to the package org.junit are used in combination to define how the framework should execute the tests, being part of the same annotation-based API. In this case, we refer to this annotation set as part of the org.junit annotation schema. In practical terms, an automated approach for extracting the schemas used in classes can identify each package from the imported annotations as a separate schema [14].

The code example in Figure 1 makes clear the strong relationship between the presence of annotations of a given schema and code responsibilities. It is clear that a class with annotations from the org.junit schema is responsible for testing. Therefore, detecting their presence in software systems may help understand how developers organize packages’ responsibilities. As another example, if the schema javax.persistence is imported in a class, we can infer that this class is performing actions related to object-relational mapping and dealing with a database. We also would expect that these classes would be grouped in the same package, which may help identify the responsibility of the package instead of a single class, giving a broader understanding of the system under analysis. In other words, the presence of classes with annotations of a given schema in a package might indicate their role in the system architecture. Some studies confirmed this possibility by performing Java source code static analysis using code annotations to identify the classes’ architectural role [15, 16]. Finally, this information also helps understand the coupling between applications to specific annotation-based APIs or metadata-based frameworks. In the first example (org.junit) the class is coupled to the JUnit framework, and in the second example (javax.persistence), the class is coupled to the JPA.

The software visualization that we are proposing and discussing in Section 5 was highly motivated by this relationship between annotation schema and class/package responsibilities. The visualization aims to display the used schemas so developers may visualize the responsibilities of classes and packages in the target system.

The visualization approach we propose - Code Annotations Distribution Visualization (CADV) - uses source code metrics values as input. Source code metrics help summarize particular aspects of software elements, detecting outliers in large amounts of code. They are valuable in software engineering since they aid in monitoring code quality and controlling complexity [17], making developers aware of possible abnormal growth of specific system characteristics.

The metrics used in CADV are a suite dedicated to measuring code annotations proposed in our previous work [1]. This subsection describes these metrics briefly to clarify how they are used in our visualization. The description focuses only on the metrics used in CADV. The code presented in Figure 2 is used as an example of how the metrics’ values would be collected.

• •

  Annotations in Class (AC): It represents the number of annotations declared on all code elements in a class, including nested annotations. In our code example, the value of AC is equal to 10.

• •

  Annotations Schemas in Class (ASC): It counts the number of different schemas present in a class, measuring how coupled a class is to different frameworks. This value is obtained by tracking the imports used for the annotations. In the code example, the ASC value is 2. The import javax.persistence is a schema provided by the JPA API, and the import javax.ejb is provided by the EJB API.

• •

  Arguments in Annotations (AA): It counts the number of arguments contained in an annotation. Each annotation has its own value for the AA metric. For instance, the @AssociationOverrides has only one argument named value, so AA has the value of 1. As another example, @AssociationOverride, contains two arguments, name and joinColumns, so the value for the AA metric is 2.

• •

  Annotations in Element Declaration (AED): It counts how many annotations are declared in a code element, including nested annotations. In the code example, the method exampleMethodA has an AED value of 2, as it is annotated by @TransactionAttribute and @DiscriminatorColumn.

• •

  LOC in Annotation Declaration (LOCAD): LOC (Lines of Code) is a well-known metric that counts the number of code lines in a given context. This metric is a variant of LOC that counts the number of lines used in an annotation declaration. As examples, @AssociationOverrides has a LOCAD value of 5, while @NamedQuery has LOCAD equals 4.

With this suite of metrics, it is possible to characterize the usage of code annotations in a given system, capturing their size and complexity and the coupling generated by them. In our software visualization approach, these values will be used to render the system. Therefore, these metrics can be considered the basis for the CADV approach.

In our previous work [1], we defined a novel suite of software metrics (as described in Section 2.2). This work also describes an exploratory data analysis that aimed to understand how these metrics values were distributed in open-source projects by collecting the metrics values from 25 selected projects. It was identified that 90% of the classes use up to 11 annotations, often declared using one line of code and one argument. Finally, they are usually not nested, being rare to find a two-level nesting. However, some outlier values found in this study indicated potential problems that at least deserve a closer inspection. For instance, we found classes with more than 700 annotations and one annotation taking 58 lines of code. Also, we found that one single code element had 27 annotations configuring it. This result is evidence of extremely high values for annotation metrics that, like other instances of large size metrics values as an extensive method or a high number of parameters [18], might negatively influence code maintenance.

Yu et al. [2] performed a large-scale empirical study on code annotations usage, evolution, and impact. The authors collected data from 1094 open-source Java projects and conducted a historical analysis to assess changes in code annotations. The study found huge values for the number of annotations considering the code element, the proportion in terms of lines of code, and the project as a whole. For instance, the authors detected that some code elements might have 41 annotations configuring them, which they considered extremely high. These results indicate the existence of abuses of this feature and confirm the result from Lima et al. [1]. Another finding revealed that code annotations are constantly changed during project evolution, implicating that some developers use annotations subjectively and arbitrarily, introducing code problems. They also demonstrated that code annotations are deleted because they are redundant (15.4%) or incorrectly placed (24.5%).

Concerning annotation repetition, Teixeira et al. [19] performed a study that investigated the source code of a web application by searching for annotations repeated throughout the source code. The findings revealed that some annotations were repeated around 100 times in code elements with shared characteristics in the target system. The repetition of annotations is also mentioned in Yu et al. [2], but no quantitative evidence was presented. The study suggested that more general definitions, such as application-specific code conventions, could significantly reduce the number of configurations.

Based on these studies, it is possible to state that code annotations are a relevant language feature widely used in Java projects. Therefore, they can positively or negatively influence those systems’ maintenance and evolution. Furthermore, these works also showed that code annotations might be misused in several ways, such as being misplaced [2], duplicated [19, 2], and overused [1, 2]. Consequently, developers might benefit from tools and approaches that aid in comprehending how code annotations are distributed in systems, and software visualization can be used for such purposes.

The primary goal of CADV is to aid in the comprehension and provide an overview of how code annotations are distributed in the analyzed software system. Then, the visualization should also provide more details of code annotations in specific classes and packages that the user wishes to explore further. Following, we describe the five goals.

• (#G1)

  - Detect annotations schemas and how they are distributed in the packages: The user should be able to spot all schemas and quickly identify where they are being used. For instance, if the code annotations from a given schema are present in different packages, it might indicate that the responsibilities related to that schema are distributed in these packages. As another example, a concentration of an annotation schema usage in a package enables the user to locate the classes that handle the respective responsibility quickly. To reach this goal, the view should represent the annotations from each schema in the whole system and should not overwhelm the user with details closer to the source code.

• (#G2)

  - Detect how annotations are distributed per class in packages: From the general view of the system, the user might want to investigate specific packages to obtain more details about code annotations in its classes. For instance, (i) how schemas are distributed inside classes, (ii) if the classes are coupled to several schemas, and (iii) if there are classes with a large number of annotations. An annotation size metric, such as LOCAD (Lines of Code in Annotation Declaration), can represent the annotations. To reach this goal, the view should present the code annotations characterized by a size metric grouped by their classes.

• (#G3)

  - Detect how annotations are distributed and grouped per code elements inside the classes: Code annotations are placed on code elements, such as methods, members, and type definitions. Several annotations can be added to the same code element. The user might be interested in how the annotations are grouped inside these code elements to identify (i) if a specific code element is overloaded with annotations, (ii) if several code elements contain repeated annotations, and (iii) how annotations from different schemas are distributed inside the class. The size of the annotations should also be represented by a metric that can characterize its size. The view should group the annotations by code elements inside the classes to reach this goal.

• (#G4)

  - Provide a navigation system between views with different granularity: For each goal presented previously, #G1, #G2, and #G3, we are proposing a different view consistent with each of these three goals. Therefore, our visualization approach should provide some mechanism to navigate these different views.

• (#G5)

  - Detect misconfigurations: Our goals are concerned with the distribution of code annotations in the system rather than detecting problems related to them. However, we might also detect potential problems if we can visualize code annotations. According to Yu et al. [2], code annotations are deleted because they are redundant (15.4%) or wrong (24.5%). The same study also points out that 19.8% of the annotation replacements switch them to the “same name” annotations from other libraries, revealing that annotation from an unexpected schema was present. The proposed approach should identify inconsistent annotations, enabling the user to locate potentially misplaced annotations.

Software visualization approaches should clearly state the target audience, which should be consistent with the goals. Following, we describe the target audience aimed for the visualization approach:

• •

  Newcomer Developer⁴⁴4The term ”newcomer developer” is used in this work to refer to someone who has just joined a team. It does not necessarily mean that he/she has no software experience since even a senior software developer can be a newcomer.: According to Dagenais et al. [25], whenever a developer approaches a new software system, he/she feels like an explorer who need to orient themselves within an unfamiliar landscape, even senior developers. The goal #G1 provides a general overview of the system that may help newcomers.

• •

  Student: Given that they are constantly learning, they are also part of the target audience. They could use the visualization like newcomer developers to understand better some concepts related to code annotations and metadata-based frameworks usage. They can also identify aspects of the software system and further study them using other methods.

• •

  Software Architect: Since he/she already has a firm grasp of the whole system, this audience can use the visualization approach for a more detailed analysis focusing on aspects such as: how to make this system better adhere to the proposed software architecture? Are annotations growing out of control in some parts of the system? Are package responsibilities clearly defined? Are there any misplaced or misconfigured code annotations? Are there any packages coupled with different annotation schemas?

The target audience does not need to adhere strictly to these definitions. A newcomer can use the visualization to detect outliers, just as a seasoned developer can also gain new insight about the system software it did not know.

The CADV, as further explained in Section 5, uses values extracted from the suite of metrics dedicated to code annotations [1]. Given the novelty of these metrics, we were unable to find a tool to compare with our solution. Therefore, to evaluate the CADV, we conducted an empirical study using questionnaires and interviews where participants had to use the AVisualizer, inspect a target software system, and answer some questions. We assessed the effectiveness of the tool in (i) program comprehension tasks related to code annotations, (ii) its perceived usefulness, (iii) its perceived ease of use, and (iv) the audience that it can potentially benefit.

The program comprehension tasks were elaborated based on real-world scenarios extracted from our previous experience [12, 13, 19, 1, 14], our visualization goals (Section 4.1), and also tasks other researchers elaborated when evaluating their software visualization approach  [7, 5, 8]

The correctness of the program comprehension tasks was defined by three authors, all with previous experience with code annotations. They were not involved in the development of the target system nor had any direct relationship with the developers. To elaborate on the tasks, they were granted access to the code. After the interview was carried out, all six developers agreed with the correctness of the answers.

The categories perceived ease of use and perceived usefulness were based on the TAM (Technology Acceptance Model) [26]. These two variables characterize how users find the technology easy to use and how users find the technology helpful. The questions we elaborated to measure them used the Likert scale, ranging from strongly disagree to strongly agree, and a strategy similar to the work of Choma et al. [27].

We generated the proposed view for a real-world system to perform the evaluation, using a module from EMBRACE as the target software. It is a web application whose goal is to show space weather information developed by the National Institute for Space Research (INPE)⁵⁵5https://www.gov.br/inpe/ - partially available in English, a Brazilian public research institution active in the fields of meteorology and aerospace. INPE has the mission to foster scientific research, technological applications, and qualifying personnel in space and atmospheric sciences, space engineering, and space technology. It is vital in environmental protection, monitoring forest fires, and deforestation. The institute requires enterprise software systems that allow processing, persisting, and making data available to the community. This work was developed in INPE, and using a system in production from this institute as a target from our study applies our approach in a real-world application developed to meet the demands of scientific research institutions.

The EMBRACE web application is composed of several modules that follow the same reference architecture [28] based on Java Enterprise APIs. The application is used to process and make data publicly available. This software system has 1314 classes, from which 837 (64%) contain at least one annotation. The application comprises six modules spanning 94 components archived in independent deployment units. We selected the module SpaceWeatherTSI to be used as the target software in the evaluation, given that it uses annotations from different metadata-based frameworks. This attribute matches the characteristics of a software system that could benefit from the CADV approach, that is currently in production. Afterward, we conducted an interview study with six developers involved in constructing the target software and a questionnaire study with 79 master and undergraduate students from three different institutions. The idea of applying a questionnaire and interviews with different groups have been used in other Software Engineering research [29, 30]. We aim to capture different perspectives from groups with different backgrounds and experiences. The design of this study is described in more detail in Section 6.

Afterward, we conducted qualitative and quantitative analyses of the data obtained from the abovementioned studies. With this information, we evaluated the approach’s effectiveness based on the number of correct answers to the program comprehension questions and how useful, and easy the tool was perceived. Finally, since the interview was semi-structured and the questionnaire also provided one open-ended question, we conducted a qualitative analysis to extract relevant points and insights about the proposed approach.

Figure 3 presents the initial page of the AVisualizer tool. The circle packing view, drawing a given system under analysis, is presented on the left side. This interface is constantly updated as the user navigates the tool. Above the view, there is a small header with the following information:

• •

  Visualization: Informs which of the three views is currently rendered. It could be System View, Package View, or Class View.

• •

  Annotation Metric: Inform the metric value used to render the leaf nodes.

• •

  Package or Class: Informs what package or class is currently being displayed, represented by the outermost circle.

On the right of the visualization, the table Annotation Schemas lists all schemas found on the project, displaying the associated color and informing their total number of annotations. For instance, there are 168 annotations from the javax.persistence schema, and 19 from the org.junit schema. By associating a color to a schema, the user can spot its annotations by searching circles of that color inside circles representing packages or classes. There is also a check box on the table that allows hiding specific annotations.

AVisualizer uses fixed colors for some common annotation schemas found in open-source software [1]. This strategy helps to standardize the usage of the AVisualizer since common schemas will receive the same color in the views generated for different target systems. For instance, given that pink is used for the javax.persistence, regardless of the software being analyzed, if the user spots a pink circle, she will associate it with persistence. If the schema found in the system is not predetermined, then the AVisualizer randomly associates one. However, white and gray colors do not represent schemas since they already have other meanings within the CADV. The following is a list with some examples of these predetermined schemas:

• •

  java.lang - Blue

• •

  javax.persistence and org.hibernate - Pink tones

• •

  org.springframework - Orange tones

• •

  org.junit and org.mockito - Purple

• •

  javax.ejb - Green

Related schemas will use a tone variation of the same color. For instance javax.persistence and javax.persistence.metamodel will have different pink tonality. Finally, the views do not display any textual information to create a clean visualization. However, the user can hover the mouse over the circles to reveal labels with more information.

The System View, which is presented in Figure 3 is the initial view displayed to the user. It displays the whole project allowing users to overview how code annotations are spread in the system under analysis. The System View displays circles representing packages and a representation of the annotation schemas usage. The light gray background color was chosen to remain as neutral as possible. The following list presents the characteristics of these circles and what information becomes available when the user hovers the mouse over the circle representing the annotation.

• •

  Packages: Every circle representing a package has a dashed outline. The outermost dashed circle represents the root package of the project. The inner “dashed outlined circles” are other packages, which can be at the same level or nested. In Figure 3, there are several packages in this system, each represented by circles with a dashed outline. This approach displays the hierarchical structure of the source code under analysis. The user can also click on these circles (packages) to perform a zoom action. When this happens, the outermost circle changes, and the header will reflect the current package.

• •

  Annotation Schemas Usage: These are colored filled circles rendered inside “dashed outline circles” (packages). They represent the occurrence of annotations of the schema mapped to the circle color inside that package. The Annotation Schemas table works as a legend for the annotation schema color. The size of these circles is proportional to the number of code annotations of that particular schema inside that package. The larger the circles, the higher the number of annotations from that schema. For instance, the System View on Figure 3 shows a large pink-toned circle, meaning that this package has a high number of annotations from the javax.persistence schema.

• •

  Label: When the user hovers the mouse over a colored circle, i.e., an annotation, a label appears with the following information: (i) the name of the schema, (ii) the name of the package and, (iii) the number of occurrences of annotations from this schema in the package.

In the System View, the classes of the packages are not visible. It does not show in how many classes the annotations are divided. To access this information, the user should click on the package to navigate to the Package View, described in the next section.

The Package View can display classes and individual code annotations inside a given observed package. Differently from the System View designed to visualize the whole system, the Package View displays a specific package. Figure 4(a) displays an example of the Package View. The circles are rendered with the following characteristics:

• •

  Package: It has the same characteristics as the System View.

• •

  Classes: Classes are rendered as white-filled circles. Their size depends on the number of code annotations used inside the class. If a white circle appears large than other white circles, it represents a class with more code annotations.

• •

  Code Annotations: These are colored (any color besides white and gray) filled circles rendered on top of white circles. They represent code annotations being used inside a specific class. Their color matches the color of their schema, present on the Annotations Schema table. The size of these circles is proportional to their LOCAD value, i.e., the default metric used in the Package View.

• •

  Label: When the user hovers the mouse over a colored circle, i.e., an annotation, a label appears with the following information: (i) the name of the package, (ii) the name of the class, (iii) the name of the annotation and, (iv) the metric used to determine the size of the circle. For the Package View, the default metric is the LOCAD (LOC in Annotation Declaration).

The Package View does not display how individual annotations are grouped in a specific code element inside a class. To access this information, the user should click on the class to navigate to the Class View. Comparing the Package View with the System View, it displays information with finer granularity but for a more restricted context.

The Class View displays classes with individual code annotations inside the observed class. It shows how code annotations are distributed by code elements, such as a method, a field, or the class itself. Figure 4(b) presents an example of Class View. The circles are rendered according to the following rules:

• •

  Classes: Just as in the Package View, they are rendered as a white circle. There is only one white circle enclosing all the others since the focus is on a specific class.

• •

  Annotations: Annotations are represented individually by colored circles, whose color represents the schema. The circle size is based on the AA (Arguments in Annotations) metric.

• •

  Gray-Circles: This color is also used to represent packages, but in the Class View they represent code elements, such as method and class members. The code annotations (colored circles) are rendered on top of these gray circles. Colored circles rendered directly on top of a white circle represent code annotations configuring the class itself. The number of colored circles rendered on the same gray circle represents the AED (Annotations in Element Declaration) metric.

• •

  Label: When the user hovers the mouse over a colored circle, i.e., an annotation, a label appears with the following information: (i) the name of the package, (ii) the name of the class, (iii) the name and type of the element the annotation is configuring, (iv) the name of the annotation and, (v) the metric value used to determine the size of the circle. For the Class View, the default metric is the AA (Arguments in Annotation).

The gray-toned background used to represent code elements is the same one used to represent a package background in the other two views. However, it does not affect the Class View since it shows a single class, and no package is displayed. Furthermore, the gray color provides a suitable neutral background. Apart from white and gray, every other color represents a schema.

Inspecting the Class View, it is unclear whether the gray circle represents a method, enum, or other code elements. This design decision was made to avoid overloading users with more information. Users, however, can access this information by pointing the mouse to it. The Class View can display the grouping of code annotations by code elements based on the size of these gray circles. Furthermore, the schema information is always available since each one has a unique color throughout the project.

The metric used to render the leaf nodes, i.e., the colored circles representing individual annotations, was the AA (Arguments in Annotations). In other words, annotations with more arguments are larger. We decided to use this metric to vary from the one used in another view since Package View already uses LOCAD.

The CADV is a circle packing-based view designed to aid in visualizing code annotations used in software systems. It comprises three different views to display information at different granularity levels. This approach allows users to switch between these views and analyze the parts of the software with the desired granularity:

System View (SV):

Related to the goals G1 and G5, displays packages and schemas used. Each schema is assigned a color, and packages are gray circles with a dashed outline.

Package View (PV):

Related to the goals G2 and G5, displays a package, classes, and code annotations used in the classes. Classes are white circles. Code annotations are assigned the schema color and are rendered on top of the white circles. Packages are gray circles with a dashed outline.

Class View (CV):

Related to the G3 and G5, displays a class, code elements, and code annotations grouped by code elements. Classes are white circles. Code elements are a gray color. Code annotations are assigned the schema color and are rendered on top of either the white circles or gray circles.

We invited developers who worked in the SpaceWeatherTSI (described in Section 4.3) web application to participate in the interview study. Members involved in this project’s construction could improve the assessment of the visualization approach. Given that they know the system, they could assess if the AVisualizer tool could effectively display a coherent view of the project under analysis. They could also analyze if the CADV brought new insights and valuable information to an already familiar system.

We invited students from Inatel⁶⁶6https://inatel.br/home/ (National Institute of Telecommunications) and IPT⁷⁷7https://www.ipt.br/ (Institute for Technological Research) in Brazil, and UniBz⁸⁸8https://www.unibz.it/ (Free University of Bozen-Bolzano) in Italy to answer the questionnaire. Seventy-nine students participated in the study, where 70% were undergraduates in Computer Science/Engineering courses, and the remaining were master students. These studies were carried out remotely for eight months. None of these students was involved in developing the SpaceWeatherTSI software and provided a neutral point of view. Using students to conduct experiments and evaluations is a viable option to advance software engineering [35, 36]. The questionnaire was applied in the context of courses that explained concepts about code annotations for the students. Participation in the study was optional, no grade was associated with the answers, and we did not keep any information that could identify the students.

We gathered the data obtained from the interviews and questionnaires to analyze and obtain the results considering the four categories mentioned in Section 4: (i) Program Comprehension Tasks, (ii) Perceived Usefulness, (iii) Perceived Ease of Use, and (iv) the Target Audience (iv).

The first item, Program Comprehension, can be analyzed by validating the correctness of the answers. For the remaining three, we combined a qualitative and quantitative approach. The quantitative data can be extracted from the Likert Scale answers. The qualitative data collected in the interviews and answers to the open-question allowed us to obtain insights from the answers to the objective questions. To analyze this qualitative data, we followed a thematic analysis. This approach is “a method for identifying, analyzing and reporting patterns (themes) within data” [37]. In this process, pieces of text are coded using labels that are later grouped in higher-order themes, increasing the abstraction level until a defined saturation point, e.g., a model, could be built. Therefore, we aim to label and categorize data using this approach to identify the strengths and weaknesses of the approach commonly mentioned by the subjects.

Cruzes and Dybå [38] identify three ways a thematic analysis could be conducted. First, researchers build an initial list of codes that will guide them in the coding process in a deductive approach. Another option is an inductive approach where researchers do not have a starting list of codes and inspect the code systematically, e.g., sentence by sentence, to make some concepts emerge from data. Finally, the authors acknowledge an integrated approach that is a “partway” between the previous two. Using this approach, researchers have a general scheme pointing to some aspects, but the themes are built bottom-up. We followed an integrated approach in our analysis since TAM already gives us two themes: perceived ease of use and perceived usefulness. Our supplementary material provides an example of the steps we used to extract some codes/themes as well as how they were consolidated/grouped in Table4 [34].

We hired an external service to transcribe the interviews. For the coding process, we employed NVivo 12⁹⁹9https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home. A researcher coded the transcriptions of the interviews and the answers to the open questions in the questionnaires. Then, another member of the authoring team reviewed the labels. The two researchers discussed the disagreements until they reached a consensus.

The number of correct answers from the program comprehension questionnaire [34] is graphically displayed in Figure 5. We used a bar chart to display the correctness of each question by all participants of the questionnaire. Questions Q2-Q6 had a high success rate, with roughly 90% of the participants answering correctly. Questions Q1, Q6, and Q10 also performed well with 80%, 75%, and 73% success rates, respectively.

Most of the questions with a high success rate are related to the System View and Package View. In other words, these are questions related to the general view of the system instead of a specific class or type. The participants could answer these without navigating far on the tool. The exception was Q4, which asked “What class contains the highest number of javax.persistence annotations?” which required the participants to reach the Class View and still performed very well with a 93% of success rate. The coding presented in Table 4 can be used to sustain this result. The theme PES-1 - Easy to see annotation’s usage and PUS-5 - Detection of Responsibilities Using Schemas, suggests that even in the Class View, the schema color information is still present, which helps detect the responsibilities. As seen in Table 4, PUS-5 had seven mentions, the third most present theme. This result helps confirm that there is a relationship between code annotations and class/package responsibilities as presented in Section 2. Finally, the CADV allows the visualization of such annotations, which helps identify and detect class/package responsibilities.

Questions Q8 and Q9, however, performed below the 70% correctness rate. These questions required the user to investigate more deeply using the Package View and the Class View. Furthermore, they require a good understanding of the code annotation metrics instead of Q4, which asked about responsibilities. As found in Table 4, the theme PEW-3 appeared in 17 answers mentioning Confusion when switching metrics between views. Also, the theme PEW-5 - Confusing how to trigger a new view was mentioned five times. These results support the lower rate of correct answers for questions Q8 and Q9 that required changing views and understanding these metrics. The theme PEW-6 also helps explain these results because it implied that users were not checking the header to see what annotation metric was used.

Compared to Francese et al. [5], they had a similar correctness rate, ranging from 70% to 87%. However, we had higher values (93%) in some questions, and one question performed lower (51%).

To measure the perceived ease of use in the questionnaire with students, we issued eight statements presented in Table 3 using the Likert Scale. Figure 6 displays a diverging bar chart with the results. Table 4 presents the coding/themes that were in the majority extracted from the interviews with developers but also the one open-ended question students answered in the survey.

From Figure 6 the majority of the respondents agree or strongly agree that, overall, it is easy to use and obtain information from the target system. The statement S3 “Learning how to use the AVisualizer was easy” had the highest strongly disagree value, suggesting the tool requires training to be used appropriately. The theme PEW-7, which appeared in eight answers, states that training is required to use the tool and supports this result.

The statement S6 had the highest “strongly agree” value, suggesting it was easy localizing in what part of the software the user was. This may seem contradictory to the theme PEW-6, which suggests the headers were sub-utilized. However, crossing this data with the Program Comprehension results, themes PEW-5 and PEW-3, the header most likely failed to inform what metric was being used rather than what package or class the user was inspecting. Furthermore, PES-3 reinforces that the circle packing provided a good comprehension of the hierarchical structure, helping users locate themselves while using the tool.

In the strength theme, the most common theme in eleven answers was PES-2 claiming that it was Easy to understand how to use the tool. Then, in six answers, the respondents said it was easy to see annotations’ use (PES-1). Three respondents also mentioned that the tool has a quick learning curve - PES-4.

In the weaknesses theme, we grouped codes considering aspects that make the tool harder to use. The most prominent was PEW-3 claiming it was Confusing when switching metrics between views, with 17 answers, followed by PEW-1 Understand the concept of annotation schema. The code PEW-3 is also related to PEW-8 Knowledge of Annotation Metrics. The issue is that most users are not familiar with the code annotation metrics. PEW-1 suggests that even though code annotations are very popular among Java programmers, some may not know the concept of annotation schema. If a user is familiar with schemas, it becomes easier to use the tool. On the other hand, if the user lacks familiarity with schemas, it is harder to reason and extract information about them.

To measure the perceived usefulness from the students, we presented them with four scenarios, mentioned in Section 6.2.3. From the results, 41% of the students would employ it in Scenario I, i.e., to detect schemas used in a given project and further learn about them. In second, with 34.62%, was Scenario IV, “Familiarize with the system before adding new features”. In third place, with 12.82%, was Scenario III, “Analyze the architecture”. The least chosen was Scenario II, related to searching for problems or misplaced annotations. These results suggest that the students perceive the tool. useful to support a learning process.

Analyzing Table 4, the most common theme for strengths, identified in eleven answers, is PUS-1 Visualization is better than code inspection to search annotations related information, suggesting the AVisualizer may outperform manual code inspection. As a respondent wrote: “It is also a great idea since checking the code (going through dozens of packages) takes quite some time and is not visual, so this tool helps [a lot].” Another interviewee mentioned that: “[…] when you put it visually, it is much easier for me to understand how they are spread in the system […]”. Besides that, the theme PUS-3 suggests that the tool can aid in refactoring and modularizing the system’s architecture. Theme PUS-4 mentions that the AVisualizer can detect the simultaneous usage of frameworks with the same function. Even though these frameworks can usually work together without errors, using only one would make the solution more consistent across different features in the architecture. Finally, theme PUS-6 reinforces that the color strategy used to identify schemas can also help identify potentially misplaced code annotations.

Detecting misplaced annotations may benefit developers when performing a refactoring process. As mentioned, the work of Yu [2] demonstrated that 19.8% of annotation replacements are related to “same name” changes. In other words, the developers identified that the annotation name was correct. However, it belonged to an unexpected schema. With the AVisualizer, these scenarios can be spotted because an “unexpected” or “different” color circle will be in the wrong package. As stated by one interviewee: “[…] it is very simple to detect potentially misplaced code annotations. If I spot a pink circle in a package with only orange circles, I would say something is wrong. Why is a javax.persistence code annotation alone in a package with only org.springframework code annotations? ’’

For the weakness, the most common code, present in 12 answers, was PUW-1 I can find large annotations, but this is not very useful isolated. These answers were rising when discussing topics such as finding large annotations or excessively used annotations. Even though the AVisualizer allows the detection of large annotations, the respondents did not see much value in this information isolated, claiming other design decisions outside the scope of the AVisualizer should also be used. This result differed from detecting misplaced annotations, which was seen as very useful.

One interesting code was PUW-2, suggesting that annotations such as @Override is trivial and probably pollute the visualization. Finally, present in the five answers was PUW-4: tool is more useful if the developer is familiar with schemas. As one respondent stated: “The colors help detect these potentially misplaced annotations. However, I argue that whoever is analyzing the system should also be familiar with code annotations and schemas. This way, a better decision is made to determine if the package or class requires further investigation or code inspection.”

To validate the intended target audience, we asked the students what role they found the tool was suited for, allowing them to select multiple roles. The students chose “Developer” as the preferred role for using the AVisualizer, followed by “Newcomer developer” and “Architect”. The least chosen was “Project Manager”. Of the interviewees, three said “Architect” was most suited, while three said “Every role in the team can benefit”, including the “Architect”.

One interviewee stated: “[…] the tool has different utilities within a software development team. The architect is usually concerned with the organization, and the tester may use it to detect packages that require more test code. A developer is usually more concerned with going straight to the code.”. Another interviewee stated: “Everyone that touches code can benefit from this tool”. However, one interviewee believes the newcomer should consider other aspects before, stating: “Newcomers should first study what design patterns are being used in the project. Verifying the annotations should be second”.

• (#G1)

  : To reach this goal, we designed the System View. The results show that both students and developers found this view the most useful and were able to get a general view of how code annotations were distributed in the system.

• (#G2)

  : To reach this goal, we designed the Package View. The results show similar results compared to the System View. Students and developers could visualize the usage and usage of code annotations within a specific package.

• (#G3)

  : To reach this goal, we designed the Class View. The results show that this view did not reach the same correctness rate as the other two. In the qualitative analysis, we identified that this result was related to switching views and code annotation metrics unfamiliar to respondents. However, in the worst scenario, it reached correctness of 51%.

• (#G4)

  : The AVisualizer tool was developed with this feature, enabling users to navigate between all three views. From the results, the users could navigate between views after proper training. Although S8 presented a good agreement about how easy is the navigation in the tool, PEW-3 and PEW-5 showed that it could be improved.

• (#G5)

  : From the qualitative analysis, we found the tool is useful for detecting potentially misplaced annotations. As for detecting large or excessively used annotations, the coding revealed that the AVisualizer tool allows their detection, but respondents did not see much value in this. We argue that future experiments and a proper study should investigate code annotations bad smells and how these large annotations potentially impact software maintenance.

This section discusses the threats to the validity of our study. We followed the guidelines from Wohlin et al. [39] defining four steps to evaluate a study: conclusion validity, internal validity, construct validity, and external validity.