Varying Annotations in the Steps of the Visual Analysis

The data preprocessing step has the goal, to collect and structure necessary information from all available data sources. These data may stem from different sources, as real-world data often are collected from various devices and scientific data often are redundantly recorded to decrease uncertainty. When these sources are merged, redundancies and discrepancies may appear. With data-preprocessing annotations, we automatically collect that additional information on source redundancy and discrepancy for each data value. These annotations are linked to the respective data-point and amended with the source name and time stamp.

Additionally, data sources may differ in reliability. This is specifically the case, if a data source contains primary and secondary information. The primary information is the main information (e.g. data dimension) from a source with a high degree of reliability. The secondary information has also been recorded within the data source, yet is not in the focus and may thus be less reliable. An example for clinical data may be from the clinical management system. There the diagnoses may be primary information, as it is important for billing purposes, while the pathologies are also kept to inform the patient or relatives, but are not used by physicians and are therefore secondary information with less reliability. If this is the case, the experts need to define a hierarchy for each data dimension and data source, in order to automatically resolve discrepancies. That hierarchy information on which an automatic decision is based, is also saved by an annotation linked to the data value. As a result, we get annotated structured data, providing the information to the user, what data sources were considered, if redundancy and/or discrepancies apply and what rule lead to the choice of value. The annotations are made available to the user by a respective encoding at later stage in the analysis. Figure 1 shows an example with detail information on existing sources, including a value discrepancy (0.8 vs. 0.5).

During data cleansing the user judges data values in regard to their plausibility and completeness. As data discrepancies are common for real-world data, this step often involves data editing in order to create a consistent data-set (see, e.g., [Sch+19]). As the data editing functions can either be manual or automatic and data discrepancies can variously be resolved, annotations should inform the user when, how, and by whom this was done. We reach that by storing the rule set on which automatic data corrections have been performed. This information can then be shown to the user when necessary and with specific visualization techniques. An example is provided in Figure 2. The annotated data is highlighted in the visual representation used for data cleansing (red circles and lines), and when clicking an annotated data value, a textbox with detail information is presented. In case of manual data corrections, a validation of the editing can be useful, especially in asynchronous collaborative environments as well as in discontinuous processing as described by Zhao et al. [Zha+18].

With annotation support during data exploration, we allow users to take notes within the system. This can be done for different purposes. First, identified findings can be marked and externalized via annotations. Second, existing externalizations, saved via annotations, can be commented or validated. Third, validated annotations can be separated by users from unvalidated or invalid annotations to support further analysis. To allow later interpretation of a marked and externalized finding, we provide (i) the verbalized finding from the expert, (ii) the visualization at the time of externalization via screenshot, and (iii) a reference to all data points visible at the time of externalization. This will give users the ability to interpret the verbalization in combination with the visual representation at the time of externalization. Moreover the reference to the data, allows other users to check for updated data and/or annotations. At any time, recorded annotations are displayed in a separate view. This way, we can provide commenting or validating functions for all annotations.

Such functions are particularly important, if the exploration process is interrupted or performed by different users. Then, it can be necessary to judge or comment previously recorded annotations. Commenting is applied, if the interpretation of early findings changes, for example since new information is available, or a user contributes new knowledge that leads to a different view of a finding. In particular, we enable users of a certain qualification to make an assessment of existing externalised information (annotation of the annotation). This way, annotations can be confirmed. This facilitates a separation into unvalidated/valid/invalid annotations. We present all annotations made in the exploration process (see Figure 3) to support the interpretation of the findings and the generation of hypotheses.

In this subsection we briefly summarize the annotation functionality of our visual analysis tool. The tool itself has already been described by Schmidt et al. [Sch+19].

Generally, all annotation management is integrated into the tool units as seen in Figure 4. Automatically generated annotations are added during data pre-processing, whereas manual annotations are recorded while data cleansing and exploration are performed. Automatically derived annotations are visible at the data preprocessing stage. On the other hand, data editing annotations are visible during data cleansing, and manual annotations on exploration are visible in that phase respectively. The annotations for each step have a specialized design, which is described below.

Automatic annotations for data preprocessing are encoded via color by a separate layer (see Figure 1). The color for the annotation is distinct from the data color coding to emphasize annotated data. As preprocessing annotations encode the reliability of the data values: The colors of the layer occlude data values (i) completely, if there is a discrepancy, (ii) partly, if there is only one source, and (iii) not at all, if the data values are derived from several sources but no discrepancies occur. This has the positive effect that experts see the colored data regions with discrepancies at first sight, while they are not wrongly influenced by seeing a possibly incorrect data value. In addition, we show further details on demand, such as corresponding data sources, data values, or source priorities.

During data cleansing annotations on editing operations are manually added by user interaction. We provide the annotation recording interface within the editing dialog. Once the recording has been performed, the annotated data are amended with additional glyphs in the visualization (see Figure 2). To avoid confusion between data values and annotation indication, these glyphs are distinct in shape and color from the data encoding. The glyphs are depicted nearby the annotated data values, and by hovering them, the full annotation content will be presented via a separate view.

The annotation design during exploration is different from the previous steps. Here, annotations are often not directly related to single data values, but to particular data features. Moreover, we want to support the judgement of the displayed data and annotations. For that reason, we decided to separate the annotation design from the data visualization view. Instead, we provide global annotation functions with a separate view (see Figure 3). Within that view all previously recorded annotations for exploration are shown. At this step, the experts focus on several points. First, they want to judge who recorded an annotation when and with what qualification, so we show a short profile in the top left corner. Second, they need a visual impression on the visualization relevant for the annotation, which we provide by a thumbnail in the upper right corner, that enlarges on click. Third, they want to read, what information has been externalized, which is shown in the lower part. Last, we provide validation functions via voting and comment options in the bottom part of each annotation. Similar to established designs from social media platforms, we also instantly display the validation feedback and recorded comments.

In this subsection, we briefly report on how our domain experts applied annotations in order to analyze their data. The feedback was gathered during several interviews and iterative application sessions and consolidated to transport key messages.

At first, the automatic creation of annotations during data preprocessing allowed domain experts to see open issues with redundant data sources at first sight. Encoding the automatically created annotations differently, depending on the existence of discrepancies and redundancies, allowed experts to focus on open issues, e.g., discrepancies. This reduced the overall effort to define usable data values. Additionally, the experts learned about the reliability of information from their clinical systems, which they considered as important for their every day work in the clinic.

Annotations created during data cleansing, eased the comprehensibility of the editing process for the experts, as reasons for editing where given and permanently recorded. An analysis of the editing annotations helped to understand existing issues in the clinical process. So, for example, visual acuity values are not always automatically transferred to the doctoral letter, but sometimes need to be copied by hand, increasing the risk of copying errors.

Finally, experts noted that the ability to create notes during exploration, leads to a common information base for the discussion on the exploration results. On top of that experts appreciated the commenting and separation functions, that help them to confirm identified findings.