Anchorage: Visual Analysis of Satisfaction in Customer Service Videos via Anchor Events

Automatic satisfaction evaluation has been conducted on diverse data types, such as texts [6, 14, 7], eye gaze trajectories [15], and electroencephalogram signals [16]. These approaches rely upon a high active participation rate and specific devices that are both uncommon in many service scenarios. Unlike other data sources, videos are more accessible, provide continuous input for in-depth satisfaction analysis [17], and do not impose an extra cognitive burden on customers [18]. For example, surveillance videos in retail stores [19] and crowd-sourced web camera videos [18, 20] have been analyzed for customer responses to products and services.

Many video-based approaches have focused on extracting facial expressions as frame-level features and evaluating overall video-level satisfaction [21, 22, 23]. Yolcu et al. [24] accumulated all the emotions of different customers as a proxy to estimate their satisfaction. They also considered the head pose in tackling poor performance when the targets’ faces are occluded. However, these methods were naively evaluated on how accurately the facial expressions in the videos were identified but not the satisfaction. They only applied a trivial model between emotion and satisfaction.

Besides the visual channels, acoustic information is also crucial for satisfaction estimation. Park and Gates [8] derived several prosodic and lexical features for SVM to model satisfaction. Seng and Ang [11] fused affective features in visual and acoustic channels with a linear model bespoke to satisfaction evaluation. Ando et al. [9] introduced a hierarchical framework to combine emotional features in the conversation and individual utterances. These approaches provided data-driven satisfaction scores for videos and were tested against real and improvised datasets. However, a video-level satisfaction score cannot identify the critical turning points and fails to distinguish the counteracted cases. Also, these approaches put little or no emphasis on the agent’s behaviors, which could be the antecedent incidents that affect the customer’s consequential reactions [1, 17]. Our work explores multimodal fusion with behavioral features and event contexts to establish background associations for interactive satisfaction analysis.

A plethora of visualizations has been proposed to summarize and represent video data in the community [25, 26, 27, 28, 29, 30]. Emotional features are commonly extracted from videos for many application problems. EmotionCues [25] summarized the emotional dynamics in classroom videos and highlighted model uncertainties by a stream graph design. EmotionMap [26] and E-ffective [29] proposed a map-based and spiral-based design to provide a temporal overview of the affective dimension in multimedia videos. These visualizations effectively presented visual cues for notable moments. However, videos often require more features than emotions alone to serve various domain-specific analytical purposes.

Recently, more modalities have been used to provide additional information missing from the visual channel. EmoCo [31] explored the emotional coherence across facial expressions, audio emotions, and transcript sentiments to mitigate over-reliance on a particular modality. Li et al. [32] visualized head pose with mouse movement data to connect multimodal behaviors in online proctoring. VideoModerator [33] engineered risk-related features from images and transcripts to assist live stream moderation. Besides verifying features, analyzing multimodalities has been useful in interpreting multimodal models [34, 35], querying large video collections [36], and annotating think-aloud usability test videos [37, 38]. Although integrating more relevant data sources increases the credibility of analysis results [39, 40], it inevitably increases cognitive loads in comprehending them. We utilize a set of well-coordinated views to facilitate the intuitive interpretation of multimodal features. In particular, we propose a scatter-based metaphoric visualization design to summarize the multimodal features and show satisfaction progression. We leverage machine logs to contextualize and grant procedural understanding to customer service videos. It avails a new perspective in summarizing video sequences.

According to Höferlin et al. [41], video visual analytics has three main goals: status determination, event detection, and model generation. Status determination identifies frame-level features such as tracking objects. Contrarily, the other two goals consider a larger portion of the video. While event detection seeks to locate the moment when a specified event occurs, model generation aims at mining common patterns from video collections that can later be used to detect events. Our work focuses on model generation to make sense of unlabelled satisfaction patterns.

As the desired patterns are vaguely defined, analysts have to explore a large low-level event space to generate high-level concepts [37]. The frame-shot-scene hierarchy in movie analysis [42] and the object-event-tactic hierarchy in sports video annotation [43] can be viewed conjunctively to illustrate the complexity and interdependency of video events. To streamline the exploration, EventAnchor [44] traced visually available objects in racket sports videos and denoted their critical change of states as anchors. These anchors were plotted on the screen to indicate the objects’ locations for interactive calibration of the machine errors. However, the computer vision-based anchors are limited in numbers and cannot be trivially applied to acoustic channels and other modalities. Also, the visual effects, such as scoreboards and scene changes, are absent in many real-life video recordings to provide the event contexts. Inspired by the anchor concept, we extend its usefulness in understanding multimodal video events and generalize it into scenarios where videos are recorded along with sequential records. We leverage semantically meaningful events in service procedures to prioritize investigative efforts among different anchors. We propose a semi-automatic framework that generates anchor candidates with anomaly detection methods and supports candidate validation with intuitive visualization designs.

Discrete event analysis usually applies to log data such as computer system records [45], which have a timestamp for each event record and a semantic meaning for each event type. The multi-scale temporal structure of these events is often harnessed for visual summarization. For example, event sequences can be aggregated by multivariate regular expressions [46] and hierarchical clustering [47] to serve different level-of-detail requirements. Distinguishing branches [48] and bundling frequent patterns in event sequences [49] have facilitated a further understanding of the diverging and converging event evolution patterns. A recent survey [50] has summarized the design space for event sequences.

Through multi-scale overviews, analysts can visually compare the temporal characteristics of different event sequences to locate abnormal behaviors [51] and process drifts [52]. Guo et al. [53] proposed a VAE-based approach to detect arbitrary ordering, absence, and duplication of events. We borrow ideas from this line of research to discover unusual service procedures. We employ a similarity-based method to find temporal anomalies and a Markovian technique to detect sequential anomalies. They formulate operational anchors and contextualize the multimodal behavioral anchors for evaluating customer satisfaction.

Customer satisfaction is widely defined as the fulfillment of customers’ expectations with the perceived service quality [1, 2]. However, as suggested by the highly subjective terms “expectation” and “perceived,” satisfaction is a complicated construct with various interpretations by different people. Therefore, services are usually recorded to prevent misinformation and avoid conflicts in complaints, providing rich sources of recordings for analysis.

Customer service refers to the organization’s assistance and service for their customers before, during, and after-sales [2]. For example, contact centers provide customer services through phone calls for handling inquiries, managing orders, and troubleshooting issues [10]. While kiosks, mobile applications, and virtual assistants have been established to let customers serve themselves, staff-assisted customer services remain irreplaceable because of regulatory requirements, business procedural complexity, and insufficient machine capabilities in directing human intents [54].

Customer services are characterized by their goal orientation and communication dynamics. We focus on the typical setting where a customer service agent assists a client in completing domain-specific goals. We outline a typical workflow in Fig. 1 and clarify the terms used in the paper. First, the agent assists the client in specifying their intents and interpreting their needs at the beginning of a service. Second, the agent follows internal guidelines to derive a list of operations and guide the client through the sequential service procedure. Third, the agent and the client usually take turns communicating and performing the operations. For example, to verify a client’s identity, the agents can ask questions verbally, prompt for entry, or make a database query for the profile. Finally, the executed operations formulate a service record that may deviate from the agent guideline due to complex real-life situations. The service is usually recorded to be the service video.

We adopted the design study methodology [55] to characterize the domain problems. During the past two years, we have collaborated with a domestic information technology company that digitalizes public services and develops remote service terminals for tax authorities. The terminals connect agents and clients at different locations through video communications. It facilitates essential operations such as transmitting legal documents and processing digital payments. The equipment collects video recordings and machine logs of the staff-assisted tax filing services.

To understand their workflows and satisfaction evaluation methods, we conducted contextual inquiries and semi-structured interviews with three frontline tax officers (E1-3) with over 5 years of service experience. We interviewed a professor (E4) with 10 years of research experience in customer relations to gain a second opinion from the marketing field. Under their influence, we read domain literature to understand important concepts and link them with visualization studies. We held a series of remote meetings with three business analysts (E5-7) with over 5 years of public service experience from our industry collaborator to refine design requirements, adapt previous findings to our application, and verify the iterative designs. None of E1-7 is a co-author of this paper.

Our goal was to design a satisfaction evaluation system for customer service providers to identify satisfaction patterns for improving their services and workflows. We identified six design requirements to support the development of Anchorage.

1. R1

   Rank satisfaction by objective metrics. Clients seldom provide satisfaction feedback after services. When they do, their self-reported evaluation is prone to cognitive biases such as the peak-end rule [5] nonetheless. E1-2 added, “some clients rushed to leave, so they randomly clicked any buttons.” The large video collections also require a ranking order to prioritize videos of interest. The system should provide uniform and objective assessments based on users’ behaviors.

2. R2

   Contextualize the satisfaction evaluation with operations. Customer behaviors should be interpreted with the antecedent events [4]. For example, expecting smooth services, clients would perceive repeated and interrupted operations as troublesome and unsatisfactory, resulting in a negative emotional response. However, clients have diverse affective reactions to provocative actions. E7 pointed out that “some people keep a poker face, but they could be furious,” suggesting the unreliability and inadequacy of using emotional features only. The system should incorporate procedural considerations as the common ground to explain and evaluate clients’ behaviors.

3. R3

   Show satisfaction progression in a service. Automatic methods usually aggregate frame-level evaluations to model satisfaction [11]. However, the aggregated service score is inferior in differentiating counteracted cases. E5 proposed a satisfied case with a client showing unsatisfied behaviors at first but becoming more satisfied with the service at last. The case would be underrepresented in an accumulative service-level satisfaction score. Assessing satisfaction by individual operations naturally magnifies their contributions to the overall evaluation [9]. The system should visualize the dynamic satisfaction progression to reveal the causal relationships between behaviors and satisfaction.

4. R4

   Provide an overview of the service record. The service records provide sequential and temporal information to indicate the service smoothness. The records of smooth services usually match the typical workflows described in the agent guidelines explicitly. Experienced agents (E1-3) could easily identify deviated operations when they read the records in semantically meaningful terms. The operation duration also implicitly hints on the procedural difficulties and the service status. The system should present adequate event contexts to foster procedural awareness and segment the services properly.

5. R5

   Highlight the anomalous operations. Satisfaction generally follows a steady progression with previous states. A significant turning point could indicate a potential satisfaction pattern induced by internal factors (e.g., exceeding expectations [1]) and external factors (e.g., agents’ misconduct [4]). E4 stated that looking into the “peaks” and “troughs” of the satisfaction level would help derive more managerial insights. They are worth more attention to be further investigated. The system should distinguish uncommon satisfaction development to identify critical transition moments.

6. R6

   Support interactive navigation of original videos. Video recordings are the strongest evidence in evaluating satisfaction. Yet, reviewing the videos from scratch is inefficient. Features extracted by machine learning models are helpful, but they might suffer from model uncertainty and multimodal interactions [35]. E6-7 expressed a need to validate the features when they convey “unreasonable and contradictory meanings.” Also, the dynamics between agents and clients are challenging to define and detect. The system should support various interactions to streamline the fast location of interested events.

We analyzed 20 authentic videos from E5-7 regarding tax service assistance in local government tax authorities. These videos had no self-reported satisfaction levels. Most clients showed a neutral face, and the services operated on average time. They could only reflect neutrally satisfying cases, as E6 and E7 provided their evaluations as ground truths. Obtaining more recordings was challenging because the Covid-19 pandemic lockdowns limited our presence at the local tax offices to obtain the client’s consent. To the best of our knowledge, there are no publicly available datasets that include both video recordings and service records for evaluating satisfaction. To overcome the imbalanced distribution per satisfaction level, we created an improvised dataset for proof-of-concept.

Participants and apparatus. We invited 26 employees (4 females and 22 males) of our industry collaborator to participate in improvising. These participants improvised either as agents or clients. Four business analysts were qualified as agents because they had expertise in the service workflows. The other participants were included as clients if they had visited government authorities for staff-assisted public services. The recording was taken with the collaborator’s terminals to simulate real illumination and occlusion settings. The study was approved by the internal IRB (#HREP-2021-0162), and the videos were recorded with written consent.

Designs and setup. We designed and exemplified typical service scenarios with different satisfaction levels. Four satisfaction types were deduced by observing the collected footage’s workflows and interviewing frontline agents (E1-3). They include:

1. ST

   a SaTisfied service with a shorter completion time than expected. The agent delivers clear instructions and demonstrates proficiency in completing their tasks. The client is thankful.

2. NM

   a NorMal service with matched expected completion time. The agent controls the time of each operation to be around average. The client is given no instructions.

3. DA

   a Dissatisfied service about the Agent with a longer completion time than expected. The agent demonstrates inattentive behaviors (e.g., using mobile phones and chitchatting with others) to prolong the service. The client is annoyed.

4. DP

   a Dissatisfied service about the Procedure with a longer completion time than expected. The service procedure is interrupted by the malfunctioned terminal, which requires the client to repeat certain operations. The client is annoyed.

We expressed the satisfaction types in high-level terms as guidelines. The participants created their own speech and reaction to improvise the services. The agents were asked to keep a neutral face to prevent emotional contagion [13]. The clients’ reactions were described in abstract terms such as “being thankful” and “being annoyed.” We refrained from scripting the scenarios to sustain spontaneity, enhance generalizability, and avoid the curse of knowledge.

The service scenario is about amending membership of social security insurance. Historical records show that the average time for the service is around 8 minutes. Since expectation significantly impacts satisfaction [1], we communicated this information as the expected time to control the temporal expectation for all participants. The typical process involves nine steps from initiate to close, as shown in Fig. 3B1. Our service scenario’s workflow is transferable to a task-oriented dataset about customer service [54].

Result. We collected 61 service videos, with at least twelve for each satisfaction type. We ensured no clients repeated acting in the same type. The total duration of the services is 5.8 hours, and each spans 3-12 minutes, averaging 5.7 minutes. The corresponding satisfaction type labels the ground truth of the videos.

We processed the visual and audio channels decomposed from videos separately and aligned them with the parsed machine logs.

Visual features. We detected the bounding boxes of faces on every frame with YOLO5Face [56] and applied triangular smooth to reduce glitches. Since occlusion is still challenging for facial expression recognition (FER) [57], we detected the head pose with FSA-Net [58] to validate the reliability and reduce the impact of misclassification. We adopted a FER model [59] and aggregated the output discrete emotions into three large classes (positive, neutral, and negative) because the correspondence between discrete emotions and satisfaction is unclear [1, 24]. E5-7 were confused with the role of sadness and fear in evaluating satisfaction in customer services. Moreover, sacrificing granularity for generalizability is a common approach in the affective analysis [17].

Audio features. We applied a speaker diarization algorithm [60] to remove noise, locate speech segments, and cluster utterances by speakers. Since there are two speakers in a video, we registered the agent with heuristics, such as identifying the common speaker across two videos with the same agent. The audio segments are piped into an audio emotion classification model [61]. The discrete emotion outputs are also aggregated, as in facial expressions.

Event features. The machine logs contain discrete events that describe operations in the service records. However, an operation could be represented as multiple unstructured free-form text messages due to inconsistent coding styles. We first transformed the logs into tuple representations that contain a timestamp, an event type, and a list of log parameters for analysis. A service is identified by matching a beginning and an ending log message with the same terminal request ID. We aggregated the co-occurring raw event types that are semantically related, and confirmed the aggregated event types, denoted as operations, with E5. The nine operations are used to segment the videos (Fig. 3B1). We counted the logs with the same operation $e$ to obtain a service record vector $E=[(e_{1},n_{1}),(e_{2},n_{2}),...]$, where $n_{i}$ is the count of consecutive $e_{i}$.

The primary purpose of operational anchors is to lift the burden of status determination (as discussed in Sec. 2.3) for analysts when they watch the videos. The operational anchors can segment operations in service records and grant semantic meanings to segments (R4). They also introduce event structure to summarize video content. To prioritize anomalous events (R5), we employed a similarity-based method to find temporal anomalies and a Markovian technique to detect sequential anomalies from the service records. The service record vector $E$ is piped into the following algorithms to obtain the corresponding anomaly scores.

Temporal anomaly locates uncommon durations of operations. The Principle Component Analysis (PCA) [62] is a popular unsupervised method for system log analysis detecting anomalous discrete events. It computes the similarity between the input and the labeled sequences based on the assumption that anomalous sequences should be dissimilar to normal ones. Service records labeled as normal $E_{n}$ are further aggregated by the operations to obtain fixed-size vectors. They are reduced to $k$ principal components to formulate the normal space $S_{n}$. A service record is said to be anomalous if $||y||^{2}>Q_{1-\alpha}$, where $y$ is the projection length to $S_{n}$, and $Q_{1-\alpha}$ is the confidence threshold defaulted at 95%. We had considered another popular method in system log analysis, invariant mining [63]. However, it is tailored to rigorous procedures in software systems and has limited generalizability. Meanwhile, PCA has the advantage of high interpretability and does not require a large training set.

Sequential anomaly locates uncommon chronological orders in service records. The Markov chain model [64] learns a transition probability distribution $P$ of different discrete states at each time frame in normal sequences. It assigns the service vector $E$ with:

where $T$ is the fixed window size. A non-zero constant $\epsilon$ is introduced to prevent zero probability when a particular sub-sequence has not appeared in the training set. We scaled up and sampled down the event records to create fixed-sized event vectors. We set the anomalous threshold for operations at $|1/n|$ such that the transition occurs at least once in the training set and for service at a constant that captures 95% variance. The Markov chain model is chosen for three reasons: (1) It does not require large training data; (2) It has good scalability by enlarging the window size to facilitate a large number of log records; (3) It identifies the exact operation when it deviates from standard procedures (R4). Moreover, customer services have predefined procedures (agent guidelines) acting as the normal training set. Although the Markovian model is not designed to identify missing and abundance events [53], its anomaly score would still reflect these conditions as they would appear in the wrong place. The Markovian model is well-suited to detecting repeated and out-of-sync operations (R2).

The behavioral anchors are the multimodal satisfaction evaluation. Similar to [24, 11], we adopt a linear model to generate a customer satisfaction score. We extended the model to cover event duration rather than affective status only (R2). The model combines facial expression $v$, audio emotion $a$, and events $e$ to evaluate satisfaction. The customer satisfaction score for a service $CS_{s}$ is calculated by:

where $N$, $M$, and $T$ denote the total number of frames, utterances, and operations. $w$ is the weights of each channel defaulted as equally weighted. We also obtained the operation’s satisfaction score $CS_{e}$ for each modality by confining the summation scope to individual operation and modality. $f$ is the normal standardization across all services. $z_{e}$ is the z-score for the event duration. We grouped them by operations before standardizing because repeated operations are usually shorter and obfuscate the calculation. For emotional responses, we assigned a magnitude weight $m$ to each discrete emotion and adopted the scheme proposed by previous work [11]. In general, positive emotion has a value of +1.0, neutral emotions are 0.0, and negative emotions tend to -1.0. We slightly modified the weightings of anger to -1.2 and disgust to -1.0 based on the domain literature [12] and discussions with E5-7. A large value of $CS_{s}$ indicates high satisfaction and vice versa. All of the above settings can be reconfigured to adapt to other needs.

The Service overview (Fig. 3A) provides an overview of all the service videos. It lists all the videos and supports fast comparison over multiple videos to search for a service of interest. Each list item (Fig. 3A1) contains three columns that show different satisfaction metrics (R1). The color encodings are unified for the visual interface (green for visual, red for audio, and purple for event). The leftmost column displays the basic information of the video. The horizontal bar chart shows the temporal and sequential anomaly scores described in Sec. 4.2. Identified anomalies are represented by filled color, and normal services are in striped color. The rightmost column is a vertical bar chart showing the satisfaction scores $CS_{s}$ of different modalities described in Sec. 4.3.

The middle column is a scatter-based design called the buoy chart (Fig. 4C). It summarizes the multimodal satisfaction scores of individual operations into a single graph. An operation is represented by two dots, which are in different colors to indicate their types (visual or audio). The vertical position encodes the dot’s satisfaction score $CS_{e}$ of its type. The dots close to the horizontal line have lower opacity, making them visually insignificant. The horizontal position encodes the reversed event score for all dots. The negation is designed to match the quadrants heuristic by converting a negative score, which denotes a shorter duration, into a positive. Moreover, two dots belonging to the same operation will be linked vertically to show a connected visual component based on the principle of continuity. The vertical and horizontal scales are centered at zero and capped within a threshold.

Justification. We developed a metaphor to flatten the buoy charts’ learning curve and lower the bar of visualization literacy [65]. The buoy chart employs the buoy metaphor as a buoy attaches to an anchor. A dot is referred to as a buoy. Normal services should have many ordinary operations; thus, dots are clustered around zero. While these buoys in proximity float on the surface imperceptibly, the significant buoys sink and rise to become visually apparent anchors. These outliers highlight anomalous operations with potential satisfaction patterns (R5). They detect counteracted cases (R3) when the polarized anchors are seen together with the near-zero aggregated scores (e.g., the audio channel of Fig. 3A1).

Moreover, the buoy chart can be interpreted with the quadrant heuristic in Fig. 4A. We estimate the overall satisfaction by checking the quadrant with the most anchors. For example, Fig. 3A3 shows many floating buoys and a few sinking anchors. The anchors’ positions suggest that the client exhibited negative emotions in a few prolonged operations. It shows how the buoy chart efficiently summarizes the distinguishing patterns (R1). The buoy chart is the visual alternative for multimodal fusion (e.g., Eq. 2) which may require extra effort to optimize the cost functions.

The buoy chart effectively encodes multimodal characteristics. Fig. 4C demonstrates the visual patterns of the three inter-modal interaction types summarized by Wang et al.[35] ($e_{1-3}$ for dominant, $e_{4}$ for complementary, and $e_{5}$ for conflicting modals). We use the four cases in Fig. 3 as examples: A2-A3 dominantly express strong emotions in one channel; A4 conveys negative emotions by complementing both channels; A1 shows conflicting behaviors for some operations. These visual cues help analyze the clients’ emotional profiles (R2), which inform the subsequent analysis.

The buoy chart can be augmented to address various design issues. Techniques applicable to scatter-based designs are also likely effective for the buoy chart. For scalability issues, we can reduce the dot size and superpose bar charts to the sides to observe the operation’s distributions in densely populated regions. To prioritize the most important items, we can use a quadrant-based heatmap to filter and rank the videos by anchor patterns of interest.

Design alternatives. We focused on designing straightforward and standardized charts to suit the diverse background of our target users, i.e., service providers and agents. Building abstractions is a popular strategy for handling numerous videos in video collections. We implemented dimension reduction techniques to generate video clusters and visualize outliers to reduce review efforts. However, the techniques neglect temporal relationships and could not detect counteracted cases. Although we can set up exemplars to guide the clusters, it is challenging to make novices aware of the technical assumptions and avoid over-reliance on unsupervised results. We used the “view sequentially” strategy [66] instead and provided the satisfaction scores in different modalities as the ranking orders.

As an alternative to the buoy chart, we considered using the stacked bar chart to display the operations sequentially, as in Fig. 4D. It is visually apparent when the multimodal scores are dominant. However, the complementary and conflicting modalities challenge the interpretability of the chart for lacking quick decision rules. The chart also creates confusion when displaying positive and negative values together, and suffers scalability issues with more operations. Another option was the parallel coordinates (Fig. 4E). However, the stronger intra-modal scalability cannot compensate for the visual clutter of lines when performing inter-modal comparisons. It is also complicated to compare three modalities simultaneously to find anomalous operations. Fig. 4C-E share the same set of data to highlight their difference.

This view (Fig. 3B) supports operation-level anchor exploration based on its service record. The timeline-based visualization (Fig. 3B1) represents each operation with a column of visual components. The horizontal position encodes the service time. Each column contains four rows. In the top row (Fig. 3Bi), we visualize the operations, turn-taking information, and the indicator of sequential anomalies to provide the procedural context and indicate sequential inconsistency (R4). The line is colored pink for the agent’s turn or yellow for the client’s. The triangle icon indicates that the operation is sequentially anomalous, as in Fig. 5A.

The bottom three bars (Fig. 3Bii) illustrate the statistics around the event, visual, and audio modalities, respectively. The first bar in purple shows the duration of the operation. The portion in dark purple indicates the amount of time exceeding the operation’s average. It signals a longer-than-usual operation and can be considered a temporal anomaly. The second bar in green and the third in red represent the proportion of time with detected features for the visual and audio channels. The striped pattern encodes the absence of features, such that green is for obscured client’s face and red is for silence. For example, Fig. 3Bii shows that the client’s face is not obscured for the whole operation, and the conversation lasts for about one-third of the time. These indicators provide background information about the reliability of the detected features. The visual components are associated with other views, so clicking on them can navigate to the multimodal features and the original service videos (R6).

The lateral buoy chart (Fig. 3B2) shows the satisfaction progression. The chart’s horizontal encoding follows the timeline above, so all dots are located in the middle of the operation. Its correspondence with the buoy chart is illustrated in Fig. 4B-C. While the two charts share the same metaphor, there are subtle differences. Each operation is represented by three dots, including the event. Here, the vertical position utilizes the z-score to unify all modalities. The visual and audio scores are summed over the operation and further standardized within the selected service. For example, $e_{5}$ in Fig. 4B contains vastly deviated scores for all modalities, while the $e_{3}$ counterparts have average scores. An anchor icon denotes higher values than two standard deviations. The scale helps detect the most anomalous service operations (R5). The buoy’s size encodes the absolute deviation rank to draw attention to the most influential anchor. The more significant deviation, the larger the buoy. The drawing order favors smaller buoys to prevent occlusion and visual clutters (see Fig. 5A).

Justification. The lateral buoy chart bridges the gap between the buoy chart and the timeline. We did not explicitly encode the operations shorter than average in the timeline because they could distort the layout. Also, they are less significant as shorter events usually have fewer behavioral anchors to verify. The missing temporal anomalous information is covered in the lateral buoy chart with the introduction of event buoys. Using familiar visual elements and metaphor reduce the burden of learning a new visualization. The correspondence could introduce higher efficiency when users are familiar with the system.

Design alternatives. During the design process, we referred to the event sequence design space [50] and quickly eliminated hierarchy-based, Sankey-based, and matrix-based designs because of the unfit tasks. We created the current design by combining bar charts and timelines for simplicity and familiarity. For visualizing feature progression, Zeng et al.[25] proposed five designs to show emotion flows. However, these designs are limited in visual summarization power because they lack event contexts. Our lateral buoy chart combines multiple visual elements to coherently express the satisfaction progression in service operations.

This view supports interactive navigation of the original videos (R6). We adopted the periphery plot [67] as the operation summary (Fig. 3C1). In the middle focused detail view, we fused the facial and audio features to assign an activation value $v_{i}=\{-1,0,1\}$ to frame $i$. The fusion favors non-neutral emotions with higher priority given to negative ones because they have a greater impact on satisfaction [12]. The activation values are plotted to show an overview of the operation. Brushing selects the period for the below features. The periphery plots on both sides allow quick navigation to consecutive operations and contextualize the focused operation with neighbors. The three bars show the count of activation values.

The audio and visual channels are one-dimensional shaded matrices (Fig. 3C2) that encode the positive and negative outputs of facial expressions and audio emotions. Obscured and muted frames are encoded with the stripped pattern as before. The head pose information is shown in line charts (Fig. 3C3). The chart scales are written with the semantic meaning of the angles directly. The proximity between visual features and head pose allows users to verify emotions with occlusion from looking down. Clicking on the views (Fig. 3C4) seeks the time point in the video.

This study describes the satisfaction evaluation process of E5 for services provided by two of his actual subordinates, S1 and S2. He was tasked with rating the services and the agents by exploring the improvised dataset (Sec. 3.3) filtered by S1 and S2. He has been closely involved in the problem characterization and improvised dataset formulation, but he had not used Anchorage nor seen the videos before the case study. He knew about the satisfaction scenarios typically known to frontline agents. He was encouraged to follow the think-aloud protocol during his exploration.

Service exploration (R1-2). Beginning from the service overview (Fig. 3A), E5 first ranked the videos by descending service satisfaction score. ST09 (Fig. 3A2; Video names were masked during the exploration) was ranked number one on the list. He noticed that all metrics favored the service because all bars showed positive values, and there was a distinctive audio anchor in the buoy chart. Reading the basic info on the left column, he quickly determined it should be the ST type. He selected the service to see who the agent was (S1) from the service video view. Then, he continued browsing other videos. After a few attempts, his attention was caught by NM08 (Fig. 3A1). He examined the near-zero audio satisfaction score and the polarized buoys in the buoy chart and suspected it was a counteracted case. He felt interested in the case and wanted to know about the conflicting behaviors. He selected the service and proceeded to explore the service record.

Operation exploration (R3-5). Looking at the anchor exploration view (Fig. 3B), he first checked the procedure summary (Fig. 3B1) and did not find many sequential or temporal anomalies. Most of the agent’s operations finished in time, and no procedures deviated from agent guidelines. He ruled out the DP type. He turned to the lateral buoy chart (Fig. 3B2) and discovered the visual anchor indicating significant negative facial expressions. He also noticed that the previous operation of the anchor was very positive. Revisiting the operations’ names (“Execute” and “Pay”), he had a clue about the incident but needed more evidence. He clicked on the anchor icon to investigate the critical transition moment.

Feature verification (R5-6). Entering the multimodal feature navigation view (Fig. 3C), he found that the most negativity was located in the latter part of the “Pay” operation from the periphery summary and the visual feature (Fig. 3C4). He clicked on the orange frames at C4 to navigate the service videos (Fig. 3D). By watching the original video, he concluded that the negative emotions came from having to pay but not because the agent was inattentive. He rejected the DA type and declared it the NM type. However, he wondered why the client had many positive behaviors during the previous operation as he read the left periphery plot (Fig. 3C1). He clicked on the plot and repeated the feature verification analysis. He was intrigued by the fact that the client was only texting on his phone the whole time. This reinforced his NM rating, despite the high satisfaction score on the visual channel. He became confident about Anchorage’s ability to detect counteracted cases. S1 accrued a few NM and ST cases to be rated as good performance.

Satisfaction evaluation (R1-6). By ranking the videos in ascending scores, E5 found DP07 (Fig. 3A4) to have the lowest satisfaction score in both visual and audio channels. He noticed that the service was detected with both temporal and sequential anomalies. He reviewed the service in the anchor exploration view (Fig. 5A). He noticed that the client took longer than usual to upload his files. The procedure was also repeated twice such that it could annoy the client. He concluded that the service belonged to the DP type because he knew it was the only scenario.

A DAP case. The visual anchor under the “Verify” operation drew E5’s attention. He was confused because the typical DP scenario does not stage like this. He investigated the visual anchor and saw the lasting orange frames in Fig. 5B1. From the head pose chart, he observed that S2 had been looking down. He clicked on the behavior (Fig. 5B2) and derived from the videos that S2 had been playing on her phone (Fig. 5C). He speculated S2 might have combined the DA and DP types to create a more unsatisfied DAP case which annoyed the client with the prolonged procedure and the inattentive agent. Nevertheless, he rated S2 as having poor performance as an agent, but good performance as a business analyst in taking the initiative to create new requirements.

We conducted the user study with a between-subject design and two conditions on the system used to evaluate the effectiveness and usability of Anchorage in evaluating customer service videos.

Apparatus. Presenting multimodal features in a VA system generally leads to better task performance than a baseline system without much computational support [32, 44, 33]. However, it could be unfair to compare Anchorage with both operational and behavioral anchors to a naive baseline because of the wide interaction gap and the compound effect. In this study, we aimed to evaluate whether the introduction of event context (i.e., operational anchors) enhances the performance of satisfaction evaluation tasks. We created the baseline system (Fig. 6) by ablating the event-based visualization components in Anchorage, namely, the buoy chart, the anchor exploration view, and the periphery plots. The remaining parts visualize multimodal features beneficial to automated satisfaction evaluation [11, 9]. The user’s mouse actions were logged for provenance analysis.

Data and tasks. We sampled videos from the dataset described in Sec. 3.3 for the satisfaction evaluation tasks. From a pilot study with E5-6, we estimated that participants could annotate three videos in ten minutes. We randomly selected 12 videos for the study in consideration of workload and duration. The videos are selected with two constraints: 1) equal coverage of all four satisfaction types, and 2) acted by different clients. They ensure the samples’ diversity and independence. We used two videos for demonstration, and participants evaluated the remaining. The remaining ten videos span 68.6 minutes and formulate the ten satisfaction evaluation tasks (T1-T10 in Fig. 7 with masked video names). In addition to rating the client satisfaction on a Likert scale from 1 (low) to 5 (high), we asked them to evaluate the agent proficiency and the service smoothness to further distinguish the videos.

Participants. We adopted snowball sampling starting from the colleagues of E5-7 to recruit 24 participants from our collaborators’ company (8 female, 16 male; age: Mean $(M)=28.4$, Standard Deviation $(SD)=5.3$). While 16 participants are undergraduates in STEM disciplines, others attain diplomas in diverse backgrounds. They have 1-13 years of related experience in customer services ($M=5.1$, $SD=3.7$). They were compensated with $7.50 USD equivalent upon completion. We randomly assigned the participants to use the Anchorage system ($P_{A}$) and the Baseline system ($P_{B}$).

Procedure. The study was conducted remotely due to quarantine restrictions. The participants could access the assigned system deployed online. We first obtained their consent, and introduced the research background and the system’s functions via recorded videos for around 12 minutes. After three minutes of free exploration with the training examples, the participants should complete the ten tasks using the assigned system. Since we did not enforce a time limit on the tasks, we provided cash incentives to prevent low-quality responses. Each satisfaction rating is $1.5 USD (maximum five rewards), if it matches the ones by E5-6 within one point scale. Finally, they filled in a questionnaire about the assigned system and their background information. All sessions spanned between 30-90 minutes ($M=56.5$, $SD=18.3$).