Human-Collective Visualization Transparency

Spatial swarm robotics are inspired by self-organized social animals [2], such as flocks of birds [15] or schools of fish [5], and exhibit intelligent, emergent behaviors as a unit, by responding to locally available information [11]. Spatial swarms rely on distributed, localized, and often implicit communication, in order to spread information across the swarm. Salient movements warning individual entities of a nearby predator [16] or rapid changes in acceleration, such as a streaking bee guiding a swarm in a particular direction [4], are examples of spatial swarm communication. Spatial swarms follow basic rules of repulsion, attraction, and orientation that enable individual entities to position themselves relative to neighboring entities [17]. Couzin et al.’s model states that individual entities in the zone of repulsion attempt to maintain a minimum distance from their neighbors, striving to avoid collisions, while entities that are far from their neighbors will move closer, as a factor of the zone of attraction. The zone of orientation causes individual entities to align themselves in the same direction as their close proximity neighbors.

Robotic colonies are decentralized systems [18] composed of entities that exhibit unique roles, such as foraging, which adapt over time to maintain consistent states in changing conditions [19]. Biological colonies use a centralized space to share information, such as inside of a nest, or embed information into the environment, such as ants depositing pheromones to indicate a route from a food site to the nest [20]. Positive feedback loops support gaining a consensus to change the colony’s behavior [21]. Recruitment for potential changes increases until a quorum is reached. The colony will transition into a decision-making state, implementing different strategies, such as the bee waggle dance [4], in order to reach a consensus. Once a consensus has been reached colony members will indicate to one another that the colony is ready to move to the new nest site, such as physically picking up ants and transporting them to the new nest site [22]. Negative feedback mitigates saturation issues, such as food source exhaustion [18].

Transparency has been defined as the communication [23], process [24], method [25], mechanism [26], property [27], emergent characteristic [28], explanation [29], and quality [30] of an interface to support operator comprehension of a system’s intent. Chen et al.’s transparency definition [30] is commonly used in the robotics domain and uses the three level Situation awareness-based Agent Transparency (SAT) framework, which leverages the human operator trust calibration 3Ps model (purpose, process, and performance) with performance history [31], Endsley’s situation awareness model [32], and the Beliefs, Desires, and Intentions Agent framework [33], in order to achieve transparency.

Transparency for this manuscript builds on Chen et al.’s definition and is defined as the principle of providing information that is easy to use [34] in an exchange between human operators’ and collectives to promote comprehension [35, 36, 37], intent, roles, interactions [38], performance [30], future plans, and reasoning processes [26, 39]. The term principle describes the process of identifying what factors affect and are influenced by transparency, why those factors are important, how the factors may influence one another, and how to design a system to achieve transparency. A subset of transparency factors that originated from human-machine literature [1], shown in Figure 1, will be used to assess the visualization transparency. Five factors impact transparency directly and are represented as blue ovals connected to the transparency node by solid black lines. The three highest total degree (number of in degree + number of out degree) direct factors are explainability, usability, and performance (dark blue). Information and understanding (light blue) are not considered high degree factors, because explainability uses information to communicate and promote understanding. Explainability and usability, which is a multifaceted quality that influences the operator’s perception of a system, are used for the implementation of transparency in the visualization. Performance can be used to assess the visualization transparency by determining how well a human-collective team was able to produce an output when executing a task under particular conditions [37].

The yellow rectangles represent factors that impact visualization transparency indirectly. The timing, frequency, and quantity of information visualized on an interface, such as collective status or feedback, requires consideration of the human operator’s capability limitations [40], system limitations, as well as task and environmental impacts, in order to be explainable [41]. Human-collective efficiency and effectiveness can be improved by visualizing information, such as predicted collective states, in a clear and cohesive manner that helps alleviate the time and effort an operator must exert to integrate information in order to draw conclusions [42] and justify particular actions [43]. Poor judgements may occur if the operator lacks an understanding of what the collective is doing due to poor visualization usability, which may cause operator dissatisfaction. Visualizations that do not provide needed information to the operator and system control mechanisms to effectively influence collective behavior will hinder human-collective performance, lower operator situational awareness, impact workload negatively, and potentially compromise safety of the human-collective team. Understanding the relationships between the direct and indirect factors, and their relationship to transparency, is needed in order to assess metrics that can quantify good design of human-collective visualizations.

Three design methods that have been used frequently to optimize transparency and desired outcomes (i.e., effective team interactions and high performance) in traditional human-machine systems are: (1) provide system features, such as status, feedback, planning mechanisms, and engagement prompts, (2) use specific guidelines, such as Gestalt principles, to design a system, or (3) train the operators and system [1]. The majority of the collective visualization transparency literature has focused primarily on the first two design methods of providing system features and using specific guidelines to design the collective visualizations. Most evaluations used training, the third method, only to teach operators how to interact with their specific systems [44, 45], but did not analyze how different training strategies may impact transparency. More training may be required for different collective visualizations due to difficulties understanding the presented information. Determining what strategies have been used and their associated outcome success rates in remote teaming with collectives, as well as methods that can be evaluated more in depth, such as training, will aid in the development of transparency guidelines for collective visualizations.

The collective transparency literature that uses traditional collective visualizations (i.e., showing the position of all individual entities) has focused on assessing operator understanding of collective behavior and human-collective performance. A variety of individual collective entity features have been visualized for operators, including current and predicted future position [46] and heading direction [45], health (i.e., speed, strength, capability, and dispersion [47]), and status. The most commonly used visual icon for an individual collective entity has been a circle, where directional information was observed either as the entity moves across a 2-D space, or the circle also encompassed a line pointing in the heading direction. Some visualization design guidance came from subject matter experts [47] and the Gestalt principles [44]. The use of multimodal cues (i.e., visual collective state color coding and written messages, spoken messages, and vibrations) aided operators in the identification and response of signals during a reconnaissance mission (99.9% accuracy), as well as shorter response times, and lower workload. Sharing collective information via multimodal cues may alleviate an operator’s high visual loads when managing multiple tasks on various displays by increasing situational awareness [47] and promoting better transparency. Operators’ using a visualization incorporating Gestalt-based design principles perceived and approximated optimal swarm performance faster than operators using visualizations containing only individual entity position information [44]. Increased visualization transparency enabled operators to learn when to approximate optimal input timing.

Information latency, which occurs due to communication bandwidth limitations, and neglect benevolence, the time allowed for a collective to stabilize before issuing new operator commands, on operator understanding of future collective behavior are important considerations [46]. Latency affected operators’ ability to control a collective, but providing additional transparency via a predictive visualization, which showed the predicted location of each individual entity 20 seconds into the future, mitigated these effects. Operators using the predictive visualization with latency performed as well as operators who experienced no latency. Transparency of human-collective systems can be improved by implementing predictive visualizations of the collective and its entities by allowing the operator more time to think about their future actions. Operators will be able to balance span, the number of individual collective entities they interact with, and persistence, the duration of the interactions with visualizations that provide heading information [45]. Aspects, such as the presence or absence of Couzin et al.’s communication model states, visualized individual collective entities’ velocity, and individual operator characteristics, such as gender, have impacted the identification of swarming behavior [48]. Understanding the influence of factors is necessary to promote perception and comprehension of collective behavior to inform future operator actions.

Abstract collective visualizations have been proposed to improve operator understanding and influence positive collective behavior. Radial visualizations, using the SAT model [30] and heuristic evaluations analyzing the application of collective metrics on visualizations [49], as well as glyphs [50], bounding ellipses [51], convex hulls, and directed arrows [52] have been assessed. Similar system features provided in the traditional visualizations were conveyed using the abstract visualizations. The radial visualizations promoted perception (SAT Level 1) by displaying mission [53] and collective state information [54], such as the direction the entities left the hub to explore environmental targets. Predictions of future collective headings (SAT Level 3), that were provided by elongating the radial display in the direction of more collective support, aided operator actions. Visualizations providing predictive information are needed to achieve transparency for human-collective systems, regardless if the visualization is traditional or abstract. Operators using a glyph were able to acquire information regarding the collective’s power levels, task type, and the number of individual entities, via one icon [50]. Additional information about particular system features was accessible via pop-up windows. Designers of abstract collective visualizations can ensure transparency by providing redundant information via the collective icon and using supplementary information windows. Conflicting results were found for evaluations assessing whether traditional or abstract visualizations aided operators better during different tasks. Abstract visualizations during a go-to and avoid task in the presence or absence of obstacles [52] performed worse than the traditional visualizations, while abstract visualizations performed as well or better when perceiving biological collective structure [55] and under variable bandwidth conditions [51]. Further analysis is needed in order to determine which collective visualization will promote better transparency for a common task by investigating how transparency factors influence human-collective behaviors.

The Individual Agents (IA) interface, see Figure 2, exemplifies a traditional collective visualization by displaying the location of all the individual collective entities [13]. The interface was divided into three primary areas: 1) the central map, 2) the collective request area, and 3) the monitor area. The map, located at the center of the interface visualizes the respective hubs, their individual entities, discovered targets, and other associated information. Both the collectives and targets were rectangular boxes with distinguishing identifiers located at the center of the icon. The collectives had Roman numeral identifiers (I-IV), while the targets used integers (0-15). Discovered targets initially were white and transitioned to a green color when at least two individual collective entities evaluated the target. The highest valued targets were a bright opaque green (e.g., Target 0 in Figure 2), while lower valued targets had a more translucent green color (e.g., Target 9 in Figure 2). Targets that were within the collective’s 500 $m$ search range had different colored outlines, depending on the collective’s state: explored targets that were not currently favored had yellow outlines, explored targets that were favored had white outlines (e.g., Target 12 in Figure 2), and targets that were abandoned have red outlines (e.g., Target 13 in Figure 2).

The individual collective entities began each trial by exploring the environment in an uncommitted state, which transitioned to favoring as targets were assessed and supported. The individual collective entities committed to a target once 30% of the collective (60 individual entities) favored a particular target. The collective executed a move to the selected target’s location once 50% of the collective (100 entities) favored the target. The individual collective entities’ state information was conveyed via individual collective entity color coding: uncommitted (yellow), favoring (green), committed (blue), and executing (blue). A legend of the collective entities’ and target border colors was provided in the lower right-hand corner, see Figure 2. The number of individual collective entities in a particular state, or supporting a target was provided via the collective hub and target information pop-up displays, which provided detailed information, represented as gray rectangular boxes, displayed directly on the map in Figure 2. The information displays, when accessed, appeared in a particular location relative to the respective collective’s hub or target. The operator was able to move the information displays by dragging the pop-up display to a desired location.

The operator had the ability to influence an individual collectives’ current state via the collective request area, located on the lower left-hand side of Figure 2. The investigate command permitted increasing a collective’s support for an operator specific target. Ten uncommitted entities (5% of the collective population) transitioned to the favoring state after receiving and acknowledging the investigate command. Additional support for the same target was achieved by reissuing the investigate command repeatedly. The abandon command reduced a collective’s support for a specific target by transitioning favoring individual entities to the uncommitted state. The abandon command only needed to be issued once in order for the collective to ignore a specified target. A collective’s entities stopped exploring alternative targets and moved to the operator selected target when the decide command was issued, which was a valid request when at least 30% of the collective supported the operator specified target.

The collective assignments area logged the operator’s issued commands, shown in the upper right-hand corner of the monitor area in Figure 2. The log displayed what commands were issued with respect to particular collectives, for example, Collective I: Abandon Target 3. The green and red circles next to each command signified whether the command was completed (red) or currently active (green). An investigate command initially had a green circle and transitioned to red once ten individual entities received and acknowledged the investigate command for a particular target. Issued abandon commands for a particular collective and target remained active. Once a collective reached a decision, all prior commands associated with that particular collective were removed from the collective assignments log. The only command the operator was able to cancel was the abandon command, which required selecting the desired abandon command, in Figure 2, the “Collective I: Abandon Target 3”, selecting the cancel assignment button.

System messages indicated the actions taken by the operator and collectives. The illegal message was displayed when an operator requested an invalid command, and explained why the requested action was not viable. Three situations resulted in illegal messages. The first arose when the operator attempted to issue an investigate command for targets that were outside of the collective’s search region. The second situation occurred when the operator attempted to abandon newly discovered targets that did not have an assigned value (white targets). The last message arouse when operators attempted to issue decide commands when less than 30% of the collective supported a target.

The Collective interface, shown in Figure 3, provides an abstract visualization that does not represent individual collectives’ entities. The Collective interface was divided into the same three primary areas as the IA interface. The collectives were represented as gray and white rectangles with four quadrants and Roman numeral identifiers located at the top center of the icon. Collective state information was conveyed via the collective icon’s quadrants, color coding, and information windows. The collective icon’s contained four state quadrants (uncommitted (U), favoring (F), committed (C), and executing (X)), which represented the number of individual entities currently in each state, where a brighter white quadrant equated to larger numbers of individual collective entities. The square target icons had integer identifiers positioned on the upper right hand corner. Targets contained two sections, where the top green section represented the target’s value, where the brighter and more opaque the green, the higher the value (e.g., Target 8 in Figure 3). The blue section indicated the number of individual entities favoring a particular target, where the brighter and more opaque the blue, the higher the number of collective entities (e.g., Target 12 in Figure 3).

The collective interface operated similarly to the IA interface with some distinctions. A target was outlined in blue, demonstrated by Target 0 in Figure 3, when the collective’s support exceeded 30%. The target transitioned to a green outline and the collective was outlined in green when the collective began executing a move to the target’s location. The collective’s outline moved from the hub to the target’s location to indicate the hub’s transition to the selected target. The interface’s legend appeared in the upper left corner, see Figure 3.

The experimental procedure for both evaluations began with a demographic questionnaire, followed by the Mental Rotations test [57], after which the participants received training and practiced using the interface. Two practice sessions occurred prior to each trial in order to ensure familiarity with the underlying sequential best-of-n ($M_{2}$) and baseline $M_{3}$ models. The $M_{2}$ model trial was always completed first, in order to alleviate any learning effects from using $M_{3}$. The participants were instructed that the objective was to aid each collective in selecting and moving to the highest valued target two sequential times. A trial began once the respective practice session was completed. Each trial was divided into two components (one easy and one hard) of approximately ten minutes each. Splitting each trial into two components allowed the simulation environment to reset with 16 new (not initially visible) targets. The easy trial contained higher valued targets close to the hub, while the hard trial placed high valued targets further away. The easy and hard trial orderings were randomly assigned, and counterbalanced across the participants. The situational awareness (SA) probe questions [14], intended to serve as a secondary task, were asked beginning at 50 seconds into the trial and were repeated at one-minute increments. Six SA probes were asked during each trial. The trial was terminated once the team completed eight decisions, two per collective, or once six decisions were made, if the trial length exceeded the ten-minute limit. Decision times were not limited. A post-trial questionnaire was completed after each trial and the post-experiment questionnaire was completed before the evaluation termination.

The participants from both evaluations completed a demographic questionnaire, which collected information regarding age, gender, education level, weekly hours on a desktop or laptop (0, less than 3, 3-8, and more than 8), and their video game proficiency from little to no proficiency (1) to high proficiency (7). The Mental Rotation Assessment [57] required participants to judge three-dimensional object orientation to assess spatial reasoning within a scoring range of 0 (low) to 24 (high). The mode is reported in parenthesis for questions that required selection to a group.

Five participants were excluded from the IA evaluation due to inconsistent methodology (1) and software failure (4). The between visualization analysis is based on 56 participants from both evaluations. The first twelve decisions made per participant using the $M_{2}$ model were analyzed. The majority of the objective metrics were analyzed by SA level (overall ($SA_{O}$), perception ($SA_{1}$), comprehension ($SA_{2}$), and projection ($SA_{3}$)), decision difficulty (overall, easy, and hard), timing with respect to a SA probe question (15 seconds before asking, while asking, or during response to a SA probe question), or per participant. Non-parametric statistical methods, including Mann-Whitney-Wilcoxon tests and Spearman Correlations, were calculated due to a lack of normality. The correlations were with respect to SA Probe Accuracy and Selection Success Rate. The Collective evaluation data was reanalyzed using the same methods. Secondary research question’s hypotheses, associated metrics, results, and discussion are presented in Sections 5-8.

Assessing variables, such as the selected target value for each human-collective decision, is necessary in order to determine whether operators were able to perceive the target value correctly and influence the collectives positively. The objective of the human-collective team was to select the highest valued target for each decision from a range of target values (67 to 100). The selected target value is the average of all target’s respective values that were selected by the human-collective teams during a trial. The descriptive statistics for the selected target value per decision difficulty (i.e., overall, easy, and hard) are shown in Table 2. Participants using the Collective interface were able to influence the collective to chose higher valued targets, regardless of decision difficulty, on average; however, the Mann-Whitney-Wilcoxon test identified no significant effects between visualizations for the selected target value.

The SA dependent variable was SA probe accuracy, which is the percentage of correctly answered SA probes questions used to assess the operator’s SA during a trial [14]. Each question corresponded to the three SA levels: perception, comprehension, and projection [32]. Participants were asked five $SA_{1}$, four $SA_{2}$, and three $SA_{3}$ questions. The $SA_{1}$ questions determined the operator’s ability to perceive information about the collectives and targets, such as “What collectives are investigating Target 3?” The operator’s comprehension of information was represented by the $SA_{2}$ questions, such as “Which Collective has achieved a majority support for Target 7?” $SA_{3}$ questions related to the operator’s ability to estimate the collectives’ future state, such as “Will support for Target 1 decrease?” An overall SA value, $SA_{O}$, represented the percent of correctly answered SA probes out of 12 total. The SA probe accuracy descriptive statistics are shown in Table 3 [13]. Operators using the Collective visualization had higher SA probe accuracy, regardless of the SA level. The Mann-Whitney-Wilcoxon tests (n = 56, degrees of freedom (DOF) = 1) found highly significant effects between visualizations for $SA_{O}$ (U = 702, $\rho$ $<$ 0.001) and $SA_{1}$ (U = 714.5, $\rho$ $<$ 0.001). Moderately significant effects were found for $SA_{2}$ (U = 572.5, $\rho$ $<$ 0.01) and $SA_{3}$ (U = 554, $\rho$ $<$ 0.01).

Local and global clutter percentages were analyzed for each SA probe question. Clutter is defined as the area occupied by objects on a display, relative to the total area of the display [58]. Presenting too much information in close proximity to one another will require the operator to search longer for information [58] and can negatively influence the accuracy of the SA probe questions. Area coverage for each 2-D item was calculated by the number of pixels the item covered on the computer visualization. The conversion between meters and pixels was different for each visualization due to differences in the display monitor size and software program. One meter for the IA visualization was approximately 1.97 pixels per meter and the Collective visualization was approximately 2.3 pixels per meter. The local clutter percentage variable was the percentage of area obstructed by items that were displayed within the 500 $m$ (i.e., approximately 254 pixels for the IA visualization and 218 pixels for the Collective visualization) circular radius from the center of the collective, or target of interest in the SA probe question. Collective IV, for example, is the collective of interest in the following SA probe question: “What is the highest value target available to Collective IV?” The items obstructing the 500 $m$ radius when using the IA visualization, in Figure 2, for the previous SA probe question are: the Collective IV, Targets 9 and 12-15, and 200 individual entities. Some SA probe questions encompassed more than one collective or target of interest, which required calculating the local clutter percentage for each collective or target and summing the values together. All calculations first required converting meters into pixels in order to ensure equivalent units. The Collective visualization computer display size was unknown; therefore, local and global clutter percentage calculations use the corresponding item and computer display dimensions from the IA visualization. Local clutter was calculated using Equation 1:

where LHA represents the area corresponding to the number of collective hubs (2464 $pixels^{2}$ per hub) inside the 500 $m$ radius. The area corresponding to the number of highlighted targets (2350 $pixels^{2}$ per highlighted target), which have outlines and are in range of the currently selected collective are represented as LHTA, while the targets that are not highlighted (1720 $pixels^{2}$ per target) are denoted as LTA. LAICE represents the area corresponding to the number of individual collective entities (64 $pixels^{2}$ per agent) inside of the 500 $m$ radius, and was excluded from the Collective visualization local clutter percentage calculation, because no individual entities were displayed. The individual collective entities were confined to the 500 $m$ search radius around their respective collective hub; therefore, the calculation assumes that the 200 entities associated with each collective are inside of the local radius. The area corresponding to the number of target information pop-up windows (32922 $pixels^{2}$ per target information pop-up window) is represented as LTIW, and the corresponding collective information pop-up windows (25740 $pixels^{2}$ per collective information pop-up window) are represented as LCIW. Only the target or collective information pop-up windows that belong to targets or collectives inside of the 500 $m$ radius are considered.

The local clutter percentage descriptive statistics 15 seconds before asking, while asking, and during a SA probe response are provided in Table 4. The maximum local clutter percentage was 177%, which indicated that the area covered by the associated items of the collective or target of interest in the SA probe exceeded the 500 $m$ radius. Local clutter percentages larger than 100% were attributed to the area covered by the collective and target information pop-up windows. The location of the information pop-up windows were not recorded; therefore, the maximum area coverage was considered when information pop-up windows did not occlude items in the environment. The maximum area coverage condition was not reflective of the real trial environment, where information pop-up windows covered items on the central map. The IA visualization had lower local clutter percentage, regardless of when the metric was collected for $SA_{O}$, $SA_{1}$, and $SA_{3}$; although, no significant effects were found between visualizations. The Spearman correlation analysis revealed no correlations were found between local clutter percentage and SA probe accuracy.

The global clutter percentage, calculated using Equation 2, was the percentage of area obstructed by all objects displayed on the entire IA computer display (2073600 $pixels^{2}$), since the Collective computer display was unknown.

where ICA represents the area of the static interface components (493414 $pixels^{2}$), which encompass the program bar, the Microsoft Windows program bar, the select trial button, the collective request area, and the monitor area. GHA represents the area covered by Collectives I-IV (9856 $pixels^{2}$), which were visible throughout the duration of a trial. The area corresponding to the number of highlighted targets (2350 $pixels^{2}$ per highlighted target), which have outlines and are in range of the currently selected collective are represented as GHTA. Remaining targets that are not highlighted (1720 $pixels^{2}$ per target), are represented as GTA. GAICE represents the area encompassed by 800 individual collective entities (51200 $pixels^{2}$), which was only considered for the IA visualization. The area corresponding to the number of target information pop-up windows (32922 $pixels^{2}$ per target information pop-up window) is represented as GTIW and the corresponding collective information pop-up windows is represented as GCIW (25740 $pixels^{2}$ per collective information pop-up window).

The global clutter percentage descriptive statistics 15 seconds before asking, while asking, and during response to a SA probe question are shown in Table 5. The IA visualization had a lower global clutter percentage, regardless of when the metric was assessed across all SA levels. The Mann-Whitney-Wilcoxon tests found highly significant effects between visualizations when responding to a SA probe question (n = 670, DOF = 1) for $SA_{O}$ (U = 64442, $\rho$ $<$ 0.001). Moderate significant effects were found for $SA_{O}$ 15 seconds before asking a SA probe (U = 64188, $\rho$ $<$ 0.01) and while asking a SA probe question (U = 63728, $\rho$ $<$ 0.01). Significant effects were found 15 seconds before asking a SA probe question for $SA_{1}$ (n = 294, DOF = 1, U = 12487, $\rho$ = 0.02) and $SA_{3}$ (n = 152, DOF = 1, U = 3445.5, $\rho$ = 0.03); while asking a SA probe question for $SA_{1}$ (n = 294, DOF = 1, U = 12301, $\rho$ = 0.03) and $SA_{3}$ (n = 152, DOF = 1, U = 3452, $\rho$ = 0.05); and during the response to a SA probe question for $SA_{1}$ (n = 294, DOF = 1, U = 12216, $\rho$ = 0.04). The Spearman correlation analysis revealed a weak correlation for the Collective visualization for $SA_{1}$ between global clutter percentage 15 seconds before asking a SA probe question and SA probe accuracy (r = 0.16, $\rho$ = 0.05).

The Mental Rotations Assessment [57], which assessed the operator’s spatial reasoning, identified no significant effects between visualizations. A Spearman correlation analysis revealed weak correlations between the Mental Rotations Assessment and SA probe accuracy when using the IA visualization for $SA_{O}$ (r = 0.17, $\rho$ $<$ 0.01), $SA_{1}$ (r = 0.18, $\rho$ = 0.03), and $SA_{2}$ (r = 0.27, $\rho$ $<$ 0.01). The Mann-Whitney-Wilcoxon tests identified no significant effects between visualizations for the weekly hours spent using a desktop or laptop and video game proficiency. Weak correlations were found between weekly hours using a desktop or laptop and SA probe accuracy for the IA visualization for $SA_{O}$ (r = 0.12, $\rho$ = 0.04) and $SA_{1}$ (r = 0.21, $\rho$ = 0.01), as well as when using the Collective visualization for $SA_{2}$ (r = 0.21, $\rho$ = 0.02). No correlations were found between video game proficiency and SA probe accuracy.

The NASA Task Load Index (NASA-TLX) assessed the six workload subscales and the weighted overall workload [60]. The descriptive statistics for the NASA-TLX demands imposed on the operator are presented in Table 6. The Collective visualization imposed a lower overall workload, had lower physical and temporal demands, and caused less frustration. The IA visualization imposed a lower mental demand, which had a significant effect between visualizations (n = 56, DOF = 1, U = 515, $\rho$ = 0.04) and less effort. The IA visualization had higher performance with a highly significant effect between visualizations (n = 56, DOF = 1, U = 159.5, $\rho$ $<$ 0.001).

The 3-D Situational Awareness Rating Technique (SART) measured the operator’s perceived situational understanding, demand on attentional resources, and supply of attentional resources [61]. An overall score was calculated using the standard calculation. The SART descriptive statistics are presented in Table 7 [13]. The minimum SART score was -1, which was unexpected as a negative score requires the supply of attentional resources to exceed the demand on attentional resources and a low perceived situational understanding. Both of these conditions are highly unlikely. The Collective visualization had a higher overall score, more situational understanding, high demands of attentional resources (although nearly the same as the IA visualization), and more supply of attentional resources, compared to the IA visualization. The Mann-Whitney-Wilcoxon test indicated moderately significant effects between visualizations for the overall score (n = 56, DOF = 1, U = 560, $\rho$ $<$ 0.01), situational understanding (n = 56, DOF = 1, U = 561, $\rho$ $<$ 0.01), and supply of attentional resources (n = 56, DOF = 1, U = 561, $\rho$ $<$ 0.01).

The analysis of how visualization influenced operators suggests that the Collective visualization promoted better transparency. The variables that directly supported $H_{1}$ are the SA probe accuracy and SART. $H_{1}$ was supported, because operators using the Collective visualization had significantly higher objective and subjective SA and lower overall workload. Transparency embedded into the Collective visualization, via color-coded icons (i.e., target value) and outlines, state information identified on the collective icon, information provided in the collective and target information pop-up windows, and feedback provided in the Collective Assignments and System Messages areas, promoted better perception, comprehension, and projection of future operator actions, making the overall human-collective team more effective. The Collective visualization operators; however, had more local and global clutter, which was mainly attributed to the number of collective and target information pop-up windows that were visible. The increased clutter has both positive and negative implications for transparency. From a system design perspective, clutter is not ideal if operators are unable to perform their tasks effectively. The Collective operators, in this evaluation, who had more global clutter were able to answer more SA probe questions accurately, which suggests that operators were not hindered by the clutter and were able to perform better than their counterparts. The dependence on having the collective and target information pop-up windows visible suggests that the collective state information provided on the collective icon was not as effective as the information pop-up window and there is a need to provide support information on the target icons. Other design strategies must be investigated to improve the efficacy of the collective and target icons for the Collective visualization. Further analyses are also required to determine what contributed to more mental demand, more effort, and less perceived performance using the Collective visualization and whether the additional stress may have been experienced due to positive aspects, such as operators being highly motivated to complete their tasks. The Collective visualization may have required more effort on behalf of the operator in order to understand what the collective was doing compared to the IA visualization that showed the dynamic behavior emerging. Collective visualization operators may have additionally been distracted by the secondary SA probe question task and required more time to enter back into the loop to determine the collective behaviors.

The Mental Rotation Assessment and video-game proficiency supported $H_{2}$, since operators with different individual capabilities did not perform significantly different using the Collective visualization. One exception to the hypotheses was that more experienced operators may have performed better because of their extensive use of computers, which may have led to faster and more accurate interpretations of information [58] (i.e., different types of iconography), or easier access to the supplemental information. Since the exception was observed in both evaluations, the behavior is inherent to working with a computer interface, rather than a particular visualization. Using an abstract collective visualization will mitigate the need for particular operator capabilities to perform the sequential best-of-n decision-making task. Operators using the Collective visualization experienced less frustration, which supports $H_{3}$. Dissatisfaction (i.e., frustration) transpires when the system is not transparent and prohibits the operator from understanding what is currently happening, or there is too much clutter and the interface appears noisy [62]. The abstract visualization may be a solution to mitigate dissatisfaction.

The operator had access to supplementary information that was not continually displayed, such as different colored target borders that identified which targets were in range and had been abandoned, or information pop-up windows that provided collective state and target support information, in order to aid comprehension ($SA_{2}$) of collective behavior and inform particular actions. The results of SA probe accuracy, which is the percentage of correctly answered SA probes questions used to assess the operator’s SA during a trial, identified that operators using the Collective visualization had higher SA probe accuracy, regardless of the SA level. Further details regarding the statistical tests were provided in the Metrics and Results Section 5.1 of $R_{1}$.

Collective left-clicks identified targets that were in range of a collective (i.e., white borders indicated that the individual collective entities were investigating the target, while yellow indicated no investigation), whether the targets had been abandoned (i.e., red borders), and was the first click required to issue a command. The number of collective left-clicks descriptive statistics 15 seconds before asking, while asking, and during response to a SA probe question are shown in Table 9. Operators using the IA visualization had fewer collective left-clicks, regardless of when the metric was assessed for all SA levels, except for $SA_{O}$ during response to a SA probe question. The Mann-Whitney-Wilcoxon tests found highly significant effects between visualizations for $SA_{O}$ (n = 664, DOF = 1) 15 seconds before asking a SA probe question (U = 64213, $\rho$ $<$ 0.001), while asking a SA probe question (U = 67670, $\rho$ $<$ 0.001), and during response to a SA probe question (U = 64710, $\rho$ $<$ 0.001). A highly significant effect was also found when responding to a SA probe question for $SA_{2}$ (n = 223, DOF = 1, U = 8317, $\rho$ $<$ 0.001). Moderate significant effects were found for $SA_{1}$ (n = 290, DOF = 1) 15 seconds before asking a SA probe question (U = 12534, $\rho$ $<$ 0.01), while asking a SA probe question (U = 12043, $\rho$ $<$ 0.01), and during response to a SA probe question (U = 12414, $\rho$ $<$ 0.01). An additional moderate significant effect was found while asking a SA probe question for $SA_{3}$ (n = 151, DOF = 1, U = 3472, $\rho$ $<$ 0.01). Significant effects were found 15 seconds before asking a SA probe question for $SA_{2}$ (n = 223, DOF = 1, U = 7210.5, $\rho$ = 0.04) and during response to a SA probe question for $SA_{3}$ (n = 151, DOF = 1, U = 3489, $\rho$ = 0.01). No correlations were found between the number of collective left-clicks and SA probe accuracy.

Target right-clicks allowed the operator to open or close target information pop-up windows, which provided the percentage of support each collective had for a respective target. Operators may have used the support information to justify issuing commands. The number of target right-clicks descriptive statistics 15 seconds before asking, while asking, and during response to a SA probe question are presented in Table 10. The Collective visualization had fewer target right-clicks for all SA levels, 15 seconds before asking and during response to a SA probe question, while the IA visualization had fewer while asking a SA probe question. The Mann-Whitney-Wilcoxon test found no significant effects between visualizations for the number of target right-clicks. The Spearman correlation analysis revealed weak correlations between the number of target right-clicks and SA probe accuracy for the IA visualization 15 seconds before asking a SA probe question for $SA_{O}$ (r = 0.17, $\rho$ $<$ 0.01) and for $SA_{2}$ (r = 0.37, $\rho$ $<$ 0.001).

The abandon command was provided to operators who desired a collective to discontinue investigating a particular target. Ideally lower valued targets were abandoned, since the objective was to aid each collective in selecting and moving to the highest valued target two sequential times. The percentage of times the highest value target was abandoned per participant is shown in Table 11. Operators using the IA visualization abandoned the highest value target less frequently, but no significant effects were found between the visualizations.

The percentage of times an abandoned target information pop-up window was open per participant was evaluated and is represented in Table 12. The operator may have used the support information in order to justify abandoning a particular target. Participants using the IA visualization had fewer abandoned target information pop-up windows open, compared to the Collective visualization operators. No significant effects were found between visualizations.

The SART results, which measured the operator’s perceived situational understanding, demand on attentional resources, and supply of attentional resources [61], were ranked higher for the Collective visualization compared to the IA. The statistical test details were provided in Section 5.1.

The analysis of how visualization promoted operator comprehension identified advantages and disadvantages associated with both visualizations. The Collective visualization promoted higher comprehension ($SA_{2}$) and situational awareness; however, because the Collective operators abandoned the highest value target more frequently, $H_{4}$ was not supported. Mistaking the roman numeral identifiers with the integer identifiers may have caused IA operator confusion and contributed to lower $SA_{2}$. Ensuring that identifiers are unique and distinct, such as integers versus letters, may mitigate misunderstanding. The target value for the Collective visualization may not have been salient enough to distinguish it from other potential targets. Further investigations are required to determine if the target value must use the entire collective hub icon area, similar to the IA visualization, in order to be recognizable, and to establish what levels of obscurity are needed to ensure that target values are distinguishable from one another.

The use of target borders (collective left-clicks), information pop-up windows (target right-clicks), and target value, were assessed to determine if operators used these types of information to accurately justify actions. Collective right-clicks, which opened and closed collective information pop-up windows, 15 seconds before asking, while asking, and during response to a SA probe question, were not analyzed in this manuscript, because the collective evaluation did not indicate which collective was selected via the right-click. None of the metrics supported $H_{5}$, because the information provided by the Collective visualization did not justify accurate actions. Collective left-clicks did not support or hinder SA probe accuracy for either visualization, but fewer target right-clicks 15 seconds before asking a SA probe question supported higher $SA_{O}$ and $SA_{2}$ probe accuracy for operators using the IA visualization. The operators may have learned to anticipate when SA probe questions were going to be asked and took preventative actions, by opening or closing target information windows, which resulted in higher SA probe accuracy. The use of target information pop-up windows aided Collective visualization users to abandon targets more than 30% of the time. Further analysis using technology, such as an eye-tracker, may provide more accurate metrics to determine operator comprehension during SA probe questions by identifying exactly where an operator is focusing their attention.

Many system characteristics were available to the operators in order to aid task completion. The IA visualization had both lower local clutter percentage, which was the percentage of area obstructed by items displayed within the 500 $m$ circular radius of a collective, or target, and global clutter percentage, which was the percentage of area obstructed by all objects displayed on the visualization. Operators using the IA visualization had fewer collective and target information pop-up windows open throughout the trial. The statistical test details were provided in Section 5.1.

The Euclidean distance (pixels) between the focus of the SA probe question and where the operator was interacting with the visualization indicated where operators focused their attention, because no eye-tracker was used. Euclidean distance can be used to assess the effectiveness of the object placements on the display. Larger distances are not ideal, because more time [63] and effort is required to locate and interact with the object. The first requirement of calculating the Euclidean distance was to determine what the collective, or target of interest was in a SA probe question. For example, Target 3 is the target of interest for the following question: “What collectives are investigating Target 3?” The second requirement was to determine where the operator was interacting with the system. The Euclidean distance between Target 3 and the operator’s current interaction (i.e., click), which was Collective IV, is identified by a dashed orange line in Figure 4. The Euclidean distance between SA probe interest and clicks descriptive statistics 15 seconds before asking, while asking, and during response to a SA probe question are presented in Table 14. Operators using the IA visualization had shorter Euclidean distances between the SA probe interest and their clicks with the visualization, regardless of when the metric was assessed for all SA levels. The Mann-Whitney-Wilcoxon tests found a moderate significant effect between visualizations while asking a SA probe question for $SA_{O}$ (n = 464, DOF = 1, U = 31052, $\rho$ $<$ 0.01). Highly significant effects were found between visualizations 15 seconds before asking a SA probe question for $SA_{O}$ (n = 557, DOF = 1, U = 43303, $\rho$ = 0.02) and for $SA_{1}$ (n = 273, DOF = 1, U = 10577, $\rho$ = 0.05), while asking a SA probe question for $SA_{1}$ (n = 229, DOF = 1, U = 7645, $\rho$ = 0.01), and during response to a SA probe for $SA_{O}$ (n = 499, DOF = 1, U = 35029, $\rho$ = 0.02). The Spearman correlation analysis revealed a weak correlation between the Euclidean distance between the SA probe’s focus interest and the clicks and SA probe accuracy for the IA visualization 15 seconds before asking a SA probe question for $SA_{1}$ (r = -0.18, $\rho$ = 0.04).

The sum Euclidean distance (pixels) between clicks was the sum of all distances between the operator’s current interaction and the immediately previous interaction. For example, if an operator interacted with the visualization four times while a SA probe question was being asked, the sum Euclidean distance is the sum between interactions one and two, interactions two and three, and interactions three and four. The sum of Euclidean distance between clicks descriptive statistics 15 seconds before asking, while asking, and during response to a SA probe question are presented in Table 15. Operators using the Collective visualization had smaller sums of Euclidean distance between their interactions, regardless of when the metric was assessed for all SA levels, with two exceptions. The IA visualization had a smaller sum for $SA_{1}$ while asking and during response to a SA probe question. The Mann-Whitney-Wilcoxon test found no significant effects between visualizations. The Spearman correlation analysis revealed weak correlations between the sum of Euclidean distance between clicks and SA probe accuracy for the IA visualization 15 seconds before asking a SA probe question for $SA_{2}$ (r = 0.2, $\rho$ = 0.04) and during response to a SA probe question for $SA_{O}$ (r = 0.14, $\rho$ = 0.02). Weak correlations were also revealed for the Collective visualization while asking a SA probe question for $SA_{O}$ (r = -0.13, $\rho$ = 0.05) and for $SA_{1}$ (r = -0.2, $\rho$ = 0.05).

Collective and target left- and right-clicks were examined per participant. Target left-clicks were the second click required in the process of issuing commands, but did not provide supplementary information. The number of collective and target left- and right-clicks descriptive statistics are presented in Table 16. Operators using the IA visualization had fewer collective and target left-clicks, while those using the Collective visualization had fewer collective and target right-clicks. The Mann-Whitney-Wilcoxon test identified no significant effects between visualizations.

Metrics showing how operators used the abandon command were assessed. IA operators had lower percentages of times the highest value target was abandoned and lower percentages of times an abandoned target information pop-up window was open per participant. The statistical analyses of both metrics were provided in Section 6.1. Instances may have occurred when the operator accidentally issued an undesired abandon command or repeatedly issued the abandon command, although the command only needed to be issued once; hence, the percent of times abandon commands exceeded abandoned targets was examined and the descriptive statistics are shown in Table 17. Operators using the IA visualization had fewer repeated abandon commands for all decision difficulties, compared to those using the Collective visualization. The Mann-Whitney-Wilcoxon test found no significant effects between visualizations.

The time difference (minutes) between the committed state and issued decide command assessed the operator’s ability to predict the collective’s future state changing from the committed state (30% support for a target) to executing (50% support for a target). The time difference between the committed state and when an operator issued a decide command descriptive statistics are shown in Table 18. Operators using the Collective visualization had smaller time differences between the committed state and issued decide commands for overall and easy decisions; however, operators using the IA visualization had smaller time differences for hard decisions. The Mann-Whitney-Wilcoxon test found no significant effects between visualizations.

The analysis of which visualization promoted better usability was inconclusive, because both visualizations had advantages and disadvantages. The Collective visualization operators were able to predict and issue decide commands faster, after the collective was in the committed state, compared to those using the IA visualization. $H_{6}$ was not supported; however, because the Collective visualization operators abandoned the highest value target more frequently, and there was a higher percentage of abandon commands exceeding abandoned targets. Operators using the Collective visualization had more local and global clutter, which suggests that Collective operators relied on the information pop-up windows to answer the SA probe questions more than IA operators. Sixteen of twenty-four SA probe questions relied on information provided in the information-pop up windows. The collective and target icons, as well as the target outlines, were intended to aid Collective operators to answer the SA probe questions correctly; however, the operators needed to use the information pop-up windows in order to see the numeric values for the collective support and collective behavior and answer questions regarding target support from a specific collective, or multiple collectives. An example question, such as “What collectives are investigating Target 3?” in Figure 3, will require using the target information pop-up window, because Target 3 is in range of Collective I and III. A target information pop-up window is not required for Target 1, in Figure 3, since it is only in range of Collective I. The need to use information pop-up windows contributed to the Collective visualization clutter. The operators may have preferred the numeric value representations versus the other visualization techniques, which may have contributed to their reliance on the information pop-up windows. The IA operators may have had an advantage, by deducing the same information as the Collective operators gained from the information pop-up windows, by observing the dynamic behavior of the individual collective entities. Relying on supplemental information pop-up windows is not ideal and suggests that improvements must be made to the collective icon to ensure the collective’s state information is more understandable. Modifications, such as indicating which collective was the highest supporting collective on the target icon, instead of solely showing that there was support through the use of color and opacity, may increase the reliance on the target icon instead of the target information pop-up window. Additional experimental design modifications can ensure a more even distribution of questions that may rely on other information providing features on the visualization, such as the icons, system messages, or collective assignments versus information pop-up windows. Operators using target information pop-up windows to verifying that a target was abandoned by a collective, may have been confused if the reported target support was greater than zero. The operators may have reissued additional abandon commands in an attempt to reduce the collective support to zero, although only one abandon command was needed. There were instances during the trial when a few individual collective entities became lost, when the collective hub was transitioning to a new location, and never moved with the hub. The lost entities may have continued to explore a now abandoned target, because they never received the message to abandon the target, which occurred inside of the hub. Strategies, such as reporting zero percent support when an abandon command is issued, may help mitigate erroneous repeated abandon command behavior, which was experienced up to 25% for some operators. Operators using the IA visualization may have also experienced confusion if they saw individual collective entities still travelling to an abandoned target. Not displaying lost entities after a specific period of time once a collective hub has moved to a new location may also reduce the number of reissued abandon commands.

$SA_{1}$ probe questions that inquired about objects nearby were answered more accurately than those that were further away when using the IA visualization. Asking SA probe questions about objects at various distances from the operator’s current focal point is necessary in order to understand how clutter, or moving individual collective entities, may affect the operator’s ability to identify the object of interest and answer the question correctly. Smaller sum of Euclidean distances between interactions, suggests Collective operators may have had fewer interactions. Further analysis is required to determine whether more interactions were needed for operators to answer SA probe questions correctly and make more successful and faster decisions. $H_{7}$ was not supported by the analysis. The IA operators may have issued fewer commands, or did not rely on target borders as much as operators using the Collective visualization. Issuing more commands suggests that Collective operators may have wanted more control over the decision-making task, which may have occurred due to lower trust, or misunderstanding collective behavior. Further investigations are needed in order to understand how the algorithm and control mechanisms may interact with the visualization of collectives to influence operator behavior. The effectiveness of the system design will be dependent on all system characteristics working together to promote optimal human-collective performance and to achieve high effectiveness.

The length of time it took the human-collective team to reach a decision, decision time (minutes), was examined. Consensus decision-making algorithms are inherently slow, which is undesirable in realistic use scenarios. Adding a human operator into the loop permits the human to influence the decision and has the potential to minimize decision time. The decision time descriptive statistics per decision are shown in Table 20 [13]. Operators using the Collective visualization had faster decision times for overall, easy, and hard decisions. Both visualizations had faster human-collective decision times compared to the simulation. The Collective visualization simulation had slightly faster decision times for overall and easy decisions, while the IA visualization simulation had faster decision times for hard decisions. The Mann-Whitney-Wilcoxon test found significant effects between visualizations with human operators for overall (n = 672, DOF = 1, U = 50921, $\rho$ = 0.03), easy (n = 375, DOF = 1, U = 15452, $\rho$ = 0.04), and hard decisions (n = 297, DOF = 1, U = 9521, $\rho$ = 0.04). Highly significant effects were found between human operators and simulation for the IA visualization overall (n = 672, DOF = 1, U = 74005, $\rho$ $<$ 0.001), easy (n = 481, DOF = 1, U = 37414, $\rho$ $<$ 0.001), and hard decisions (n = 384, DOF = 1, U = 23194, $\rho$ $<$ 0.001) and for the Collective visualization overall (n = 672, DOF = 1, U = 79468, $\rho$ $<$ 0.001), easy (n = 461, DOF = 1, U = 38786, $\rho$ $<$ 0.001), and hard decisions (n = 392, DOF = 1, U = 26887, $\rho$ $<$ 0.001).

The selection success rate was the number of correct decisions (the collective moved to the highest valued target) relative to the total number of decisions. Selection success rate descriptive statistics per decision are shown in Table 21 [13]. Collective operators had higher selection success rates for all decision difficulties. The human-collective teams had higher selection success rates for both visualizations for all decision difficulties compared to the simulation. The Collective simulation had higher selection success rates for overall and hard decisions, while the IA simulation had higher selection success rates for easy decisions. The Mann-Whitney-Wilcoxon test found highly significant effects between visualizations with human operators for overall (n = 672, DOF = 1, U = 64008, $\rho$ $<$ 0.001) and easy decisions (n = 375, DOF = 1, U = 19845, $\rho$ $<$ 0.001). A moderate significant effect significant effects between visualizations with human operators was found for hard decisions (n = 297, DOF = 1, U = 12761, $\rho$ $<$ 0.01). Highly significant effects were found between human operators and simulation for the IA visualization overall (n = 672, DOF = 1, U = 34650, $\rho$ $<$ 0.001), easy (n = 481, DOF = 1, U = 18242, $\rho$ $<$ 0.001), and hard decisions (n = 384, DOF = 1, U = 13088, $\rho$ $<$ 0.001) and for the Collective visualization overall (n = 672, DOF = 1, U = 22162, $\rho$ $<$ 0.001), easy (n = 461, DOF = 1, U = 10795, $\rho$ $<$ 0.001), and hard decisions (n = 392, DOF = 1, U = 9449.5, $\rho$ $<$ 0.001).

The Spearman correlation analysis revealed a moderate correlation between human-collective team decision time and selection success rate using the IA visualization for easy decisions (r = -0.42, $\rho$ $<$ 0.001). Weak correlations were revealed between the human-collective team decision time and selection success rate using the IA visualization for overall decisions (r = -0.27, $\rho$ $<$ 0.001) and when using the Collective visualization for overall (r = -0.11, $\rho$ = 0.05), easy (r = -0.18, $\rho$ = 0.02), and hard decisions (r = 0.18, $\rho$ = 0.03). Moderate correlations were revealed between simulation decision time and selection success rate using the IA visualization for overall decisions (r = -0.53, $\rho$ $<$ 0.001) and when using the Collective visualization for overall (r = -0.57, $\rho$ $<$ 0.001) and easy decisions (r = -0.44, $\rho$ $<$ 0.001). Weak correlations were revealed between simulation decision time and selection success rate using the IA visualization for easy (r = -0.35, $\rho$ $<$ 0.001) and hard decisions (r = -0.14, $\rho$ = 0.03).

SA probe accuracy, which is the percentage of correctly answered SA probes questions used to assess the operator’s SA during a trial, results identified that Collective operators had higher SA probe accuracy, regardless of the SA level. Further details about the statistical tests were provided in Section 5.1.

Additional Spearman correlation analyses were conducted to see if any correlations were identified between the weekly hours that participants’ used a desktop or laptop, video game proficiency, the mental rotations assessment, and selection success rate. A weak correlation was found between weekly hours participants’ used a desktop or laptop and selection success rate for the IA visualization (r = 0.16, $\rho$ = 0.02).

The post-trial questionnaire assessed the participants’ understanding of the collective behavior, never (1) to always (7), and their ability to chose the best target for each decision, never (1) to always (7). The post-trial performance and understanding subjective ranking descriptive statistics are presented in Table 22 [14]. Performance and understanding rankings were higher for operators using the Collective visualization. The Mann-Whitney-Wilcoxon test found a significant effect between visualizations for understanding (n = 56, DOF = 1, U = 513, $\rho$ = 0.04).

The analysis suggests that the Collective visualization promoted better human-collective performance. $H_{8}$ was supported, because the Collective visualization produced higher objective primary and secondary task performance, as well as higher subjective performance. Operators using the Collective visualization had significantly faster decision times and higher selection success rates compared to the IA visualization. Realistic human-collective user scenarios will require high performance in short decision times, especially in proposed high-risk environments. Longer decision times contributed to higher success rates for both visualizations for overall decisions and for easy decisions for the IA visualization; however, faster decision times contributed to higher selection success for hard decisions using the Collective visualization. The design of an effective human-collective system must enable the human-collective team to fulfill primary objectives, without hindering other metrics, such as decision time. Devoting more time to ensure high task performance is a common trade-off behavior observed in teams. Expedited decisions may have occurred if higher valued targets were further away from other objects (less clutter), making them more salient, or if impatient operators influenced collective behaviors more to make decisions faster. The Collective visualization enabled operators with different capabilities to perform relatively the same, unlike the IA visualization, which found that individuals with more weekly desktop or laptop exposure had higher selection success rates.