Investigating the impact of virtual element misalignment in collaborative Augmented Reality experiences

According to Azuma[1], Augmented Reality (AR) is defined as a technology with three characteristics: 1) it combines real and virtual, 2) it is interactive in real-time, and 3) registration of digital with real objects.

These features not only enhance individual user experience but implicitly create a framework conducive to collaboration. This framework allows users, whether in the same physical space or remotely, to interact with a mutually understood and shared augmented context, thus promoting a cohesive and synchronized working environment.

Consequently, the concept of shared perception is fundamental to AR [2]. This involves the mutual understanding of virtual content among users. For example, when multiple individuals observe a virtual object placed on an actual table, it fosters an environment where they can collectively discuss, adjust, and interact with that object [2].

Collaborative Augmented Reality (AR) presents numerous challenges and considerations for developers and users. One central concern is synchronization, where maintaining consistent alignment of virtual content across all user devices is critical to the collective experience [3]. Any lag or misalignment can significantly disrupt the collaborative interaction. Additionally, heightened spatial awareness is essential, especially in co-located AR scenarios, as participants must be cognizant of both virtual and real-world elements, as well as the actions of fellow participants, which may lead to cognitive strain in immersive environments [4].

Beyond technical and cognitive challenges, interpersonal dynamics are integral to collaborative AR experiences. Unlike single-user AR scenarios, collaborative AR involves complex interactions among multiple users, each contributing unique perspectives and reactions to the virtual environment, unpredictably influencing the overall experience and collaborative outcomes [2].

Inconsistencies in collaborative AR experiences pose a significant challenge, as revealed in various studies, ranging from technical glitches to bad user feedback [5, 6, 7]. This study seeks to address this gap by investigating the impact of virtual element misalignment in collaborative AR experiences. The research goals guiding this work are formulated as follows:

RG1. Influence of virtual element misalignment on user communication and collaboration: This study assesses the impact of virtual object misalignment on the communication dynamics among users in a shared AR setting. We explore how variations in the perception of object positions affect collaborative tasks and user interactions.

RG2. Influence of virtual element and avatar misalignment on user communication and collaboration: We investigate the function of avatars within an AR environment, focusing on their ability to guide user orientation and teamwork. Specifically, the study examines whether avatars can compensate for misalignments, reducing the necessity for precise positional synchrony during collaborative tasks.

To address these questions, we introduced two experimental settings. The first involves introducing virtual element misalignment in a cooperative task requiring sorting virtual objects by color and shape. In the second setting, the misalignment of virtual objects and avatars is gradually introduced in a different cooperative task.

Collaboration within shared immersive environments accentuates these mediums’ distinctive advantages and challenges. A body of research examining these dynamics and exploring alternative collaborative approaches [8, 9, 2] contributes significantly to understanding this domain. Billinghurst et al. [2] delineate the evolution of collaborative Augmented Reality (AR) since the early 2000s, noting initial solutions akin to traditional videoconferencing systems. For instance, Schmalstieg et al. [10] present the ”Studierstube” architecture, catering to multi-user AR for visualization, presentation, and educational purposes. This system facilitates collaborative work by offering synchronized 3D stereoscopic graphics via lightweight see-through head-mounted displays, allowing users to adjust viewpoints and data layers independently. Such features strengthen cooperative efforts, particularly among visualization experts.

Other systematic investigations, such as that by Pidel et al. [9], offer comprehensive reviews of collaborative efforts in virtual and augmented reality, elucidating specific benefits and challenges. This research sheds light on often overlooked forms of collaboration, including asynchronous methods and joint efforts integrating AR and VR. While synchronous co-located and remote AR collaborations have received ample attention, asynchronous methods remain relatively unexplored. These methods unfold over time, removing the requirement for simultaneous participation, and often involve activities such as annotating and reviewing content post-engagement [5].

The term ”co-presence” denotes users’ shared local physical presence in both the real world and the shared AR environment, serving as a pivotal concept in investigating collaboration in AR settings. Efforts have been made to develop assessment tools for co-presence, aiming to integrate virtual elements seamlessly into users’ physical surroundings [11]. Assessing this psychological ”presence” becomes vital, especially in shared virtual environments, with studies highlighting the role of Mixed Reality (MR) in shaping collaborative settings [12]. Moreover, the research underscores the significance of social presence in cooperative AR, emphasizing design insights to enhance cooperation support [13]. Different interaction techniques such as hand-tracking [14] might also impact cooperative AR experiences, but this potential influence is still understudied.

Avatars are commonly employed in AR/VR environments to represent other users’ positions and gestures within the shared space. Studies have investigated their influence on collaboration and social interaction [15], as well as optimal visual representation in augmented reality [16]. The latter study explores avatar aesthetics’ impact on Social Presence and user perceptions in AR telepresence scenarios, revealing preferences for realistic full-body avatars in remote collaborations. These theoretical underpinnings set the stage for subsequent experiments and analyses to expand our understanding of collaboration in AR environments. Notably, there is a shortage of research investigating the effects of virtual element misalignment on user experiences in AR despite their potential relevance for designers and developers. Hence, our study aims to fill this gap, recognizing the significance of such exploration for advancing AR design practices.

ATI Analysis. The average score for Affinity towards Interactive Technology (ATI) was $\textit{M}=3.82$ ($SD=1.04$). We evaluated the significance of three group variables: gender, age groups, and experience. We adjusted the significance level using the Bonferroni correction [23] to $\alpha_{1}=0.0166$ to account for multiple comparisons. However, after this adjustment, none of the comparisons reached statistical significance.
UEQ Analysis. In the context of User Experience (User Experience Questionnaire - UEQ), mean values from two combinations of experimental conditions were examined. The first comparison consists of two conditions that examine the influence of perceptual differences on communication, with a focus on the learnability of the system.:

• •

  Identical virtual objects for both users (Convergence)

• •

  One differing virtual object for both users (Divergence)

The mean for the first two conditions (Convergence & Divergence) was $\textit{M}=2.49$ (${\textit{SD}=0.62}$). In the first condition (Convergence) it was $\textit{M}=2.61$ ($\textit{SD}=0.48$) and in the second condition (Divergence), $\textit{M}=2.37$ ($\textit{SD}=0.72$). The difference was statistically significant, with a two-sided significance level of $p=.027$ (t(47) = 2.28). Additionally, an examination was conducted to determine whether gender significantly impacted user experience. For this purpose, five additional analyses were conducted separately, analyzing results for both men and women. These comparisons included Convergence and Divergence for both genders, cross-comparisons of Convergence and Divergence between men and women, and a total cross-comparison. Since the six-fold measurement repetition, the significance level was adjusted using the Bonferroni correction to $\alpha_{1}=0.0083$. However, the results of these comparisons were found to be statistically not significant. The second comparison consists of four conditions to investigate the influence of synchronicity and the use of an avatar on collaboration and orientation in an AR environment. The conditions of the second comparison yielded a mean of $\textit{M}=2.32$, $\textit{SD}=0.82$. A two-factorial analysis of variance with repeated measures was conducted to analyze the results from the individual conditions. It was found that both synchronicity ($\textit{F(1,47)}=63.20$, $p<.001$, $\eta_{\textit{p}}^{2}=.57$) and the avatar ($\textit{F(1,47)}=7.12$, $p=.010$, $\eta_{\textit{p}}^{2}=.13$) were significantly associated with the user experience regarding the learnability of the AR environment (Figure 3). Bonferroni-corrected pairwise comparisons of estimated means showed that the synchronous environment ($\textit{M}=2.678$, $\textit{SE}=.06$) was significantly rated higher than the asynchronous one ($\textit{M}=1.962$, $\textit{SE}=.11$). The assessment of the use of an avatar ($M=2.42,SE=0.09$) or its absence (${M=2.23},{SE=0.08}$) depends on the situation. The avatar was rated lower in the synchronous context, while in the asynchronous context, it was rated higher. The interaction of synchronicity and avatar was also significant ($F(1,47)=13.07,p<.001,\eta_{\textit{p}}^{2}=.22$).

FSS Analysis. When examining the flow experience using the Flow State Scale, similar to the UEQ, individual conditions were tested for their significance. Likewise, six measurements related to the participants’ gender were conducted in the first comparison of experimental conditions, but no significant results were found. In the second comparison of experimental conditions, a two-factorial analysis of variance with repeated measures was again conducted. It revealed that synchronicity (${F(1,47)=16.40},{p<.001},{\eta_{\textit{p}}^{2}=.25}$) was significantly associated with the experience of flow in the AR environment (Figure 4). However, the use of an avatar (${F(1,47)=1.17},{p=.286},{\eta_{\textit{p}}^{2}=.024}$) could not be significantly linked to the experience of flow. Bonferroni-corrected pairwise comparisons of estimated means showed that the synchronous environment ($M=2.68,SE=0.06$) was significantly rated higher than the asynchronous environment (${M=1.96},{SE=0.11}$).

IPQ Analysis. In addition to ATI, UEQ, and FSS, the IPQ (iGroup presence questionnaire (IPQ) was used to assess the sense of presence in the virtual environment. The calculated mean was M = 3.34 (SD = 0.72). Since the results here should also be tested for significance considering the three group variables (gender, age groups, and experience), the significance level of $\alpha=0.05$ was lowered to $\alpha_{1}=0.0166$ using the Bonferroni correction. The results of the comparisons with the corrected significance level were not significant.

Further on, the results of qualitative analysis have been performed. Responses to open questions were methodically grouped into clusters based on shared themes or recurring statements. Clusters were organized using a Miro board, facilitating a systematic examination of the data. Key themes and patterns were discerned from the analysis of the clusters, which are presented below.

Results for Communication Changes. Participants’ responses coalesced into clusters, highlighting the following key themes: (i) Difficulties in Task Handling: Participants frequently encountered challenges when communicating in situations where everyone did not share the same visual perspective (6.23% users); (ii) Negative Experiences/Emotions: Some participants reported experiencing negative emotions, such as frustration, confusion, or dissatisfaction, during communication under these conditions (8.34% users); (iii) Improvement in Communication: Others noted positive changes, such as enhanced clarity, precision, or seriousness (41.67% users); (iv) Misunderstandings: Several participants initially experienced misunderstandings but managed to resolve them through adjustments or clarifications (8.34% users); (v) Empathy: Empathizing with the perspectives of others was deemed crucial by some respondents for improved communication and to prevent misunderstandings (8.34% users); (vi) Conflict: Instances of conflict or disputes among participants during communication were reported in some responses (4.17% users); (vii) Minimal Change: Some participants barely noticed any significant change in their communication. The predominant theme was positive statements regarding experience and communication change (10.41% of users). The remaining responses (12.5% of users) could not be assigned to any previously mentioned clusters.

Results for Future with AR/VR. Responses were categorized into clusters, revealing the following primary themes: (i) Loss of Reality: Many participants expressed concerns about a potential loss of their sense of reality due to the extensive use of AR/VR technology (22.92% users); (ii) Communication Change/Adaptation: Participants underscored the necessity for adapting and changing their communication methods to accommodate the impacts of AR/VR technology (29.17% users); (iii) Negative Attitude: A subset of participants conveyed a negative attitude toward anticipated changes in communication, citing issues like confusion, misunderstandings, and other negative consequences (10.41% users); (iv) No Change: Some participants did not anticipate significant alterations in communication patterns resulting from the proliferation of AR/VR technology (10.41% users). The dominant theme was negative statements regarding isolation and loss of sense of belonging or reality. The remaining responses (10.41% of users) could not be assigned to any previously mentioned clusters.

Results for Ideal AR System. Participants’ responses were organized into clusters, revealing the following key themes: (i) Hardware and Software Optimization: Many participants expressed a desire for enhanced hardware and software components, including smaller, more manageable, and body-integrated devices, as well as improved screens and cameras (25.00% users); (ii) Productivity Aspect: Some participants viewed the AR system as a valuable tool for daily life and as a means of enhancing productivity (14.59% users); (iii) Gaming: Several responses highlighted the potential for using AR for gaming and entertainment purposes (10.41% users); (iv) Vacation/Foreign World: Participants also considered using an AR system to explore new places or immerse themselves in virtual vacations or fantasy worlds (25.00% users); (v) Anti-AR Movement: A minority of participants fundamentally rejected AR systems. The dominant theme was leisure, fun, and entertainment options (10.41% of users). The remaining responses (12.5% of users) could not be assigned to any previously mentioned clusters.

Observations from Comparisons. In addition to the data collected during the comparisons, observations were made regarding the behavior and communication of the participants.
Comparison 1. In the “Convergence” condition, participants displayed a positive and engaged demeanor, effectively collaborating to complete tasks. They found the task novel and exciting, leading to successful teamwork. In contrast, in the “Divergence” condition, participants’ moods underwent a notable shift, resulting in raised questions and disagreements among them. Mutual questioning was a recurring response in this scenario.
Comparison 2. Participants employed various forms of navigation to solve tasks, with reactions to avatars spanning from amusement to brief statements. Many participants perceived the avatars’ appearance as unclear and abstract, carrying limited significance. The asynchronous environment posed a challenge for all participants, leading to stuttering communication and instilling doubts about their abilities and judgments and those of their navigating partners. Criticism of navigating partners was frequently observed.

Based on the hypothesis for the first comparison (“Convergence” and “Divergence”), it was assumed that the discovery of differences would lead to increased uncertainty among participants, which, in turn, would result in enhanced communication and cross-checking. The results of the study, along with observations, confirmed this assumption. A significant increase in participant communication was observed, and interactions became more precise. Additionally, heightened scrutiny of other virtual objects was noted. This implies that in the future, divergences in shared AR experiences could be further investigated as a means to increase attention and communication.

Furthermore, it was surprising that disputes arose among participants in the “Divergence” condition. This probably happened due to misunderstandings and disagreements stemming from divergent perspectives. This result highlights the importance of proactive communication and awareness of potential conflicts in divergent shared environments. A practical implication of this is that if an AR application aims to promote different perspectives and enable divergent views, it is crucial to alert participants in advance. This can be done through clear instructions, training, or prompts within the application. By providing this advance notice, participants are made aware that divergent perspectives exist and conflicts may arise. This allows them to proactively deal with these differences, avoid misunderstandings, and ultimately enhance the effectiveness of collaboration. Moreover, since divergent multi-user AR environments are often challenging for users and require effective communication, they could be utilized in practice for team communication training.

For the second comparison, which examined the synchronicity of user orientations and the use of an avatar, the assumption was that the biggest challenges in task completion would occur in the asynchronous environment without an avatar. It was expected that including the avatar would mitigate the effects of asynchrony. According to the hypothesis, tasks in the “Sync-w/o-A” and “Sync-w-A” conditions should be completed smoothly and quickly. In the “ASync-w-A” environment, somewhat slower task completion was anticipated due to increased complexity but still faster than in the “ASync-w/o-A” environment. The results of the study and observations confirmed the hypothesis. Synchronicity was found to have a crucial impact on task completion while using the avatar showed a less significant influence.

Overall, the results of this study provide valuable insights for the design and optimization of shared AR applications.

The study’s findings reveal significant theoretical implications for communication, interaction, and collaboration in VR/AR environments. It examined how varying object perceptions influence communication dynamics, finding no significant impact on interaction. However, qualitative research highlighted the influence of participant conduct and character on collaboration. Furthermore, the study investigated avatars and positional synchrony in VR/AR environments, finding that synchronization is crucial for effective and seamless collaboration, while the use of avatars had no significant impact on interaction. These findings enhance our understanding of communication and user interactions in VR/AR research. Beyond theoretical advancements, they also offer valuable practical insights, guiding the practical application and development of future collaborative AR/VR environments, such as informing participants in advance about divergent perspectives and utilizing divergent multi-user AR environments for team communication training.