DualStream: Spatially Sharing Selves and Surroundings
using Mobile Devices and Augmented Reality

Immersive collaboration takes place in shared digital worlds via VR or by making use of each user’s local environment via AR. Interaction in space is a necessary characteristic of such immersive systems. However, questions of how best to represent one’s real self, and incorporate aspects of real-world places, still remain.

Similar to how immersive collaboration incorporates aspects of space by default, video conferencing tools rely on capturing and streaming real-world audio-visual information of remote participants in real-time. Prior research has studied the impact of screen-based spatial metaphors on the sense of space and co-presence [19]. These metaphors include viewing remote video feeds around conference tables [45], and enabling video feeds to move in and out of virtual rooms to indicate informal and formal meeting spaces [36]. While these systems offer a strong virtual frame of reference, the visual information is still limited to being viewed on fixed 2D screens, and the resulting sense of spatial interaction is inferred rather than direct. One way in which video conferencing has been made more spatial is by having displays and cameras that literally move in space. Through cameras controlled by remote users [40], robotic-arm-mounted displays that mirror head gestures [42], and telepresence robots [27, 32], studies have investigated how spatial cameras and displays can impart a greater sense of co-presence and a common shared reference frame. However, like the more advanced AR/VR collaboration systems discussed earlier, these spatial video conferencing setups are also cumbersome, expensive, and difficult to scale to more general communication. In contrast, mobile devices have brought video calling to the masses. Mobile video calls support informal “small talk” interactions, task-oriented functional conversations, as well as “show and talk” interactions where real-world objects are the focus [33]. The presentation of spatial information on fixed 2D screens, however, presents challenges around the asymmetries of control, participation, and awareness [22].

Through DualStream, we enable users to share representations of their selves and surroundings, and interact with remote collaborators in a spatial manner. Thus, DualStream replicates features provided by more elaborate volumetric setups [35, 44] and goes beyond existing mobile AR systems by focusing on the spatial sharing of both self and place-based information. A key difference between DualStream and prior systems such as Mobileportation and ARCritique [53, 52, 28] is the ability to share 3D information about oneself. DualStream offers more control over what is being shared, and enables users to freely combine 2D and 3D information depending on the context and level of detail required. DualStream also employs a fundamentally different scheme of capturing, compressing, streaming, and rendering depth information. We colorize and compress two streams of depth simultaneously (self and surroundings) and stream them using conventional video networks. The freedom to use DualStream anywhere and anytime presents possibilities not available in immersive AR/VR systems. Further, we lean into the familiarity of mobile devices and videoconferencing, by enabling users to seamlessly switch between spatial AR calls and screen-based video calls. By presenting information about self and shared places in AR, DualStream helps provide a strong frame of reference for inherently spatial media. The ability to visualize video feeds in 3D enables DualStream to simulate the notion of displays moving in space, as employed by related work on spatial video calls [42]. When taken together, DualStream enables spatial remote communication in personal spaces, truly remote assistance, and supports the creation of blended spaces for interaction.

When starting a call using DualStream, users must first scan horizontal surfaces in their local environment to help the phone track its own position and orientation. Users then touch an appropriate location on the screen to place the “anchor” object (represented as a small blue cube) for the shared room. The position and orientation of the user’s phone relative to the anchor is then streamed to any remote participants. This data helps to consistently position the shared user and environment representations.

Once users have set up the shared reference frame, they can use the buttons on the interface to begin sharing representations of themselves. These representations are placed and moved based on user movement, thereby creating remote representations that look and move the way we do. The two main user representations in DualStream are (1) 3D Hologram, where the local user sees a point-cloud representation of the remote user’s face, and (2) Spatial Video, where the front-facing camera feed of the remote user’s head and shoulders is displayed in real-time, with or without the remote background (DualStream: Spatially Sharing Selves and Surroundings using Mobile Devices and Augmented RealityC). As with other video-conferencing applications, users can see their self-view in the form of a small video in the bottom-right corner of the screen. Users have complete control over which representation of themselves is shared with remote participants, and can also turn off their representation, displaying a small white cube instead. These representations enable users to feel like the person they are talking with is in the same space as they are.

Users can also share their environments in real time. Similar to the self representations, users have full control over what environment representation they share remotely. DualStream offers two options for sharing environment information (DualStream: Spatially Sharing Selves and Surroundings using Mobile Devices and Augmented RealityD):

1. 1.

   Environment Hologram: A 3D representation of the environment is projected as a point cloud from the position of the remote user, to appear to be situated in the local space. As the remote user moves around, this point cloud also moves to represent the changing surroundings.

2. 2.

   Environment Video Feed: A user shares the view of their immediate surroundings as seen on the phone screen. The local user sees a rectangular frame with this video feed attached to the remote user at a fixed distance from their self-representation, mapped to the perspective and field of view of the remote user. As the remote user moves, this rectangular frame moves around with them, functioning as a portal into the remote space. This representation is useful for sharing parts of the environment that are primarily 2D in nature or distant objects whose depth cannot be accurately captured.

DualStream consists of a basic user interface with a row of buttons in the bottom-left of the screen corresponding to the user and environment representations. By pressing these buttons, a secondary set of options appears, allowing users to switch between representations, take snapshots, activate the front-facing depth camera, and re-position the anchor object (Figure 2L).

When users launch the DualStream application, they automatically join a shared room and can begin streaming information to other remote participants. During testing, we used DualStream to support communication between up to four simultaneous users.

While mobile video conferencing helps show places to remote viewers, DualStream enables users to bring friends and family into their homes from afar, and take them along during travels near and far. For example, Elijah can call Daneel via DualStream from an art galley, and share views of a painting with Daneel at home. They can also revert to a screen-based video call at anytime, whenever the spatial component of the conversation is completed (Figure 2 A, B). Both Elijah and Daneel could be hiking outdoors, and can use DualStream to have a quick chat to share views of monuments, trees, and the creek (Figure 2 C, D). DualStream can also support larger spatial gatherings. For example, instead of conducting a daily check-in meeting over desktop video-conferencing, Elijah and their teammates could join a quick spatial call via DualStream, and appear to all be present around each other’s workspaces (Figure 2 E, F, G). Once everyone has shared their updates, the members can return to mobile or desktop-based video calls for longer discussions.

DualStream can potentially enable people to seek spatial remote assistance from anywhere given a mobile device and internet connection. For example, Elijah could ask Daneel for help in fixing the battery of their broken-down car by sharing a 3D hologram that Daneel can view and point at from their home (DualStream: Spatially Sharing Selves and Surroundings using Mobile Devices and Augmented Reality C and D). Elijah can also see Daneel spatially move around them. Similarly, Elijah could share a 3D hologram of a laser cutter that they are setting up, to give a remote walk-through to Daneel who will work on the machine soon (Figure 2 H, I). The option to capture and stream the environment video feed enables DualStream to function outdoors as well, and share video information of distant objects that depth cameras cannot capture. For example, Elijah can share 3D close-up views of the soil, and 2D views of the tree when seeking gardening advice from Daneel (Figure 2 J, K).

Unlike existing work in which environment sharing is determined by the system, DualStream gives users agency to construct their own blended, shared place. This enables users to combine and overlap remote environments. For example, Elijah and Daneel can share 2D snapshots of objects in each others’ rooms such as paintings and a TV, while also sharing a 3D hologram of a piece of furniture they plan to swap. Elijah can then see the remote couch overlaid on the real couch in their own space (Figure 2 O, P). This agency enables users to capture not just space, but moments of time in space. Elijah can take a snapshot of a painting, move it to another wall, and take a second snapshot. In Daneel’s space, there now exist two copies of the painting, which they can use to advise Elijah on which arrangement looks better (Figure 2 M, N).

To evaluate DualStream in a controlled environment, we put out an open call for participants in our university community. Of those who responded, we excluded members who were already aware of the prototype through demonstrations of earlier versions, and thus we recruited five in-person participants. We also invited five participants from outside our city to use DualStream from their locations and engage in a remote call with a researcher. These participants were known to the researchers, but were unaware of this project and had not used the prototype before. We used the remote evaluation as a means of validating DualStream in real contexts where such a prototype is likely to be used. Details about the participants, their familiarity with AR/VR technology, and usage of mobile remote communication tools can be found in Table 1.

The session began with an introductory discussion where the participant and researcher were in the same physical lab space. The participants answered questions about their usage of remote communication tools and experience with AR/VR technologies. The participants were then directed to a separate lab space, where the researcher helped them start using DualStream by placing the anchor object and briefing them about the various features of the application. The researcher then returned to the original lab space and joined the call. Both lab spaces were controlled and had a variety of 2D and 3D media to enable a rich range of possible shared content (Figure 3 Top). Both the participant and the researcher had a mobile phone with an external front-facing depth camera, enabling the bi-directional sharing of self and surroundings. Once in the DualStream call, the researcher demonstrated the various user representations, environment-sharing capabilities, and interactions. The participants were then asked to try the same features. This demonstration took between 15 and 20 minutes. The participant and researcher returned to the first lab space and engaged in a semi-structured interview regarding the participant’s experience with the prototype, seeking feedback for future development. We recorded audio and took notes during this interview, which lasted between 15 and 30 minutes.

The DualStream application was first sent to each participant, along with instructions on how to install it on their phones. The researcher and participant then joined a common video call where the researcher conducted a similar initial interview and briefing as the in-lab sessions. The researcher walked the remote participant through the process of starting DualStream and placing the anchor object. After this, the researcher joined the shared experience using DualStream running on a mobile phone with an attached depth camera. The participants did not have an external depth camera and were thus unable to share self-representations. The rest of the experience was functionally identical to the in-lab study, with the concluding interview taking place over the video call. Figure 3 (bottom) shows examples of the screen view of remote participants during the study session (included here with permission). Despite the large distances between study sites, the DualStream application was able to function smoothly without much latency. The screen views also demonstrate that the quality of rendering at remote sites was comparable to what was achieved in controlled settings.

After completing all sessions, two members of the research team qualitatively analyzed the notes and interview transcripts. After independently coding the data into various categories, the researchers discussed their findings and compiled a final set of observations based on overarching themes. These themes relate to the perception of users and environments, interactions in shared spaces, areas for improvement, and potential contexts of future use.

In developing DualStream, we were primarily concerned (like many contemporary projects in the aftermath of the COVID-19 pandemic) with adding to the experience of “being there” [5]. This objective naturally led to incorporating more realistic spatial representations of each other. In doing so, we uncovered interactions that empower users to not only share, but blend their surroundings in ways that go “beyond” in-person interaction [21]. These interactions have analogues in the space of immersive collaboration. Projects have discussed ways to merge [20, 48], duplicate [54], and remix [29] immersive environments. Some of these ideas also emerged from participant reactions to the DualStream prototype. Participants wondered if shared environments could be scaled, edited, and invited to merge in ways that would foster personal connections. Future work can explore how such concepts and ideas from immersive AR/VR can be adapted in mobile contexts. Mobile Spatial Communication is also uniquely positioned to act as a bridge modality for cross-reality systems. Applications such as Mozilla Hubs already allow users to join shared virtual spaces using personal computers and AR/VR headsets. By incorporating features from DualStream into these systems, mobile devices can help users to better navigate the transition between the desktop computing paradigms of today, and the immersive computing environments of the near-future.

Ongoing work in the areas of machine learning and computer vision can potentially help improve the fidelity and completeness of representations created by DualStream. Systems such as Pose-on-the-Go [1] already leverage mobile sensors to predict human pose and gestures. Combining this full-body information with real-time facial capture would be a step towards more complete user representations. Recent work on creating photo-realistic avatars using smartphone cameras [7] demonstrates how mobile devices might soon be capable of computing and rendering high-quality faces for applications in AR/VR communication. Integrating methods for 3D reconstruction such as Neural Radiance Fields is another promising direction, with steps taken towards mobile device-based capture and processing of high-quality environment renderings [41, 37]. While these approaches generate realistic faces and spaces, there needs to be a balance between information that is streamed as-is (like DualStream) or computed, based on the context at hand.

DualStream is a proof-of-concept of capabilities that are likely to be available on mobile devices in the near future. Throughout the design and development process, we made decisions that prioritized wider access, such as choosing cloud-based networking services to reach a broader audience outside the lab. There are potentially other combinations of present-day hardware (using iOS devices with depth cameras, or local networking setups) that might yield more optimized systems for in-lab studies. While the quality of 3D representations can be improved, testing DualStream at this stage revealed how even 2D information, when presented spatially in AR, might suffice for specific tasks. Our formative evaluation provides initial insights into how users interact with DualStream and the contexts in which they might want to use it. The study helped us understand which of the many features we should prioritize, while also uncovering some unexpected and interesting findings. The open-ended nature of the session was instrumental in achieving this. However, the lack of a concrete task makes it difficult to comment on the more immediate utility of DualStream. We intend to conduct follow-up evaluations focused on specific aspects of the prototype (e.g., user representations, pointing and gestures) in applied contexts, in order to gain a deeper understanding of how such a system can be designed to best support remote collaboration.