VirtualNexus: Enhancing 360-Degree Video AR/VR Collaboration with Environment Cutouts and Virtual Replicas

Given the respective merits of 360$\degree$ video and 3D reconstructions, prior work has explored combining the two in remote AR/VR collaboration. Teo et al. and Gao et al. proposed toggling between the modes of using 3D reconstruction or 360$\degree$ video  (Teo et al., 2019b; Gao et al., 2021). However, the need to switch between two different media prevents simultaneously harnessing the merit of both. The authors also reported that frequently switching between perspectives and interactive modalities offered by different modes is challenging to adjust to. Young et al. extended this work, providing seamless transitions based on distance between users instead  (Young et al., 2020). Teo et al. also proposed follow-up works (Teo et al., 2020, 2019a) that can insert 360$\degree$ panorama as bubbles into 3D reconstructions. However, the 3D reconstruction in the proposed system has a static texture and is mostly used as context. Although users may update the 3D reconstruction’s context with newly captured 360$\degree$ images, they rely mostly on the live 360$\degree$ video mode (Teo et al., 2020) or live 360$\degree$ insertion (Teo et al., 2019a) for real-time interaction. In our work, both content delivered through 360$\degree$ and 3D reconstruction are live. We simultaneously provide a live 360$\degree$ environment and live environment cutouts (spatial mesh textured with live video texture). We additionally provide enhanced interactivity with virtual objects and replicas. Thus, we now review common interactive requirements in AR/VR remote collaboration and how they apply to 360$\degree$ video.

To enhance the presence of the remote guest and the effectiveness of AR/VR remote collaboration, prior research has explored a variety of interactive modalities, and we review them as follows.

It is challenging to provide remote users access to the physical environment they are telepresent in. Recent research has taken mechanical and robotic approaches, allowing remote users to move physical objects in the local user’s space with mini-robots (Ihara et al., 2023) or deformable interfaces (Follmer et al., 2013; Leithinger et al., 2014). However, such methods usually have a limited area of operation (e.g., a delegated platform like a desk) and introduce additional hardware overhead. An alternative approach is to provide indirect physical access through virtual replicas (Oda et al., 2015; Elvezio et al., 2017). However, most prior work requires virtual replicas to be created in advance with CAD tools (Zhang et al., 2022; Oda et al., 2015; Elvezio et al., 2017; Wang et al., 2023, 2021) or only support creating from 2D contents or sketches (Huang and Xiao, 2024; Hu et al., 2023; He et al., 2020). While depth-based methods such as Kinect-Fusion (Izadi et al., 2011) can quickly reconstruct an object or a scene, more recently, Neural Radiance Fields (NeRF) (Deng et al., 2022; Barron et al., 2021; Mildenhall et al., 2021; Müller et al., 2022) allow object and scene reconstruction with high quality. Notably, Instant-NGP (Müller et al., 2022) drastically reduces the training time of NeRF, making it feasible to reconstruct individual objects within seconds or minutes. In our work, we incorporate virtual replica creation with Instant-NGP into our collaborative telepresence system.

360$\degree$ VR telepresence allows a user to explore a remote space omnidirectionally with immersion. However, regular 360$\degree$ videos lack spatial information to allow users to virtually interact with physics and collision (e.g., draw annotations on a wall, and bounce a virtual object on a desk), thus reducing the sense of being physically present. To solve this, we propose to align a spatial reconstruction with the 360$\degree$ Video. While we render virtual objects with the 360$\degree$ video, they behave like reacting with an actual physical environment when users manipulate them. The aligned spatial reconstruction should be transparent to preserve the higher reality of the 360$\degree$ video. As most state-of-the-art AR headsets maintain a spatial map behind the scene, the process of creating and aligning a 3D reconstruction should be seamless and hidden from the users.

In 360$\degree$ telepresence, with a regular stationary 360$\degree$ camera setup, users can only rely on far-hand manipulation (e.g., dragging an object with a long ray pointer) to access faraway regions in the scene, precluding precise interactions. Therefore, we introduce the concept of environment cutouts, allowing the remote VR user to create a “slice” of the 360$\degree$ environment that can be interacted with at a different scale or position. For example, the VR user can select a part of the real world to make a copy, optionally scale it down (similar to a miniature diorama), pull it closer, and interact with this cutout (for example, placing virtual objects on this smaller world) while any such interactions are also reflected on the original world location. While the remote VR user can use the ray pointer to access farther objects, the ability to pull an environment cutout closer allows users to harness near interactions (e.g., grab, near draw) that have a higher precision. To convey the intention of the VR user to the AR user, the AR person will see the VR user’s avatar moving toward the physical counterpart of the cutout as they pull an environment cutout (e.g., the VR person pulling a whiteboard closer is rendered as them moving toward it).

In immersive environments, virtual replicas are useful props for referring to objects, conveying ideas, and prototyping rapidly (Elvezio et al., 2017; Zhang et al., 2022). VirtualNexus enables ad-hoc creation of 3D virtual replicas in remote AR/VR collaboration. The AR user can conveniently set an object on a platform, scan around it, and obtain a shared virtual replica. VirtualNexus additionally stores the scanned virtual replica, enabling a “scan once, create many” experience.

Co-location in a spatially aligned and synchronized environment is fundamental for remote AR/VR collaboration systems. Therefore, VirtualNexus offers synchronized ray pointers, annotations, and shared virtual objects, which are essential elements to maintain group awareness (Buxton, 2009; Gutwin and Greenberg, 2002). For coherence, these features also adapt to the aforementioned system design: 1) annotations and virtual objects are able to collide and physically interact with the hidden spatial reconstructions, and 2) the environment cutout maintains a cloned copy of annotations and virtual objects that are synced with the original 360$\degree$ environment and the AR physical environment.

We implemented VirtualNexus (Fig. 2) using Unity 2021.3.20f1, which can be configured as either a VR or AR application. VirtualNexus uses Microsoft HoloLens 2¹¹1https://www.microsoft.com/en-us/hololens for AR and Meta Oculus Quest 2²²2https://www.meta.com/ca/quest/products/quest-2/ for VR. An Insta360 X3³³3https://www.insta360.com/product/insta360-x3 360$\degree$ camera omnidirectionally streams the local user’s environment at 5.6K resolution and 30fps to the remote VR side. For efficient 360$\degree$ video streaming, we re-implemented a foveated video compression pipeline introduced by prior work (Huang et al., 2023) on a desktop machine with an Intel Core i7-9700K 3.6GHz CPU, 32GB RAM, and an NVIDIA GeForce RTX 2060. QR codes on the front and back of 360$\degree$ camera’s tripod serve as the spatial anchors, aligning the VR world’s origin with the 360$\degree$ camera’s lenses. The local AR user sees a virtual avatar overlaid on the 360$\degree$ camera with synchronized head and hand poses of the remote VR user. Finally, VirtualNexus runs virtual replica processing and VR-side rendering on the same machine, which has an Intel Core i9-12900KF 3.2GHz CPU, 64GB memory, and an NVIDIA GeForce RTX 4090 GPU. The AR user can scan a physical object and send the resulting photos to the virtual replica creation server. The server pre-processes the images, reconstructs a virtual replica with Instant-NGP, and sends it back to both AR and VR as shared virtual objects.

VirtualNexus embeds a spatially aligned 3D reconstruction of the physical environment with the 360$\degree$ video, laying out the basis for spatial interaction with physicality. To achieve this, we utilized the spatial meshes created and maintained by Microsoft HoloLens 2, which is the foundation of a mixed-reality experience. As HoloLens creates or updates a spatial mesh, VirtualNexus extracts the vertices and triangles from the spatial meshes and transforms them into the VR world’s coordinates. VirtualNexus then sends the mesh information through a TCP connection to the remote VR side and reconstructs the spatial meshes in real-time. To accurately align the 360$\degree$ video with the reconstructed spatial mesh, we reverse-engineered the 360$\degree$ camera’s intrinsic parameters and projected the 360$\degree$ video to the skybox with equidistant fisheye mapping⁴⁴4https://docs.opencv.org/3.4/db/d58/group__calib3d__fisheye.html. We show the alignment between the spatial meshes and the 360$\degree$ video in Fig. 3. We implement spatial mesh synchronization in a silent thread to keep it seamless for both AR and VR users.

In virtual and augmented reality, virtual contents are rendered binocularly to generate a depth cue. However, as most 360$\degree$ videos are monocular, users can only tell depth using their empirical knowledge of objects’ sizes (i.e., closer objects look bigger and farther objects are smaller). We started with overlaying binocularly rendered objects in front of a monocular 360$\degree$ video. However, we found that this causes an inconsistency regarding depth perception: a virtual object looks closer than a physical object in the 360$\degree$ video even if they are placed in the same position. To mitigate this, in the VR build, we shifted the position of the right-eye camera leftwards by the VR headset’s inter-pupillary distance (IPD), causing the VR headset to effectively render in monocular mode, thus rendering virtual objects as if they belonged to the 360$\degree$ video. Such an adaptation may lead to difficulty in perceiving depth during object manipulation. In the future, we can opt to use binocular 360$\degree$ cameras (already available as commodity products) creating 360$\degree$ videos with binocular depth perception.

As mentioned in 4.1, we aligned the AR and VR space using the QR Codes attached to the 360$\degree$ camera as the spatial anchor. To facilitate synchronized remote collaboration, we implemented synchronized ray pointers, annotations, and virtual objects.

To define an environment cutout, the VR user first makes a selection of 4 points with their ray pointer cast onto the depth mesh (Fig. 4a). These raycast points and the camera position define a selection frustum. We then select the triangles of the spatial mesh that lie in the selection frustum, which form new mesh objects that define the cutout. While users can create both 2D (Fig. 4b) and 3D cutouts (Fig. 4c), the latter may be subject to occlusions.

The cutout supports standard VR manipulations, such as grabbing, rotating, and scaling. Users can “select” a cutout to make it active, which causes the VR user’s actions to be performed relative to the cutout, and causes their avatar to be rendered in AR relative to the cutout’s physical counterpart (e.g. as shown in 7(b)). With no cutout, or if the cutout is deselected, the VR user’s interactions will occur with respect to the 360$\degree$ video, and they will be rendered at the location of the 360$\degree$ camera (as in Fig. 7d).

We sync interactions across this copied cutout and the world-space 360$\degree$ video. When the VR user creates a virtual object (annotation or mesh) while a cutout exists, they see two objects — one corresponding to the world space (the “original object”), and one relative to the cutout (“copy object”), for example, in Fig. 4b. Movements and edits are synchronized between the original, copy, and the virtual objects displayed to the AR user, but the copy itself is only visible to the VR user when using a cutout.

VirtualNexus allows the AR user to create shared virtual replicas from physical objects in the environment. We chose Neural Radiance Fields (NeRF) to create virtual replicas from a futuristic standpoint: NeRFs have shown promise in producing photorealistic scans of objects and scenes with high-quality lighting and texture, and we feel that future object scanning pipelines may involve such technologies for fidelity, rather than traditional pipelines like KinectFusion (Izadi et al., 2011). However, for compatibility with existing mesh-rendering pipelines, we produce both a NeRF model and traditional vertex-coloured mesh from our object scanning pipeline.

To the best of our knowledge, our system is the first to adopt NeRF reconstruction for remote AR/VR collaboration. Among the variants of NeRF scene reconstruction techniques, Instant-NGP (Müller et al., 2022) strikes a balance between training time and quality. In our preliminary exploration, reconstructing individual objects with Instant-NGP (i.e., as opposed to an entire scene) with sufficient quality only takes 1–3 minutes on our NVIDIA GeForce RTX 4090 GPU machine. As we require a user to walk around the targeted object, our pipeline is best suitable for smaller desktop objects (e.g., small appliances, toys, hand-held tools). VirtualNexus’ end-to-end virtual replica creation pipeline has a server-client architecture (see bottom left of Fig. 2). The server is implemented in Python and incorporates Instant-NGP’s Python API ⁶⁶6https://github.com/NVlabs/Instant-NGP. Examples of intermediate results at each stage of the pipeline can be seen in Fig. 5.

Collective prototyping is a key application domain for collaborative mixed reality (Huang and Xiao, 2024; Nebeling et al., 2020). VirtualNexus facilitates such real-time content authoring through the creation, sharing, and manipulation of virtual objects and annotations for remote users. Furthermore, the scanning and creation of instant replicas allow both users to integrate shapes beyond basic primitives, bringing in virtual objects that mimic real physical items. The cutout feature provides additional options for the remote user to interact and prototype, allowing increased precision by bringing further areas closer.

For example, in Fig. 6, we use VirtualNexus to create a collaborative virtual scene on a desk that the local user can walk around and view from different angles. In this scene, basic primitives such as cubes and spheres form the environment, and virtual replicas are used to create more detailed characters. Annotations are used to define areas in the environment (i.e. a path). The domain of content creation and prototyping also forms the basis of our user study (Section 6), which involves collaboratively building a storyboard.

Remote mixed reality systems also apply to remote education and instruction (Wu et al., 2013; Kamińska et al., 2019; Sharma et al., 2013). Online learning platforms have become increasingly important in an increasingly digital world, and such platforms facilitate communication between educators and students despite distances (Cowit and Barker, 2023). We demonstrate a virtual classroom environment in which a teacher, in a classroom, can call upon a student, who may be joining remotely, to answer a question on a whiteboard (Fig. 7a and 7b). The teacher first annotates a question on the whiteboard. The student might find the whiteboard to be too far to interact with precisely. In real life, the student may walk up to the whiteboard. VirtualNexus allows the student to achieve the same by bringing the whiteboard closer using its environment cutout. The student can then annotate their answer on this closer cutout, which is then reflected on the original whiteboard.

To extend this education scenario, the teacher might ask students to mirror their interactions with virtual objects in a virtual hands-on lesson (Fig. 7c and 7d). To illustrate, we outline an arts-and-craft exercise in which the teacher is teaching about modelling and colouring while the student follows along. The teacher’s desk has equipment and objects found in the classroom (i.e. markers); the student can replicate it using virtual objects. However, some required objects for the task may not be physically present for the student (i.e. the model pig). Thus, the teacher can scan the object locally and create a replica for the remote student. The student can then use these virtual objects to replicate the teacher’s instructions.

Mixed reality mediums are often used in games and other recreational domains to encourage exercise and socialization. Using VirtualNexus, we can develop collaborative recreational activities that use virtual objects for remote users. We illustrate an example using a bowling game situated on a virtual alley overlaid on the observed local environment (Fig. 8). Users can create virtual bowling pins and lay them at the end of the virtual alley. Then, either user can create a ball, which can be rolled at the pins. By taking turns rolling the balls, the users can experience a fun virtual bowling session situated in a physical environment.

We recruited 14 participants (8 females, 5 males, averaged 24.8 years old. 1 participant reported N/A for both demographic questions) through convenience sampling, forming 7 dyads. All participants had some prior experience with remote collaborative tools (e.g. Google Docs) and video communication tools (e.g. Zoom). Almost all participants had some experience with using VR in the past (13 out of the 14 participants); experience with AR headsets was rarer (8 out of the 14 participants). Our study was reviewed and approved by the institutional ethical board, and all participants reviewed and signed a consent form prior to the study.

Upon arrival, the participant dyad was split into a local AR user and a remote VR user. Each participant underwent an individual short guided tutorial regarding interactions using VirtualNexus features. After the participants familiarized themselves with the system, they worked together to create a virtual storyboard for a researcher-provided narrative. This collaborative task took users through all features of the system - participants discussed and ideated the storyboard through cutouts and annotations, and then created it using virtual scanned objects. We instructed the participants to scan and create the first physical object they decided to use in the task, allowing them to experience the full virtual replica creation process. In the interest of time, we provide pre-scanned models for additional objects users may need afterwards. We continued the task until the users had finished creating enough scenes, or when we reached the allotted time. After completion, the researchers wrapped up the study through a final questionnaire. It involved 5-point Likert-scale questions that related to immersion, presence, ease of use, and task performance using the VirtualNexus system, as well as an optional open-ended field for each question (Fig. 9 and 10). The entire study took approximately 90 minutes, and participants were reimbursed $24 CAD.

Participants generally were positive towards VirtualNexus (Fig. 9 and 10). Fig. 1(d) shows an example storyboard created in the study. We present the findings based on their responses to our questionnaires, especially those relevant to the two novel interactive techniques. A1-A7 and V1-V7 refer to the AR and VR users respectively.

Past research has studied the trade-off between using 360$\degree$ videos and 3D reconstruction for telepresence (Teo et al., 2019b, a, 2020). In 360$\degree$ videos, despite the higher visual quality, remote VR users cannot move in the shared world without involving locomotive equipment such as robots (Jones et al., 2021; Heshmat et al., 2018). In contrast, telepresence systems with 3D reconstructions (Teo et al., 2019a, 2020) allows users to walk around the remote environment, but the reconstructed scene suffers from holes and artifacts due to imperfect scanning and occlusion.

Teo et al. (Teo et al., 2019b) set out to balance this trade-off between 360$\degree$ videos and 3D reconstructions in telepresence. This work allows the remote guest to switch between the 360$\degree$ video and the point-cloud reconstruction of the same physical environment. However, to utilize the merits of both, the user needs to frequently context switch between two distinct sets of interactive modalities, potentially increasing the mental workload. VirtualNexus takes a slightly different approach: while the remote user always sees the physical environment in a high-quality 360$\degree$ video, we align a transparent 3D reconstruction with the 360$\degree$ video to enhance the interactivity. We believe such a design provides a more coherent interactive experience and better physical presence (Q1V from the questionnaire): In our study, we observed the remote users naturally annotating and placing virtual artifacts on remote physical surfaces, like they are physically in the remote environment wearing an AR headset.

In the direction of improving the interactivity of 360$\degree$ video telepresence, Rhee et al. (Rhee et al., 2020) incorporated virtual annotations and artifacts. However, the main drawbacks of 360$\degree$ telepresence remain: The inability to walk around and interact with the remote environment.

In co-located collaboration, users can freely walk around and utilize their surroundings. Such freedom is limited for remote users telepresent with 360$\degree$ videos. Inspired by the concept of WiM(Danyluk et al., 2021), VirtualNexus implements the environment cutouts, which does the opposite by bringing parts of the environment toward the remote user. In our study, we found the cutouts also improved the clarity and precision: For further away areas, the remote user can pull environment cutouts closer for more precise annotation.

VirtualNexus additionally provides the remote user with better access to individual physical objects with ad-hoc creation of virtual replicas. These objects mimic reality, but take advantage of digital affordances — they can be replicated, scaled down or up, and can have different physics properties. Echoing past research on using virtual replicas for remote collaboration and instruction (Oda et al., 2015; Elvezio et al., 2017; Zhang et al., 2022; Huang and Xiao, 2024), participants in our study found virtual replicas enhance interpretability. In contrast, representing objects as primitives adds an interpretation layer and hinders efficiency. Currently VirtualNexus takes 1–3 mins to create a unique virtual replica. Usability research by Nielsen (Nielsen, 1994) suggests that a response time over 10 seconds risks losing users’ attention, but can be alleviated by providing a progress bar. As we managed the virtual replica creation in a separate thread behind the scenes, we expect the users can work on other sub-tasks in parallel. For example, we observed some AR participants proceed to sketch on the whiteboard with the remote VR user. Future work can reduce the processing time for object reconstruction by replacing Colmap (Schönberger et al., 2016; Schönberger and Frahm, 2016) with refined HoloLens camera poses.

Our study surfaced two future improvements for VirtualNexus: 1) the local AR user’s awareness of the remote VR user, and 2) context switching between the cutout space and the original environment.

AR users have more freedom to move and a smaller FoV, so they have a reduced awareness of the VR user. Such awareness is further reduced when the VR user relocates to the area they cut out from the environment. Therefore, it is reasonable to smooth out the VR user’s relocation (e.g., by animating their motion) when they activate/deactivate the cutout. We can also provide additional cues to the local AR user when the VR user tries to communicate (Piumsomboon et al., 2019, 2017).

Presently, the cutout space arbitrarily overlays the original world. The remote user needs to context switch as they redirect their focus from the cutout space to the environment and vice versa, making it hard to understand whether an object belongs to the cutout space or the original world. Future work can localize the cutout using a “snow globe” metaphor — the cutout acts as a localized miniature world: Objects outside this space are hidden, and the objects within would be more easily understood as belonging to the cutout.