Mixed Reality: The Interface of the Future

Mixed reality[3] is the technology that fuses the physical with the virtual world to provide upcoming evolution in human, computer, and environmental interactions. It’s the next wave in computation. Before it, we had mainframes, PCs, and smartphones. Now, the concept of AR, VR, and MR are emerging which is mainstream for consumers and businesses. It liberates people from screen-bound experiences through interactions with data in the physical world. Mobile AR[11] nowadays offers mixed reality solutions, especially on social media. Generally, the applications to mixed reality also include environmental understanding i.e. anchors and spatial mapping, and human understanding i.e. eye-tracking, hand-tracking, and speech input, positioning, and locations in both physical and virtual spaces.

To create true mixed reality interactions and experiences, there are three essential elements:

• •

  Input methods

• •

  Perceptions from the surrounding

• •

  Cloud-driven computer processing model

The reality-virtuality continuum[1] as shown in Fig. 2 represents the spectrum of realities as the mixed reality spectrum. At one end, we have the real physical system in which humans exist. On the other, we have the relative digital reality.

Nowadays, many big tech companies are doing research on AR and VR. Snapchat uses AR features on image/video filters. However, most mobile devices and apps that exist on the market have little environmental perception capabilities. They cannot fully blend physical and digital realities. Augmented Reality[11] is achieved by overlaying holograms, graphics, images, and video streams in the physical world. Virtual reality has one difference in visualizing the real world. In VR, we cannot see the actual world which makes the experiences fully immersive digital experience. Mixed Reality can be experienced by transitioning between AR and VR which places a hologram in the physical world as if it is physically present. It represents the avatar digitally and personally present in the physical world in collaboration with people and objects.

Mixed Reality perception is achieved by wearing a glass-mounted headset from which we can see the physical world to interact using digital assets/objects. The user can manipulate virtual objects to change the environment and interact using different sensors. For instance, a user can resize an object to fit into a room. There are several mixed reality devices available in the market in which hardware devices, chips, sensors, and software are integrated to perform a task in a coordinated way. MR devices can also act as standalone device, can be usable with glasses, and allows users to interact with their physical world. One limitation of MR headsets is it is still in the early stage of development which limits their production. Additionally, its field of view is often narrow. These advantages and limitations help to explore more possibilities in applications and challenges across Mixed Reality. Table I shows the different mixed reality devices being developed[12]. Fig 4. represents the decreasing price of VR over the years.

The integration of AR and MR technologies in mobile devices has opened up a lot of opportunities for applications in smart education. It overcomes the problems in the traditional way of teaching by showing its benefits. According to a recent study[7], the usage of mixed reality enhances students to learn complicated logical problems in Chemistry like a synthetic bond. They mentioned that mixed reality technologies help to picture the basic ideas of covalent bond to identify the objectives of chemical bonds. In the same way, a virtual learning environment builds interaction and cooperation among understudies and teachers. In the normal way of teaching, it’s much more complicated to visualize higher-dimension data. Teachers can use Mixed Reality devices to simulate the mathematical analysis and graphs (higher dimensions) and teach their students so that it is easier for them to understand.

Pridhvi et al.[7] conducted the experiment to show the effectiveness of MR in the education system. They included different application development using Unity 3D and Unreal Engine to accomplish a practical look for learners/students. They visualized the Roman Colosseum in VR and also a low-resolution model of a maze. They also learned about the behavior of animals according to their adaptation to environments. S.V Viraktamath[14] and Touel et al.[13] their research mentions that MR devices can be used as a real-time analytical and interactive tool for instructors to promote the student’s learning process. They concluded that the environment resulted in better prospects for education compared to the existing teaching methodology.

The study was conducted at the Norwegian University of Science and Technology and Gerup et al.[15] reveals that Mixed Reality is suitable for use in healthcare sectors because of its 3D capabilities and interactions, with scalable objects interacting simultaneously by various participants, and its ability to provide a deeper and better understanding of complex issues. In the same way, because of COVID-19 and social distancing, this technology has accelerated a lot.

Similarly, Sánchez-Margallo JA[16] performed a study of the application of MR in medical training and surgical planning that focuses on minimally invasive surgery. They mentioned MR will reduce the time and effort while performing the training and surgery as medical training is a long and tiresome process. They conducted experiments on training in urology that allow visualization of 3D models, perform manipulation in different angles and dimensions, and able to interact independently with them. They found MR is a significant tool for studying and visualizing the human pelvic system and anatomy both individually and in groups.

Construction projects contain a series of interdependent and interrelated activities. There are many hazardous works that could lead to serious accidents for the workers. Fei Dai et al. (2018) evaluated the feasibility of applying an emerging MR technology to enhance communication in the construction industry[17]. They developed a holographic application in Microsoft Hololens that helps in real-time collaboration in 3D space where the application was evaluated through trial and feedback from workers/participants in the construction industry. The experiment was conducted among 50 workers where they can communicate without being physically present at the site. Their experiments demonstrate that 58 percent of total responses agree MR has a positive impact on risk communication 34 percent are neutral. This helps to get a sense of how MR could be huge potential for applications in the construction industry.

Similarly, MR can be used for engineering project planning to execution. For instance, MR can overlay certain elements onto a building plan so that investors/stakeholders can get a better understanding of the project. MR can be used to show 3D models, interact with them, provide tours, and track and document the progress of the project. In addition to this, they can be used for labeling and smooth collaboration, construction training, etc. These things seem to create a lot of potential in the construction industry using MR.

Games and Entertainment are one of the major industries where Mixed Reality has been impacted the most. MR games use collaborative technologies to entertain, educate, motivate, and inspire people. The design of MR games and interactive gameplay environments has received a lot of attention across multiple disciplines. Elizabeth Bonsignore et al.[18] has mentioned that one example of gamification is the growth of context-aware apps like Foursquare which uses social and location-based services. Other games include iSpy and Parallel Kingdom which uses location-based services that offer a playful lens for visualizing experiences, and relationships of ideas grounded in real spaces i.e. crossing the boundary between real and virtual.

Likewise, many social media platform that includes interactive mixed-reality experiences are being developed. Roofie Du et al.[19] from Google presented Geollery, which is a social media platform to share, create, and explore geotagged information. It includes a real-time ’rendering of an interactive and collaborative mirrored world with 3D buildings, user-generated content, and geo-tagged social media. Using these features, users can view, chat, and collaborate with other participants remotely in the same spatial context in a virtual environment. This demonstrates MR features are being developed in the gaming and entertainment industries.

Mixed Reality also has applications across governmental bodies and agencies. A recent study conducted shows that AR and MR could be used for security in customs borders and protections. Similarly, they also mentioned envisioning transportation security 4.0 which reduces error and risk in security screening. In the same way, MR features can be used to study the probable natural calamities region and also have easy management to search and rescue.

MR can help to fill the lines between brick-and-mortar retail experiences and e-commerce. The products can be scanned using their tags and leverage important information from there so that people don’t need to manually fill in the information. For instance, a retail mobile app can be used to identify all the products in a store automatically and add them to the shopping cart by adding computer vision and machine learning features.

It’s very important to design a suitable architecture to run MR applications smoothly. The architecture needs to support multiple users, who can simultaneously manipulate the objects in the environment. There should be non-noticeable latency in every operation so that user interaction would be smooth. As its architecture is based on client-server, there is a single point of failure. So, the server should be up and running every time for execution. This induces the challenges to be faced by developers in system design.

Networking is a most important part of MR that acts as a communicator between server and client. By distributing IP, every client can communicate with each other and access data through the server. If there is any delay in network communication, latency could be introduced which makes user interaction less convenient. Thus, the challenges also lie in the networking part.

Data Loss and Security are one of the greatest challenges in Mixed Reality applications. The MR devices help to overlay the objects in the physical world where there is continuous data flow across the environment. The data could be lost on the way to overlaying the objects. In the meantime, anyone can interrupt the data flowing over the network which could create trouble. Moreover, the MR applications used in the government sector need to be more secure in terms of data handling and processing.

When MR is used in educational settings, it changes past pedagogical techniques. The study by Glover and Miller[20] in 2001 mentions that it is a prerequisite to accepting technology by teachers and students. Both teachers and students should be prepared for such technologies. Similarly, learning activities should include well-designed interactions with friends and the system. There should be minimum information overload and field-of-view should be accurate. Otherwise, the visualizations are perceived as obtrusive.

MR technology and devices are still in their infancy stage. Pascal Knierim et al.[21] mentioned that it includes a small view field, limited battery life, uncomfortable wearing devices for a long time, etc. From the fast technological advances across MR, these challenges could be solved in the future. Currently, many advanced features like object recognition, hand gesture recognition, and speech identification are being integrated into MR. But it’s still a research question of how to design an interactive concept for MR user experiences.

• •

  The experiments that are performed to analyze the applications of MR in different fields above show that MR has the potential to be used for solving the existing problem and a lot of benefits can be leveraged from it. It has increasing benefits across education, health, construction, gaming, etc. which could be an interface of the future.

• •

  Despite some negative impacts that MR could bear, we could see a lot of positive impacts of MR in human lives. This answers our first research question.

• •

  As seen from the increasing number of companies and MR devices year to year in Table I, a lot of big companies are spending billions to adopt MR technologies as shown in Table I. Also, the applications seen in various sectors have also shaped MR to be one of the emerging technologies. This demonstrates MR technology as the interface of the future.

• •

  Certain challenges to adopting MR technologies in various fields are studied above. In the same way, there are some uses of MR that can impact human lives negatively. These all challenges should be cautiously handled in order to fully utilize the benefits of Mixed Reality. These aspects answer that there are certain challenges to exploiting the use of Mixed Reality.