Unveiling the Era of Spatial Computing

Since the advent of the information age, user interfaces (UIs) have experienced a myriad of reforms and innovations. The evolution from Command Line Interfaces (CLIs) to Graphical User Interfaces (GUIs) has made computers increasingly user-friendly. With the rise of smartphones and tablets, touch-based interfaces have become mainstream for Internet browsing. The latest advancements are in Extended Reality (XR) technologies and spatial computing, heralding a new era of user interfaces.The road map of Use interfaces is visualized in 2.

Spatial computing allows the collection of dynamic physical reality data to construct a digital space. This space, though low in dimension, is crafted to be rich enough for human perception. It encompasses a variety of physical data, including spatial location, audio information, peripheral vision, distance metrics, eye and body movements, and even subjective judgements and attitudes on a broader scale. The resulting digital environment can take various forms and effects, contingent upon the specific concepts it serves. For instance, advances in XR technologies like Mixed Reality (MR), Augmented Reality (AR), and Virtual Reality (VR) offer differentiated levels of virtual immersion. Moreover, the latest iteration of internet-based applications, such as the metaverse, integrates social and economic systems within virtual worlds, striving for multi-technology convergence[8]. Spatial computing aims to achieve a higher consistency and a reduced sense of discord in virtual synthetic human experiences, serving as an ”external eye” for users.

Technologies encompassing all realities serve as the foundational precursor to spatial computing, demonstrating transformative impacts across various fields and heralding a promising future for spatial computing. This discussion begins with an overview of the concepts underpinning all realities, followed by an examination of their applications and the enhancements that future spatial computing technologies may afford.

X-reality encompasses at least three definitions, determined by their relationship within the mixed reality spectrum[6]. A simplified way to understand this is by referring to the mixed reality continuum represented in 1, which illustrates the degree of virtualization[7]. Adjusting a one-dimensional “fader” shifts the experience between different reality technologies like AR and VR. Moreover, by examining the extent to which reality can be augmented, we introduce the concept of mediated reality, which directly alters our perceived reality[6]. The subsequent sections detail the applications of these technologies.

AR (Augmented Reality): AR enhances human-computer interaction by superimposing computer-generated 3D content onto the real world, aiming to enrich multiple sensory modalities and thereby augment the perception of reality. AR finds direct application in visualizing geographic data [4], measuring and depicting damage in post-disaster assessments[5], and enhancing virtual travel experiences by displaying virtual models across various environments[10].

VR (Virtual Reality): VR offers a simulated experience that immerses users in a virtual environment, aiming to isolate them from real-world surroundings. This isolation facilitates interactive simulations for rapid assessment and iterative development of novel design concepts unconstrained by financial or security limitations[2]. A notable application is the European Space Agency’s use of VR in designing the Argonaut lunar lander, allowing for immersive experiences that elicit genuine and effective user feedback.

Mediated Reality: This extends beyond AR to either modify or mediate our perceived reality. It can be intentionally mediated, such as through invert-vision glasses, or unintentionally mediated, often used in treating PTSD with head-mounted displays.

Spatial computing, in essence, seeks to merge the best aspects of AR and VR while striving for the authenticity afforded by mediated reality. It operates on the principle of understanding and interacting with the physical world through digital means, fulfilling all related demands. This approach facilitates a seamless integration of digital and physical elements. Below is Table1, which compares the aforementioned concepts.

In this section ,we will shed a light on the versatile application potential of spatial computing.That we will break it into five aspects to delineate it.

Artificial IntelligenceCombining the physical environment and digital information of virtual objects requires artificial intelligence to understand the scene, identify and track objects in the virtual environment, recognize gestures, detect interactions between objects, understand and handle object occlusion, and map them with the real-time spatial layout. Artificial Intelligence can be perceived as the core technology and fundamental concept of spatial computing, as it can evolve from every progress made in computer vision tasks.

IOT The Internet of Things (IoT) is the pivotal concept of communication between different kinds of devices, including those with sensors, processing ability, and display capabilities. Spatial computing requires various sensors and cameras to collect data from the physical environment. By processing this data, spatial computing devices can also serve as IoT network center processors, helping humans better control the IoT and revolutionizing how we interact with technology.

5G and beyondThe high demand for low latency in spatial computing cannot be handled without 5G and beyond technologies. By facilitating a deeper connection between physical and digital spaces, 5G and beyond technologies enable seamless transitions between virtual reality and the real world. Only the advent of 6G technologies will bring immersive experiences to social contexts, opening up possibilities for using spatial computing in education, entertainment, and various other industries highly dependent on networks.

Cloud/Edge Computing For portability, spatial computing may rely on both cloud and edge computing as part of a distributed architecture to solve heavy computing tasks like rendering 3D models. The ability to manage and store data also plays a crucial role in spatial computing. Distributed or edge computing can save bandwidth and reduce costs when spatial computing becomes ubiquitous.

BlockchainThe security of a similar concept like the metaverse is mainly protected by blockchain technology, which also has value when spatial computing aims to provide internet digital worlds. Due to its decentralized nature, people can use spatial computing devices more privately without worrying about third-party services stealing their biometric information.

In 2023, spatial computing made significant strides with the launch of its first notable product, Apple’s Vision Pro, a headgear endowed with AR, VR, and MR capabilities[11]. Devices like the Apple Vision Pro chart user surroundings using an array of sensors and cameras, employing infrared cameras to establish a sophistical high-performance eye-tracking system[1]. Additionally, it uses a pair of high-resolution cameras for collecting reality data at a high refresh rate, enabling the rendering of dynamic spatial digital images. The integration of a LiDAR Scanner and a TrueDepth camera in the Vision Pro works in tandem to generate a composite 3D map. With these technologies, spatial computing can overlay virtual objects into the real world seamlessly, provided a spatial 3D map of the area is obtained and there is an understanding of the objects within it.

However, the Apple Vision Pro faces several challenges. The first significant hurdle is its high cost, attributed not only to the substantial investment in its development phase but also to the intrinsic expenses related to its hardware and software requirements[9]. Additionally, discomfort during extended use poses a second major challenge[3]. There are also reports indicating a slight discrepancy between the real position of the user’s fingers and their representation in digital spaces, which could prove detrimental for precision tasks. These issues underscore the fact that spatial computing still has considerable ground to cover before it can be fully embraced by the public.

The paper begins by charting the evolution of user interfaces, culminating in the conceptualization and application of spatial computing—a technology that marries the physical and digital through the perception and interaction within a constructed digital space. Various All Realities technologies such as AR, VR, and mediated reality are discussed, including their specific applications and the transition towards a more integrated experience provided by spatial computing. The enabling technologies section outlines AI, IoT, 5G, cloud and edge computing, and blockchain as essential components that drive spatial computing capabilities. A case study on Apple’s Vision Pro is presented, showcasing an example of spatial computing’s application and the challenges faced, such as cost, user comfort, and integration accuracy. The paper highlights the transformative potential of spatial computing across numerous domains, despite the nascent challenges that currently limit widespread adoption. must include your signed IEEE copyright release form when you