Beyond Screens: Supporting Co-located Augmented Reality Experiences with Smart Home Devices

Recent research has shown how co-located AR experiences can enable people to have fun together, but most consist of only digital visual augmentations using phone-based AR (Dagan et al., 2022). Additionally, prior work has shown that AR is well-suited for co-located experiences because it can be grounded in the environment (Wetzel et al., 2008), for example by using physical objects as enablers, or inputs, of the experience (Dagan et al., 2022). However, these augmentations are constrained to pixels on a screen rather than augmenting the physical space itself or using physical objects as outputs in the experience. The recent increase of IoT devices in people’s homes (Koskela and Väänänen-Vainio-Mattila, 2004) presents an opportunity to use smart home devices to extend the augmentation of the physical world for co-located experiences.

Leveraging this opportunity, we explore using IoT devices, in particular a smart light, as a core aspect of an experience and novel form of interaction. We present Spooky Spirits, a mobile AR game¹¹1Implemented as a Snapchat Lens using Lens Studio https://ar.snap.com/ built for two users to play together in person along with a smart light ²²2Used Kasa Smart Lightbulbs https://www.kasasmart.com/us/products/smart-lighting. Our system allows users to easily connect their smart lights to the mobile experience via a web interface.

We evaluated our system with 10 pairs of participants by observing how they use the experience and interviewing them to learn more about the immersive value and impact of augmenting a shared physical space using an IoT device. We hope to motivate and inform future designs of mobile AR and IoT co-located experiences.

The Spooky Spirits experience piggybacks off of existing spirit summoning and fortune-telling games such as Ouija boards³³3Ouija board game:https://en.wikipedia.org/wiki/Ouijaand Magic 8-Balls⁴⁴4Magic 8-Ball game: https://en.wikipedia.org/wiki/Magic_8_Ball . This narrative embraces the magic and spookiness of a smart light changing colors without manual input. To set up the experience users must 1) have a smart light in their space 2) connect their light to the mobile game using a website, and 3) find a partner to play with in person.

Device arrangement is a key design consideration for co-located experiences (Isbister et al., 2018)(Lundgren et al., 2015). Our experience involves two users, a phone running the game, and a smart light. When playing Spooky Spirits, one user, player 1, holds the phone with the game running and guides their partner, player 2, through the playful “summoning ritual” by communicating the instructions they receive via the mobile UI. When the experience first begins, the game creates the spooky ambiance by turning the smart light blue. Player 1 then directs their partner to stand in front of the camera and position their body in a specific gesture, for example, a T Pose, which is detected using Full Body Triggers ⁵⁵5https://docs.snap.com/lens-studio/references/templates/object/full-body-triggers (see Fig. 1 - left). After the body gesture is detected, the light turns white providing visual feedback for both users indicating that the “spirits” are listening and users are prompted to ask a yes-no question about the future aloud (see Fig. 1 - Middle). For example, users might ask ”Will I win the lottery?”.

To make the light a salient part of the experience we chose to use it not only as an enhancement to ambiance but also as a source of information in the narrative of the experience. After users ask a yes-no question aloud, the light will turn green or red representing the answer as yes or no respectively (see Fig. 1 - Right), making the experience dependent on the information from the state of the light. In this design, the smart light acts as the sole source of immersion and augmentation for player 2. This allows us to explore if the experience is immersive even without mobile visual augmentation for all players.

This design encourages users to work together and leverages the physicality of users’ space by augmenting their environment beyond pixels on a screen. This provides an opportunity to observe the impact and immersive value that the smart home device brings to the experience since only one user can see the UI and AR on the mobile device, while the other user can only see the effects of the light in their environment. This experience extends augmentation beyond mobile for both users.

Our system involves a web app⁶⁶6https://spookyspirits.letsplayirl.com/ we built for users to connect to their smart lights (see Fig. 2). We chose to use Kasa Smart Light Bulbs as they have a public API, are easy for non-technical users to set up, and are relatively inexpensive. As set up for our experience, users use our web interface to log in to their existing Kasa account, select the bulb they would like to use, and obtain a 5-digit randomly generated code that is stored in DynamoDB. Next, users launch the game⁷⁷7They launched by scanning a code in Snapchat and enter the 5-digit code to connect the game to their smart light. By using an external interface for login we hoped to simplify the connection process and reduce friction within the mobile experience. The details of the system consist of AWS Amplify running a Next.js app with 1) a server-side component that handles API calls to get and set the settings of the smart light and 2) a client side component that displays the website to the user (see Fig. 3). The Next.js app also polls from a DynamoDB database periodically to check for changes for each device, represented by the generated 5-digit code, and if there are changes, it sends the new data to the Kasa Cloud using the Kasa public API, which modifies the state of the physical devices. The Snapchat Lens will make get and set calls to the AWS Remote API using the entered 5-digit code to change the state of the device.

We performed a pilot study with 10 pairs of participants playing with the Spooky Spirits experience together with a smart light and conducted semi-structured interviews.

We aimed to explore the impact of a smart home device as a core aspect of a co-located AR experience to help inform future designs of similar experiences. Here are six main observations

1. (1)

   Augmentation of the physical world beyond mobile superseded the setup effort. Users noted that the setup friction was high due to the smart light, however, they felt that the light added to the overall immersive value.

2. (2)

   Augmentation via IoT devices enabled immersion beyond mobile and overcame asymmetric device access. Although one user did not experience mobile visual augmentation, generally they still felt immersed and involved in the experience because of the augmentations from the smart light.

3. (3)

   Different setup decisions led to new interactions. We observed users who placed the bulb in a ceiling socket try to reach up towards the light and pay direct attention to the bulb itself, whereas users who placed the bulb in a desk or standing lamp did not try to touch the physical bulb and instead paid attention to the effect the bulb had on overall lighting.

4. (4)

   The state of the physical environment affected its augmentation. Users who played in dark lit rooms felt the experience was more immersive than those who played in rooms with more secondary light because the changes in the color and brightness of the light were more salient.

5. (5)

   IoT devices could have played a more important role in giving feedback. Users sometimes struggled to know what to expect from the smart light or when to wait for the light to change.

6. (6)

   IoT devices were able to serve multiple roles, from creating ambiance to providing information. During the game, blue light was used to create spooky ambiance, and green or red light encoded information (yes or no answers to users’ questions) that users easily decoded.

In the future, we hope to build and study more co-located AR experiences that use a variety of IoT devices as novel forms of augmentation and interactions. We hope this inspires future designers to continue to design more mobile AR and IoT co-located experiences.