Avatar Fusion Karaoke: Research and development on multi-user music play VR experience in the metaverse

The importance of the simultaneity of 3D space and UGC is mentioned in the modern definition of the metaverse (Matthiu Ball)[Bal22]. The underlying motivation for this study was based on the idea that we need to create an atavistic experience in existing or future HMD devices from the avatar-driven social networking service (SNS) that is expanding on smartphones today. People have many expectations and fantasies about the metaverse; therefore, it must first be defined and scoped. A 3D-avatar social networking service, the metaverse has been the focus of much attention in recent years, with ZEPETO, REALITY, and other services growing rapidly. Assuming an inevitable future when extended reality (XR) devices, such as HMDs and augumented reality (AR) glasses are widely adopted by current smartphone users, various related topics will require research. In addition to human-computer interaction (HCI) research using current virtual reality ready devices, such as PCs and stand-alone HMDs, R&D will be needed on the avatar UX that contemporary, casual smartphone users would naturally find attractive as an extended-reality social network service (XRSNS).

The user avatar is another crucial subject. Users such as VTubers and non-fungible token (NFT) artists employ varied identities as avatars; avatars are also used as Virtual Beings in distributor culture and as icons on many platforms and social media. One trend maintains that our 3D avatars should be photorealistic, but this depends on the UX and functions of the service, the rendering performance, and the relationship (culture) between one user and another. In our research, simulating reality is not necessarily the goal, as making realistic avatars faces the challenge of the uncanny valley. Although many avatars attempt to overcome the uncanny valley, it is necessary not only to improve the quality and elements of CG rendering, such as reflection and lighting, but also to remain attentive to eye tracking and reaction speed. Research on Telexistence’s robots, such as the Telenoid developed by Ishiguro et al. (https://robots.ieee.org/robots/telenoid/), propose that it is preferable to build only abstract facial elements in a general-purpose avatar. However, when many metaverse services and XR devices are released, such discussions from a R&D perspective may be meaningless. The strongest reason for the debate between photorealistic and abstract avatars is the availability of consumer hardware, such as the Meta Quest. In the past, PCVR-only metaverse services, or legacy environments in the SecondLife era, only needed to create a comfortable UX environment; current metaverse services should focus on more generic users who will not consider the specifications. Setting up standard hardware as represented by smartphones and the Quest, and developing a comfortable UX evaluation method for quality assurance in the metaverse can increase the number of users.

REALITY is a smartphone application that focuses on manga and anime aesthetics and streaming features to create a world where teenagers, Generation Z, and manga and anime enthusiasts can enjoy chatting with friends, giving gifts in a manga and anime world, and wearing anime and manga–themed clothing. What style and quality should the renderings in such a world have? A photorealistic style would not be suitable. Abstract avatars, or ones that significantly differ from one’s actual physical features, would be more appropriate. Users could also choose their avatars to suit a manga or sci-fi aesthetic. Rendering methods based on a cartoonish style and unlit lighting can be used to avoid the uncanny valley. As several studies paradoxically prove, it is not necessary to be photorealistic to match realistic attributes. AgileAvatar takes feature vectors from a live-action photo and applies them to the placement parameters of encoded avatar elements (like the Nintendo Mii). Although this method sets reality as the Ground Truth, it approximates a rendering of a one-shot cartoonish image, so there may be a change of purpose. Users who want an avatar of themselves rendered like a 2D photo that they can use to interface with society are only a subset of all users. Instead, users want an ideal picture of themselves, so a process similar to using a beauty filter is more in demand.

This paper proposes a standard R&D process in the metaverse as a hypothesis. First, there is the overall picture of the new UX that metaverse users seek and the functions required of avatars that cannot be fixed. Since these diverge infinitely, they have different characteristics from game content. This study defines user characteristics as “a proposal of attractive engagement that teenagers who like anime and manga will experience using XR devices in the near future.” Therefore, the challenge will be to research and achieve a PoC that can answer these user characteristics. R&D should develop a PoC that facilitates evaluating a practical application as a commercial service through quality and user testing conducted on the platform. Figure 1 illustrates the process that identifies the technology groups and R&D problems to be solved to complete the technical PoC and prepare for evaluation through internal and user testing.

When considering UX in the school-life metaverse, we wanted to create an experience as close as possible to a teenager’s school experience. We researched how to make a cross-platform metaverse that used PCVR and standalone HMDs, such as Meta Quest 1 and Meta Quest 2 as possible environments. Avatars in our target application have an average polygon count of approximately 30,000 and a material count of 6 14, heavier than avatars used in general mobile applications. Figure 3 shows the loads of the avatars in Quest 1 and Quest 2. When the frames per second (FPS) drop below 60, a slight delay is perceived, and when the FPS drops below 40, the delay becomes obvious. On Quest 2, the FPS starts dropping when more than 22 avatars are rendered. The benchmark for avatar rendering (see Figure 2) points the limit of 10 (Quest 1) or 23 (Quest 2) simultaneous avatars, but in real scenes, when buildings and particle effects are also rendered, the actual limit is probably lower. We also consider the IPD. Instead of a world established with fixed players at a fixed distance, as in a soccer game, Proxemics in Virtual Reality: What Should We Put to the Test in Social VR? (Welsch et al.)[WRM21] argues the appropriate distance and IPD should be designed with culture in mind. We assume that various avatars communicate at a maximum distance of 1–2 meters, while generating collisions for individual avatars and avoiding overlap. Considering the interference in stereophonic sound, we consider that 5–11 avatars can be displayed simultaneously while maintaining quality communication. After creating and testing a school life metaverse experience, we found that it is possible to provide users with a musical experience using a variety of avatars and effects if the number of simultaneous connections is approximately four (see part V/B.).

In a metaverse space, avatar materials, shaders, and room lighting must be set so that the avatar’s appearance does not change significantly. Additionally, in a cross-platform metaverse where multiple platforms are connected, the avatar may need to be optimized for the user’s device environment. In such cases, it is important to consider atlasing the avatar’s materials and textures (see Figure 4), and maintaining compatibility with the physics applied to the hair and clothing. We must consider the types of movements an avatar will make when deciding on its costume. These could be fairly limited, such as walking, or extensive, such as dancing. For instance, one potential issue is when the legs often penetrate the costume if the avatar is wearing a skirt (see Figure 4). We now know that our PoC should account for various limitations such as the performance, IPD, and look dev.

To realize avatar karaoke, we created an automatic avatar music player system using SMF files (music scores, Standard MIDI, .mid) and AI Fusion (see Figure 5). Similar attempts already existed, such as AI Concert Creator (https://www.concertcreator.ai/). The play is superb and precise. However, if the AICC example is watched in stop-motion, we observe ingenuity in piano playing, not only in the music but in the fingering and the rests. Musicality can generate these elements, such as in jazz or classical piano. As a metaverse service, they should be processed as the real-time movements of avatars in individual user performances (rather than being uniquely generated from scores or sound sources). Musical experiences in the metaverse are not only about playing instruments but also about singing. The technology for generating facial expressions from audio, especially mouth shape, has been available in the film and animation industry using LipSync for many years. It is available as a real-time animation generation technology in the Meta Oculus OVR LipSync. However, many basic technologies are based on vowel classification in speech signals. While the degree of mouth opening and closing and related shapes are classified, the technologies that change the degree of mouth opening and closing according to singing intensity are not generalized. In addition, performers express themselves not only with their mouths when singing but also with their facial expressions and body movements. Considering the current use of stand-alone HMDs, the system targets photorealistic avatars and avatars with limited control rigs, such as in manga and anime. In the future, this could be controlled by machine learning, and we already know that a classifier for unique facial expressions is possible, as shown in AlterEcho: Loose Avatar-Streamer Coupling for Expressive VTubing (Tang et al.)[TZP21]. This study examines the generation of avatar facial expressions using recorded speech in multiple language environments while highlighting the limitations of expression in singing choruses. Can the results analyzed by audio files with NVIDIA Audio2Face be used for singing choruses? If they can, what are the potential limitations and possible solutions proposed?

As previously introduced, the AI Fusion system uses a mix of MIDI-generated and motion-capture animations. First, we examine how the MIDI-generated animations are handled in Unity.

For avatar karaoke in VR to be successful, the avatars must make different facial expressions in sync with their singing. They must easily convey emotions, and because the metaverse is used globally, these emotions need to be shared among people from different cultures and languages. This can be achieved using the international language of emoticons; they do not require translation and can be understood by everyone. The exaggerated faces of avatars are similar to emoticons. Most animated avatar expressions, such as those of VTubers’ avatars, are animated using facial recognition from a camera. However, using an HMD does not allow for accurate facial recognition. Therefore, we need to investigate voice and speech recognition to create synchronized lip sync and facial animation. Also, tracking data cannot be applied directly to exaggerated faces. We explored various options, including rule-based state machines and deep-learning base classifiers (see Figure 10), using random forest, the depth camera of an iPhone X, and blendshapes. The result is a vision system for faster smartphone tracking, with frame rates of 30 fps or more and fewer control points for a more realistic representation. In other words, high-frame-rate feedback loops were found to handle diverse user-intended representations better than complex decision algorithms.

Another option was to use NVIDIA’s Audio2Face application. In theory, this application can simulate a wide variety of facial expressions and emotions by using only voice input. However, we quickly understood that it would be difficult to implement the complexity of all the parameters that Audio2Face manages onto anime-type avatars, as they have far fewer parameters available to modify (see Figure 11). At the time this paper was written, Audio2Face supports only some Latin-based languages such as English, French, or Spanish. Japanese, Chinese, or Korean are not supported, meaning that the lip-syncing is not accurate. Lastly, we have yet to test Audio2Face’s real-time lip sync in an application with Audio2Face.

Tang et al. [TZP21] propose a solution to dynamically play animations using various sources of input such as voice, facial capture from ARKit, and personality factors. In this case, speech recognition cannot control facial expressions directly; it triggers premade animations. While the system can work in a chat scenario, it cannot convey the emotion and intensity of a singing session.

In this research, we evaluated the current state of the art of various AI-driven animation technologies, what these technologies can create, and the related issues that still need resolution.

QueTra is a simple motion capture system that uses Meta Quest to create avatar animations compatible with Unity’s humanoid format. It supports animation output in BVH format, making it suitable for a range of uses. In addition to production for internal projects, the system was used in a children’s workshop, where participants created their own metaverse. Children made avatars with dance motions in Unity using QueTra, which they ran on Web3D and viewed from their smartphones (see Figure 12). The initiative to turn Quest into a simple cloud-based motion-capture system has various potential applications, although Quest does not have an interface to a PC. Its development will continue in cooperation with AI Fusion techniques.

In the GVBand application (see Figure 13), the user enters a metaverse and practices for a performance as a high-school music club member. Up to four users can play at the same time. No guitar, bass, drums, or piano knowledge is required. As participants play along with the music, their “excitement level,” which is visualized as a numerical value, increases. As the excitement level rises, effects are displayed, and gifts appear. Up to four independent Quests can be in session simultaneously, including voice chat via the cloud. Using AI Fusion, the neck, head, and one hand can be used for expression while playing an instrument. This may not sound like much movement, but as mentioned previously, we found that individual musical expression includes gestures of the neck and the direction of the gaze and occurs during rests and intervals in the performance and in the consensus building among individual users.