SportsXR - Immersive Analytics in Sports

Sports analytics has become popular with the success of the MLB Moneyball story in 2002. Oakland Athletics General Manager Billy Beane, with a limited budget, used sabermetrics to draft undervalued players and subsequently won games against rich teams like the New York Yankees [8]. The recent explosion of computational power, motion capture technology, and statistical methods make advanced sports analytics feasible in many scenarios. This includes game outcome prediction, measurement and evaluation of player performance, analysis of rules and adjudication, and within-game strategy.

However, these fully automatic methods do not integrate the knowledge of decision-makers (i.e., coaches, scouts, and managers) in an effective way [2]. Data visualization and visual analytics have been integrated into the sports analytics workflow to enable human-in-the-loop analytics. For example, Seidl et al. [13] use NBA game tracking data to perform data-driven ghosting defense prediction. They also offer a tablet-based interface for coaches to perform real-time strategy planning (see Fig. 1). Wu et al. [18] create a holistic visualization system for table tennis match data and empower domain experts to find unnoticed patterns (see Fig. 2.1). Perin et al. [10] have compiled a comprehensive survey of these systems. However, there is still a clear demand for more natural and transparent interface design to intuitively interact with sports data to perform meaningful data analytics and prediction.

Immersive analytics leverage new display and interaction techniques, such as AR/VR headsets, for visual analytics [9]. Compared to traditional displays, AR/VR headsets can render very large displays at low cost with a high degree of portability compared to wall displays and CAVEs. Therefore, AR/VR headsets can create immersion through situated and collaborative visualization. This opens up opportunities for direct interaction with visuals that are radically different from desktop computing paradigms. Situated analytics is a form of in-situ interactive visual analysis. In situated analytics, visual representation of information is immediately linked to physical objects to facilitate sense-making [16] (e.g., see Fig. 2.2).

VR for sports training has been preliminarily explored, e.g., for basketball and baseball: Tsai et al. [17] use a VR simulation to train basketball players’ decision-making skills, and Zou et al. [20] visualize simulated pitch and bat swing data in VR to improve a baseball player’s batting eye. Applying AR in sports is largely unexplored and can bring unique opportunities for the integration of digital information and physical movements.

There are several innovative commercial products applying AR/VR for sports training, coaching, and fan experiences. STRIVR  [14] uses VR as an immersive learning platform to simulate real game scenarios to train decision making and mental preparation for professional football players, skiers, and basketball players. Rezzil [11] provides similar VR training with coaching and analytic features for professional soccer players. Second Spectrum [12] experiments on player tracking and AR features for visual information augmentation on broadcasting and video re-play. CourtVision [5] provides information and stats overlays in AR for enhancing live sports viewing experiences for NBA basketball fans within minutes of the live game. However, those applications only cover some sub-areas of SportsXR, leaving many other areas still being unexplored.

Sports data are inherently complex. Typical datasets contain a combination of heterogeneous, multi-dimensional, and unstructured data. Examples include box score data, tracking data, scouting reports, game video clips, players’ mentality, and many other qualitative metrics [10]. Standards for data collection need to be enforced to ensure that the collected data is unbiased and compatible with the subsequent analytic workflow. Meeting these standards requires considerable efforts for real-time data recording, pre-processing, data cleaning, and formatting. Furthermore, in the context of situated and in-game analytics, data often needs to be extracted from video feeds and live sensors. This, in turn, requires state-of-the-art computer vision techniques to track game objects and players. For example, basketball analytics collects dynamic movement data of all players and detects shot types and defensive vs. offensive tactics to evaluate the performance of each player, while baseball analytics focuses more on pitch type and base rate to evaluate player value.

The first challenge, therefore, lies in identifying, collecting, and extracting various data required for different sports and contexts.

Research on optimal statistical models and visualization tools for game prediction and evaluation in various sports has seen a huge growth in the last decade. However, current sports analytics only occur asynchronously in the hands of data scientists, leading to inefficient communication and inaccurate decision making on the athlete side. By bringing in-situ analytic power to domain experts such as coaches and athletes, immersive analytics will play an important role in eliminating the gap between data-driven strategies and executions.

The second challenge, therefore, lies in making analytics models not only understandable to coaches and athletes but to also enable fast in-game and in-situ analytics.

One distinguishing feature of SportsXR is that analytics need to be performed in-situ and need to be updated dynamically to maximize its impact. However, coaches and athletes primarily have to focus on real-world movements and events. Thus, the visualizations in SportsXR need to be situated, highly dynamic, concise, and context-dependent to play an auxiliary role. For example, a basketball coach using SportsXR to evaluate offensive options needs different analytics based on a player’s location and a defender’s movement. A spatial shot percentage map of the ball handler might be helpful when the defense is wide open, but when the player is being contested, the shot percentages of other shooters on the team are more important. Deciding when and how to change the presented data to optimally support data-driven decision making will be a key design consideration in SportsXR.

The third challenge, therefore, lies in defining the context-dependent factors based on real-world events to decide how to present and interact with dynamic situated data.

User-centered design has been proven to be an effective way of engaging end-users [7]. Batch et al. [3] worked closely with economic data analysts to integrate immersive visualization into their actual workflow. In contrast, domain experts in sports rarely perform data analytics themselves but make in-game decisions based on experience rather than based on data. To understand the decision-making workflow of coaches and athletes, researchers first need to gain in-depth domain knowledge. Based on this domain knowledge, they can then identify and extract analytic components from the heuristic insights in order to improve the accuracy of their decision making. Furthermore, researchers need to educate their sports collaborators on analytics and visualization methods, as well as introduce them to novel AR interfaces for situated analytics.

In our current research project, we collaborate closely with both Harvard Men’s and Women’s Basketball teams to gather first-hand expert knowledge. Building personal sport expertise and working with data analysts on an interdisciplinary sports analytics team can lead to more effective collaborations with coaches and players.

The fourth challenge, therefore, lies in establishing an effective collaboration between immersive analytics researchers and sports experts.

Technological innovations in sports training are not just crucial to improve training effectiveness with instant performance feedback and overall body condition monitoring, but also to reduce injuries and to excel former records. Most data collected during training are interpreted and communicated to athletes by their coaches. However, embodied data analytics can empower athletes with the ability to self-evaluate and modify their techniques in real-time. Furthermore, immersive visualizations can enhance the collaboration between coaches and athletes by embedding the visuals spatially into the real world (see Fig. 1). A pioneer example of this was demonstrated by Eliud Kipchoge breaking the 2-hour marathon barrier with the help of a projected pacing visualization in real-time [4].

So far, sports analytics has had its highest impact on coaching decision making. Data scientists in the backroom apply advanced machine learning and computer vision to convert video and statistical data into strategic insights. On top of the advanced statistical models that track detailed winning factors in players’ performance, the close collaboration between analysts and coaches has been the key factor for the successful team rebuild of the NBA Philadelphia 76ers [6]. However, for many teams and individual sports like tennis or track-and-field, employing a professional data analytics team is unfeasible. The spatial immersion and in-situ decision making of situated analytics allow SportsXR to embed analytics into the coaching workflow for a seamless integration of analytic and coaching insights.

Immersive technologies have gradually become a big part of social experiences, such as AR filters on Snapchat or Instagram and VR game watching of the FIFA World Cup. To bring fan experience to the next level, personal storytelling through data visualization and content creation plays an important role in deepening a sports fan’s engagement. SportsXR can add digital information overlays in real-time, and provide a more engaging interactive experience such as video annotation, dynamic data lookup, performance comparison, and customized view manipulation. The design challenges for immersive visualization and interaction, in this case, are largely unsolved research questions.