\systemname: A Visual Analytics Tool to Explore Human Behaviour based on fNIRS in AR guidance systems

Abstract

The concept of an intelligent augmented reality (AR) assistant has \revisionsignificant, wide-ranging application\revisions, with potential uses in medicine, military, and mechanics \revisiondomains. Such an assistant must be able to perceive the environment and actions, reason about the \revisionenvironment state in relation to a given task, and seamlessly interact with the \revisiontask performer. These interactions typically involve an AR headset equipped with sensors which capture video, audio, and haptic feedback. Previous works have sought to facilitate the development of \revisionintelligent AR assistant\revisions by visualizing these sensor data streams \revisionin conjunction with the assistant’s perception and reasoning model \revisionoutputs. However, existing visual analytics systems do not focus on user modeling or include biometric data, and are only capable of visualizing a single task session for a single performer at a time. \revisionMoreover, they \revisiontypically assume a \revisiontask involves linear progression from one step to the next. We propose a visual analytics system that allows users to compare performance during multiple task sessions\revision, focusing on non-linear tasks where different \revisionstep sequences can lead to \revisionsuccess. In particular, we design visualizations for understanding user behavior through functional near-infrared spectroscopy (fNIRS) data as a proxy for perception, attention, and memory as well as corresponding motion data (acceleration, angular velocity, and gaze). We distill these insights into embedding representations that allow users to easily select groups of sessions with similar behaviors. We provide \revisiontwo case studies that \revisiondemonstrate how \revisionto use these visualizations to gain insights about task performance using data collected during helicopter copilot training tasks. Finally, we evaluate our approach by conducting an in-depth examination of a think-aloud experiment with five domain experts.