Influence of Video Dynamics on EEG-based Single-Trial Video Target Surveillance System ¹¹1 ^††thanks: This work was supported by the Agency For Defense Development Grant Funded by the Korean Government (UI233002TD).

A total of 13 healthy male and female subjects participated in the EEG data acquiring task. All subjects were aged between 20 and 40 years old and had no neurophysiological or mental disorders. Four of the subjects participated in the offline EEG data acquiring process, which is used for training dataset of the target detection model. The other nine subjects participated in the online testing experiments which are for evaluating the model performance in a real-time asynchronous environment.

Three video clips ‘Video1’, ‘Video2-N’, and ‘Video2-AI’ are used as stimuli for acquiring EEG signals from subjects. Each video clip is simulated video surveillance footage, assumed to be recorded in a mountain forest area. Each of these clips is divided into quarters, which shows various scenes of surveillance cameras in different settings of angles and positions. ‘Video1’ is 300 seconds long, while ’Video2-N’ and ’Video2-AI’ are 480 seconds long.

For Video2-N’ and ’Video2-AI’, surveillance cameras rotate for three seconds to monitor other sites every five seconds periodically, in turn. Moreover, the weather of the surveillance location changes into foggy or rainy as time goes by.

In each video clip, three types of targets ‘deer’, ‘wild boar’, and ‘soldier’ suddenly appear and last for one second several times, in a specific time point and location. Deer and wild boars are non-hostile targets which can be classified as error-targets, and soldiers are hostile targets representing true-target.

Two video clips ‘Video2-N’ and ‘Video2-AI’ are essentially identical except for one difference, ‘Video2-AI’ shows a red bounding box that guides the locations of targets when they appear, whereas ‘Video2-N’ does not. ‘Video1’ has different timing, location, and the number of times in target appearance with ‘Video2-N’ and ‘Video2-AI’. Table I shows the differences between video clips.

Each of the four subjects was instructed to concentrate on video clips to detect each target while equipping the EEG acquiring devices. EEG signals were recorded at a 250 Hz of sampling rate, with the BrainVision Recorder software provided by BrainProduct (GmbH, Germany), using 32 Ag/AgCL electrodes placed in 10-20 international systems. During the acquisition, the impedance of each electrode was maintained under 10 k$\Omega$ for excluding noises caused by poor contactivity of electrodes. To minimize artifacts, the recording proceeded in a soundproof experimental booth while the lights were off, and the subjects were asked to minimize facial and body movements. Each subject watched all of the video clips twice each, six sessions of watching in total, and had two minutes of resting time when a session ended. Fig. 1 represents the overview of the configuration of the EEG data acquisition process, and Fig. 2 shows the example of an EEG acquiring session.

We hypothesized that there would be general EEG patterns related to the target detection across subjects. Whereas, bio-signals like EEG have high variations in patterns between individuals [18, 19, 20, 21], even if they perform the same task or are in the same condition, resulting from unique neurological and physiological characteristics of individuals. To consider both of these factors, we pre-trained the model using all of the subject’s EEG data, and fine-tuned model parameters with calibration when testing.

To consider the imbalance of each class, The performances of the model were evaluated with macro $F_{\beta}$, which can be defined as

where $N$ is the number of classes, $F_{\beta}^{c}$ is the $F_{\beta}$ of the class $c$, and $\beta$ is the parameter to adjust weights of recall and precision. In this study, $N$ is three because we defined the problem as a 3-class classification task, and we set $\beta$ as two to give larger weight on the recall.

In the experiments with ‘Video1’ for five subjects, the proposed model achieved the macro $F_{\beta}$ of 0.7000, 0.5238, 0.7568, 0.5238, and 0.7568 for subject 5, 6, 7, 8, and 9 respectively, and the overall performance was 0.6522. When the model was trained with the EEG data of ‘Video2-N’, the performance resulted in 0.1200, 0.1569, 0.0392, and 0.1967 for subject 10, 11, 12, and 13 respectively. The model trained with the EEG data from ‘Video2-AI’ showed the performance of 0.0784, 0.1091, 0.1200, 0.0952, and 0.1007 for subject 10, 11, 12, and 13. The model trained with EEG data of ‘Video1‘ showed the highest performance, while the data of ‘Video2-AI’ resulted in the lowest performance. Table II shows the overall performances of the experiments.

The model trained with the EEG data of ‘Video1’ showed promising performance in the EEG-based target detection task. In contrast, the model trained with the EEG data of ‘Video2-N‘ and ‘Video2-AI’ showed poor performance, even if other factors that affect model performance, such as the model architecture, the number of parameters, and training hyperparameters, were set in the same condition with the model of ‘Video1’. For this performance gap, we analyzed the model and input EEG signals to identify any other factors that might affected the results.

ERPs are commonly considered important EEG feature that highly influences the decision of the EEG-based target detection models [5, 4]. Therefore, we calculated the grand average of all trials of ERPs from ‘Cz’, ‘C3’, and ‘C4’ channels, to identify if the ERPs played an important feature of the model. Fig. 3 represents the result of the ERPs by classes, in camera rotation. In the figure, subject 11 and 13 showed better performances compared to the subject 10 and 12, but their ERPs showed less clearly distinct differences in patterns between classes. This result implies that the model did not count temporal patters of ERPs as important features, for classifying each class.

We also analyzed the model using a saliency map [25], to visualize the importance of each EEG channel for the model inference. In Fig. 4, the area colored in deep blue represents that the model performance decreases when the EEG channel data of the area is removed, meaning the EEG channel plays an important role for the trained model in inference. The EEG channels of the red area represent the relatively lower importance of the channel compared to others. As shown in the figure, important channels for the trained model were mainly are located in central, temporal and occipital regions, which are related to the passive visual perception, like camera rotation, rather than the frontal and temporal-parietal regions which are mostly related to dicsciminative response task, such as, target detection [26, 27].

These results would be caused by differences in video dynamics, such as camera rotation and weather changes, arousing the passive visual perception-related EEG features more than the discriminative response task-related EEG features to the subject [28]. Imbalanced inductions of EEG features would lead the model to be biased on passive visual perception-related EEG features, therefore, the performance of target detection would be poor.