Identification of Abnormal States in Videos of Ants Undergoing Social Phase Change

Inference and prediction of current and future collective states is potentially useful in a number of applications. In human crowds, intelligent surveillance cameras can detect abnormal collective states (e.g., conditions consistent with group-level panic or rioting behavior) (Mehran, Oyama, and Shah 2009) so that authorities can prioritize surveillance resources and execute proactive mitigation strategies. Alternatively, an individual robot in a multi-robot system can use local information of the pose of nearby robots to infer the large-scale formation of its team and then alter its own trajectory to more effectively achieve a group-level response to a stimulus encountered at distal ends of the team (Choi, Pavlic, and Richa 2017; Choi, Kang, and Pavlic 2020).

In behavioral ecology, however, modeling efforts have been focused either on the coarse-grained collective scale or the fine-grained individual scale but rarely the connection between the two. For example, many mathematical and statistical models have been developed for understanding the evolution of group-level states and how they adapt to changes in the environment (Couzin et al. 2002; Reid et al. 2015; Sasaki et al. 2016; Pratt et al. 2002; Buhl et al. 2006). These approaches provide insights into the overall function of collective states but do not provide so much insight into how to map observations of an individual to the collective context of that individual. On the other end of the spectrum, more recent deep learning approaches for image segmentation or object detection have been tuned to track individual animals from video frames (Bozek et al. 2018; Nath et al. 2019). These efforts are focused on accelerating data acquisition for existing statistical pipelines that human researchers employ and not on making automated inferences across the individual–group scales. Calhoun, Pillow, and Murthy (2019) used an unsupervised learning framework to discover latent states in Drosophila melanogaster flies during courtship, but the inference scale was only limited to the group of two engaged flies while our work deals with much larger social groups (an entire animal society).

Classical OC methods, such as One-class Support Vector Machine (OC-SVM) (Schölkopf et al. 2001) and Support Vector Data Description (SVDD) (Tax and Duin 2004), either use a hyperplane or a hypersphere tightly bounding the known-class data for separation. Recently, these methods have been augmented with autoencoders that highly reduce input dimensions of graphical data without supervision (Xu et al. 2015; Ribeiro et al. 2020). As an extension, DSVDD is designed to optimize the objective of SVDD in an end-to-end deep neural network pipeline (Ruff et al. 2018). In particular, the autoencoder’s encoder is fine-tuned to generate a feature space in which the in-class samples lie densely close to a pre-defined central vector $\vec{c}$, while the out-of-class samples are sparsely away from them (Fig. 2b).

Although DSVDD showed competitive performance with several benchmark datasets, we argue that its distancing scheme – simply using the distance to $\vec{c}$ – is not sufficiently rich to distinguish novel samples, and we combine DSVDD with a generative approach to improve OC for this case.

Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) can also be used in OC by synthesizing fake outcomes consistent with typical samples that are then used in training to improve the generalizability of the OC (Sabokrou et al. 2018; Yadav, Chen, and Ross 2020; Perera, Nallapati, and Xiang 2019). However, these approaches adopt the conventional min–max scheme of GANs to closely emulate the data distribution of the available class (Fig. 2c) although the ultimate goal is to identify novel samples from a different distribution. Instead, our IO-GEN generates fake outcomes even closer to the idealized central vector $\vec{c}$ (Fig. 2d). These more prototypical samples allow the subsequent classifier to learn sharper discrimination in the DSVDD feature space between normal and abnormal samples.

DSVDD in our framework follows its original design from Ruff et al. (2018). It is built from the encoder part $\phi$ of a pre-trained autoencoder that is used to learn a feature space $\mathcal{F}$ in which the samples of known class have a lower average distance to a central vector $\vec{c}$ than those of novel class. Specifically, we adopt One-class DSVDD, which minimizes the objective:

$$\min_{W}\frac{1}{n}\sum_{i=1}^{n}||\phi(x_{i};W)-\vec{c}||^{2}\ +\frac{\lambda}{2}\sum_{l=1}^{L}||W^{l}||_{F}^{2}$$

where $||\cdot||_{F}$ is the Frobenius norm. The first term is closing the distance between $\vec{c}$ and the feature representation of each sample $x_{i}$ in encoder $\phi$ parameterized by $W$, and the second term is a weight decay regularizer for $L$ layers with $\lambda>0$. In the original method of DSVDD, the trained parameters $W=W^{*}$ are used to generate a distance:

$$s(x)=||\phi(x;W^{*})-\vec{c}||^{2}$$

that is a proxy for how atypical a sample $x$ is. For some threshold $\tau>0$, $s(x)>\tau$ classifies $x$ as atypical. Our method, however, substitutes the distancing heuristic $s(x)$ by IO-GEN and Classifier, described below, that we argue better utilize key features in the normal set in order to discriminate abnormal data after training.

As shown in Fig. 3, IO-GEN $G$ is designed to operate with both the pre-trained DSVDD $\phi$ and a discriminator network $D$, as a generative model of optical flows. With the discriminator, an adversarial learning is performed following the standard objective:

$$\min_{G}\max_{D}\Big{(}\mathbb{E}_{z\sim\mathit{N_{\sigma}}}[\log(1-D(G(z)))]+\mathbb{E}_{x\sim p}[\log(D(x))]\Big{)}$$

where $\mathit{N_{\sigma}}$ is the zero-mean normal distribution with standard deviation $\sigma$, and $p$ is the probability distribution of real optical-flow data. Due to the first term, the outcomes from IO-GEN are adjusted to appear sufficiently realistic to deceive the discriminator. Also, the DSVDD is used to force the learned synthetic data to be inner outliers close to $\vec{c}$ in $\mathcal{F}$, while the parameters of itself are not updated. In particular, we use the feature-matching technique proposed by Salimans et al. (2016), which incorporates the minimization:

$$\min_{G}||\mathbb{E}_{z\sim\mathit{N_{\sigma}}}[\phi(G(z))]-\vec{c}||^{2}_{2}$$

The composite loss function for IO-GEN is:

$$\mathit{L}_{G}=\mathbb{E}_{z\sim\mathit{N_{\sigma}}}[\log(1-D(G(z)))]\\ +\lambda\Big{(}||\mathbb{E}_{z\sim\mathit{N_{\sigma}}}\ [\phi(G(z))]-\vec{c}||^{2}_{2}\Big{)}$$

where hyperparameter $\lambda>0$ determines the relative weights between the two terms to minimize. In other words, IO-GEN is trained to produce ant behavioral flows that look real and also feature the closest proximity to $\vec{c}$ in $\mathcal{F}$ of DSVDD.

Classifier utilizes real data from stable colony states as well as the generated IO-GEN inner outliers to learn to predict the likelihood of unstable behaviors on given $m$ instant frame images as in general binary classifiers. We introduce a novel strategy, label switch, during training where the real stable samples are labelled as “unstable” (atypical) and the synthetic ones are labeled as “stable” (typical). This technique leads the Classifier to eventually make low-, mid-, and high-range likelihood predictions for synthetic, stable, and unstable data, respectively, as though the augmented state of inner outliers was the “most stable” state . That is, Classifier offers likelihood outcomes somewhat consistent with the class distribution around $\vec{c}$ allowing for a clear separation between real stable and unstable data, which will be demonstrated in the following sections.

A Deep Convolutional Autoencoder (DCAE) is used as the backbone of DSVDD and IO-GEN once it has been trained with the data of stable class to minimize the reconstruction error in the Mean Squared Error (MSE) between the input encoded and the decoded output. Per input, $m$ optical flows are all stacked one another to constitute an input $x\in\mathbb{R}^{64\times 64\times 2m}$ after normalization to range in $[-1,1]$. In the encoder, three convolutional layers with $32,64,$ and $128$ 2D kernels are employed in series as each kernel is of $3\times 3$ size. Also, every output is followed by a ReLU activation and 2D maxpooling. The decoder has the reversed architecture of the encoder with two modifications: 2D upsampling instead of maxpooling and an added output layer with $2m$ kernels and hyperbolic-tangent ($\tanh$) activation. An additional $32$ convolutional kernels are placed as the encoder–decoder bottleneck to obtain a compact encoding scheme $\vec{v}\in\mathbb{R}^{1\times 2048}$ when flattened. DSVDD takes advantage of the pre-trained encoder to reshape the space of $\vec{v}$ by learning data description $\mathcal{F}$ as the reference vector $\vec{c}$ is the mean of available encoded samples according to Ruff et al. (2018).

IO-GEN essentially employs a fully connected layer with the ReLU activation that takes a noisy vector $\vec{z}\in\mathbb{R}^{1\times 100}$ as input. It is then connected to a replica of the pre-trained decoder so that realistic synthesis can be learned faster from the prior knowledge of reconstruction. The discriminator network builds an extra fully connected layer with a sigmoid activation on top of encoder, but its weights are all reinitialized because otherwise it appears to easily overwhelm IO-GEN in performance causing unstable adversarial training. Also, $\lambda=10$ was empirically found most effective to minimize $\mathit{L_{G}}$.

Because Classifier comes after DSVDD, it has an independent architecture in which five convolutional layers learn $8,16,24,48,\text{ and }48$ one-dimensional kernels, respectively. Each layer has a LeakyReLU activation ($\alpha=0.3$) and 1D average pooling, and lastly, a fully connected layer is deployed to provide a predicted likelihood of unstable state via a sigmoid function. All codes are also available online²²2https://github.com/ctyeong/IO-GEN.

We first attempt to make use of the optical flow dataset to discover insightful motional patterns without any complex model. As with Mahasseni et al. (2013), we compute an optical flow weight $w_{i}$ for each frame $i$ by averaging the magnitudes of flow vectors at all locations. Fig. 4 displays the obtained weight signal in time as $m=1$ optical flow frame is considered for each sampling interval.

For the first two days, the weights generally stay in a narrow range implying some behavioral regularity maintained among ants. Then there is a noticeable increase in weights starting at D+1 just after the gamergates are removed because the removal triggered a social tournament involving frequent aggressive behaviors. As the transient period evolves, the magnitude continues to decrease and finally recovers the original extent roughly on D+10 even though several ants continue to present hostile interactions at that time.

Consequently, a simple model might be built that uses the overall rise of flow weight as the only feature to distinguish the unstable colony from the stable especially at early development of the unstable state. However, following our results, we will provide concrete examples that show the limitations of such a design and the need of more complex models for reliable predictions.

Here, we demonstrate the OC performance of our proposed method. We first describe an ablation study to find the best number of optical-flow frames per input. Next, baselines used for comparison are introduced that will help us explore our method’s overall reliability and prediction robustness in various time windows during colonial stabilization.

As in previous works (Ruff et al. 2018), the Area Under the Curve (AUC) of the Receiver Operating Characteristics (ROC) are measured for each model to reflect the separability between classes. Moreover, the average over three splits is reported with the standard deviation when needed.

Figure 6 compares synthetic optical flows from IO-GEN to real optical flows; the generated optical flows are visually similar to real flows. Furthermore, Fig. 7a illustrates that the lowest distance distribution to $\vec{c}$ is measured with IO-GEN, as designed, whereas GEN behaves similarly to the stable dataset. Fig. 7b finally shows the predictive outcomes of Classifier, which are likelihoods of unstable state. With the label switch, the confidence becomes positively correlated with the distance to $\vec{c}$ viewing inner outliers as samples from the most stable colony. Clear differences between classes imply that learned knowledge to discriminate stable and more-stable states in DSVDD can be transferred for classification of another pair as stable or unstable.