Matching Neuromorphic Events and Color Images via Adversarial Learning

Data acquired from heterogeneous sensors constitutes our digital modern life. As multi-modal data grows rapidly, cross-modal retrieval has drawn great attention due to its widespread application prospects, such as multi-sensor information fusion, object recognition and scene matching. It aims to take one modality of data to retrieve relevant data of another modality. Till now, multi-modal retrieval has been widely studied, including retrieving infrared images with visible images [14], retrieving videos with texts [28], retrieving photos with sketches [30], etc. The major concerns are how to perform the cross-modal correlation modeling to learn common representations for various modalities of data so that the similarity between different modalities can be measured.

There is a significant amount of work in learning a common subspace shared by different modalities of data. Most studies extract the common features which are robust in multi-modalities of data and achieve multi-modal matching by comparing the similarities between the features. Edge features are widely used since there is usually a certain relationship between the edges in different modalities of data [24]. Besides, some existed descriptors developed in single modality like SIFT [16] are optimized and improved as feature representation methods for cross-modalities. Although there are several ways to extract features for neuromorphic events and color images, the methods developed on neuromorphic events is not applicable to color images and vice versa. Another line is to project the features of different modalities to a common feature subspace, in which the similarities are measured. Due to the recent progress of deep learning, the Convolutional Neural Networks (CNN) acting as high-level feature extractors can generate modality-invariant representations, which can be categorized into two types: the classification-based network and the verification-based network [34]. The former is trained to classify architectures into pre-trained categories and the learned embedding, e.g., FC7 in Alexnet, is usually employed as cross-modality feature. The latter may use a siamese network and employ the contrastive loss to learn common representations.

Although cross-modal matching among data of multi-sensors is a hot spot in the field of computer vision, to our best knowledge, there is no studies about cross-modal matching between neuromorphic events and color images yet due to the novelty of event-based vision field. Owing to the unconventional output of event camera and its inconsistent encoding content with conventional camera, existing cross-modal retrieval methods fail to tackle the EBIR task. In this work, we embed the adversarial training strategy into the process of representation learning to bridge the gap between neuromorphic events and color images.

One of the key barriers towards EBIR task is the scarcity of well-annotated datasets. Prior to our work, several datasets for development and evaluation of various event-based methods have already been collected [6]. For example, N-Caltech101 is an event-stream dataset for object recognition which is converted from existing computer vision static image dataset Caltech101 [5] using an actuated pan-tilt camera platform [21]; DET is a high-resolution dynamic vision sensor dataset for lane extraction [4]; and the MVSEC dataset is used to perform a variety of 3D perception tasks, such as feature tracking, visual odometry, and stereo depth estimation [35]. However, none of them are designed for the EBIR task.

Since a limited number of datasets are available in the community of neuromorphic vision, it is very difficult to develop a neuromorphic vision dataset by crawling data directly from the web like developing a computer vision dataset. Inspired by the methods of creating datasets in [21, 12], which convert existing computer vision static image datasets into neuromorphic vision datasets, we convert the popular frame-based instance retrieval dataset UKbench [20] to a dataset suitable for EBIR task, named as “N-UKbench”. Moreover, the neuromorphic events converted from static images may be different from that obtained by photographing the real objects. To provide access to evaluating algorithms applied to real scenes, we further provide EC180 dataset, which is collected by finding different objects to record. Such two datasets are important to exploit methods of cross-modal matching between neuromorphic events and color images.

As the most straightforward way to encode an event stream into a tensor, the event stacking has been applied in many tasks including steering prediction [18], visual-inertial odometry [25], and video generation [29], etc. Unlike previous stacking methods, we stack all events ignoring their polarity into one tensor $S(\mathbf{x}),\mathbf{x}=[x,y]^{T}$ to prevent the context from being separated into two dispersed part. Thus we can integrate the events within the interval $T_{s}$ into a tensor using:

where $\delta$ is the Kronecker delta, and $[x_{i},y_{i}]$ is the pixel position of an event.

The time surface is a spatial-temporal representation of asynchronous event stream. It has been adopted in tasks like object classification [10] and action recognition [23]. To generate a time surface $T_{i}(\mathbf{u})$ of an event $ev_{i}$, we need to first generate the Surface of Active Events (SAE) [1]:

where $\mathcal{A}_{i}$ is the SAE around the incoming event $ev_{i}$ for the pixels in the $(2R+1)\times(2R+1)$ square, centered at the pixel position $\mathbf{x_{i}}=[x_{i},y_{i}]^{T}$. And $\mathbf{u}=[u_{x},u_{y}]^{T}$ is the pixel in that square, $u_{x}\in\{-R,\ldots,R\}$, $u_{y}\in\{-R,\ldots,R\}$. Then we can apply the exponential decay kernel with time constant $\tau_{e}$ to generate a dynamic spatial-temporal time surface $T_{i}(\mathbf{u})$:

Besides above representations, we can also analyze events from frequency domain. The event frequency [19] counts the event occurrence at each pixel within a range as activation frequency to encode the event stream into a tensor. Since the noise in an event camera has relative low frequency, the representation is more friendly to noise tolerance. The event frequency $F(\mathbf{x},n)$ can be generated by using:

where $n$ is the total number of the activated events at pixel $\mathbf{x}=[x,y]^{T}$ within a time interval $T_{f}$.

We focus on the task of neuromorphic event and color image bimodal representation learning. We divide the event stream in 90 ms intervals, where each bin is converted to an “event image”. To make full use of the temporal information of the neuromorphic events, we further divide each bin into three consecutive parts. Each part is processed by the event representation method and acts as a channel of the event image. Let $\mathcal{C}$ be the set of all possible instances in a given dataset; $\mathcal{E}=\{(e_{i},c_{i}^{e})\}^{N}_{i=1},c_{i}^{e}\in{\mathcal{C}}$ and $\mathcal{R}=\{(r_{i},c_{i}^{r})\}^{M}_{i=1},c_{i}^{r}\in{\mathcal{C}}$ be the set of event images and color images respectively, where $e_{i}$ is an event image, $r_{i}$ is an original color image, $\{c_{i}^{e},c_{i}^{r}\}$ are their corresponding instance-level object identity label, and $\{M,N\}$ are the total number of event images and color images respectively. As event images $\mathcal{E}$ and color images $\mathcal{R}$ clearly have different statistical properties and follow unknown distributions, they fail to be directly compared against each other. Our goal is to learn a common subspace in which the similarity between data items from these two modalities (neuromorphic events and color images) can be assessed directly, i.e., learned features have the same or very similar distributions in the two modalities of data.

Towards this goal, our proposed ECFL method described in Fig. 2 embeds adversarial training strategy into the process of representation learning, so that the learned features combine discriminativeness and modality-invariance. Specifically, the aim of our framework is to learn two feature extractors $G^{e}:\mathcal{E}\rightarrow\mathbb{R}^{d}$ and $G^{r}:\mathcal{R}\rightarrow\mathbb{R}^{d}$ which respectively map the event image and color image into a common embedding space. The cost function which guides the learning process to provide the embedding with the desired properties contains two parts. The first part learns features that are distinguishable for different instances, including cross entropy loss and contrastive loss. The cross entropy loss is calculated by the output of identify classifier $C:\mathbb{R}^{d}\rightarrow\mathcal{C}$, which takes the feature representation as input and predicts the object identity label. It makes sure that the learned embedding space preserves instance-discriminative information. The identify classifier is composed of a softmax layer, whose number of output nodes equals to the number of instances $N_{ins}$. The contrastive loss is calculated directly in the embedding space, which ensures that the distance between the event image and color image is minimized if they denote the same instance and is greater than a margin value if they denote two different instances. The second part utilizes the adversarial training strategy to minimize the distribution distance between data items from two modalities, such that the feature distributions are well aligned in the embedding space, where a modality discriminator $D:\mathbb{R}^{d}\rightarrow[0,1]$ acts as an adversary. It predicts $0$ for event image and $1$ for color image. We regard a representation as a modality-invariant feature when the modality discriminator fails to discriminate modalities they belong to. The modality discriminator is composed of two FC layers and a sigmoid function.

The architecture of feature extractors $G^{e}$ and $G^{r}$ are based on that of ResNet [8], which has enough capacity of representations considering large margin of statistical properties between the event images and color images. The weights of the two feature extractors are shared everywhere. The weight-sharing constraint allows to align the distributions of two modalities from the very beginning of the feature extraction. Besides, there is another strategy: without any weight-sharing. It learns features of two modalities independently and relies entirely on the cost function to guide the alignment of them. But it can preferably deal with the heterogeneous nature of the input. Both strategies are evaluated in our experiment.

The feature extractors learn representations that are invariant to different modalities by constantly trying to outsmart the modality discriminator, which is working to become a better detective. At the same time, they learn representations that are distinguishable for different instances. Therefore, the learning stage contains two steps: (i) Update the modality discriminator $D$ by minimizing the loss of the modality discriminator; (ii) Update the feature extractors $G^{e}$ and $G^{r}$ by maximizing the loss of the modality discriminator while minimizing the cross entropy loss and contrastive loss. Let us consider a training batch $\mathcal{D}=\{e_{i},r_{i}\}^{n}_{i=1}$ including $n$ event image-color image pairs, where $e_{i}\in\mathcal{E}$ and $r_{i}\in\mathcal{R}$. The objective of the modality discriminator is to predict the modality of the input given its representation and can be written as:

where $F_{i}^{e}=G^{e}(e_{i})$ and $F_{i}^{r}=G^{r}(r_{i})$. The objective of the feature generator is now to defeat the modality discriminator, i.e., the modality discriminator should not be able to predict the modality of the input given its representation. In addition, it has to minimize the cross entropy loss and contrastive loss. The complete objective is then:

where $L_{id}$ is the cross entropy loss for predicting the object identity label:

$L_{ct}$ is the contrastive loss, whose goal is to make the distance between an event image and a color image of the same instance closer than that the one of two different instances:

$s=1$ if $e_{i}$ and $r_{i}$ denote the same instance else $s=0$ and $m$ is the margin. $\alpha$, $\beta$ and $\gamma$ control the contribution of the three items respectively. Suppose $\theta_{G}$, $\theta_{C}$ and $\theta_{D}$ are the parameters of feature generators, identity classifier and modality discriminator, respectively. We apply an alternating gradient update scheme similar to the one described in [7] to seek $\theta_{G}$, $\theta_{C}$ and $\theta_{D}$, as shown in Algorithm 1.

After learning, the modality discriminator and the identity classifier are stripped off during testing. We use the feature generators to map the event images and color images to the d-dimensional feature representations. Thus the similarity of neuromorphic events and color images can be simply measured by Euclidean distance. Color images in database are ranked according to the similarity, and the ones with similarity at the top are set as matching results.

We use the Pytorch to train our models. The modality discriminator is trained with Adam [9], using a learning rate of 0.002 and $[\beta 1,\beta 2]=[0.5,0.99]$. The feature generators and the identity classifier are trained with SGD [2], using a learning rate of 0.001 and a momentum of 0.9. The weighting factors $\alpha$, $\beta$ and $\gamma$ are set to 1, 0.01, 0.01. The maximum epoch of training iterations is set to 20.

We randomly select 2040 instances from N-UKbench as training set and the remaining 510 instances are for evaluation. We rotate each event image/color image to 9 angles ranging from $-45\degree$ to $45\degree$ and double the numbers by flipping them horizontally. All event images and color images are resized to the same size of $224\times 224$ pixels.

The purpose of this paper is to retrieve images of a particular object given an event stream as query. To the best of our knowledge, there are no studies about cross-modal matching between neuromorphic events and color images. Nevertheless, we still compare ECFL with Scale Invariant Feature Transform (SIFT) [16] and Local Self-Similarity (LSS) [26] because the event cluster in each stream forms the shape of the object. SIFT is the most representative in describing the structure information. We employ SIFT in combination with spatial pyramid [11] to describe both event images and color images and measure the similarity by histogram intersection. LSS ia able to capture shape features of a large image region and it is suitable for images with complex radiometric differences, which is widely adopted to multi-modal image matching, such as optical-to-SAR image matching [13] and sketch-to-image matching [26]. We use LSS with BoW [27] to describe event images and color images and measure the similarity by Euclidean distance. Tab. 1 shows the mean average precision (mAP) and the precision of top K retrieval results (acc.@K, K=1,3) of SIFT, LSS and ECFL while employing different event representations. Fig. 4 shows the precision of top K retrieval results of ECFL and Fig. 5a shows examples of retrieval results of ECFL using event frequency as representation. For each sample, the top 5 retrieved results are shown in each row.

It is shown that our proposed ECFL method (for each event representation) performs the best. It is able to bridge the large gap between neuromorphic events and color images and return the true matches. SIFT and LSS descriptors performs poorly for the novel EBIR task. Because the event image is different from the color image, it is produced by brightness changes, which results in the poor distribution consistency of their features. As shown in Fig. 4, the retrieval performance of the three event representations is very close. Among them, the event frequency performs slightly better. Comparing the other two event representation methods, the event frequency can significantly filter out the noise caused by the sensor since the occurrence frequency of noise at a particular pixel is low. Our proposed ECFL method, including ResNet-18 and ResNet-50 as backbone, produce a significant performance improvement. ResNet-50 architecture performs better than ResNet-18. It is expected that deeper architecture have more parameters, and can therefore better cope with the inconsistent distribution of features.

It has been verified that the strategy of all weight-sharing is better suited for situations where two inputs are comparatively similar, e.g., image and sketch [32], while the strategy of no weight-sharing is better suited for situations where two inputs differ somewhat, e.g., image and text [32]. So, what about the neuromorphic events and color images? In this experiment, we compare two weight-sharing strategies described above. Tab. 2 shows the quantitative evaluation results of our proposed model with the strategy of no weight-sharing. It is easy to see that our ECFL method with all weight-sharing performs better than without any weight-sharing in aligning the feature distributions of two modalities of data. We consider that although the output of event camera is sparse and non-uniform spatiotemporal event signal, it is also a type of vision sensor. After converting the event stream into event images, the two modalities of data can be regarded as similar.

To make full use of the temporal component of the neuromorphic events, we propose a simple but effective approach to model the temporal component of the neuromorphic events, i.e., discretizing each bin into three consecutive parts and treating these parts as different channels in the first convolutional layer. To examine whether temporal component helps the improvement of retrieval performance, in this experiment, we show the quantitative evaluation results of our model with no use of the temporal component. We simply convert each bin into an event image and feed it into the network. Tab. 3 presents the results. We can clearly see that the performance of discarding temporal information is poorer than that of preserving temporal information. We consider supplementary details can be observed along the time axis. By discretizing each fixed time interval into consecutive parts, we can preserve more details and avoid them being offset by each other.

Here, we investigate the contribution of adversarial learning. In our proposed ECFL model, we use a modality discriminator as an adversary of feature generators. We consider that the feature generators learn modality-invariant features by confusing the discriminator while the discriminator aims at maximizing its ability to identify modalities. In order to evaluate the contribution of adversarial learning, we remove the modality discriminator and train the feature extractors and identity classifier in one step. Tab. 4 shows the quantitative evaluation results of our model with no adversarial learning. Obviously, for each event representation method, when the modality discriminator is removed, the retrieval performance is degraded. Therefore, adversarial learning can narrow the modality gap between neuromorphic events and color images effectively.

To verify the performance of our algorithm in real scenes, we also evaluate the proposed ECFL method on EC180 dataset. Considering the subtle differences between the neuromorphic events converted from the static image and the neuromorphic events recorded from the real object, our model may suffer performance degradation when applied to real scenes. To avoid it, we take the model trained on the converted N-UKbench dataset as the preliminary model and use the real EC180 dataset to fine-tune it. We randomly select 144 instances in EC180 as training set and the remaining 36 instances are as test set. Tab. 5 shows the quantitative results after fine-tuning and Fig. 5b shows examples of retrieval results using event frequency as representation. We can find that our model performs well for real scenes. The retrieval performance is somewhat worse than that on converted N-UKbench dataset. Because the color images in EC180 contain significant amounts of backgrounds and are captured from perspectives with large variants.

Fig. 6 shows the visualized results of different layers in feature generators. The visualization is achieved using the method in [17], which can invert the learned representation to reconstruct the image to analyze the visual information contained in it. The retrieval performance is often not determined by all the information of the input, but by the main target area. As the layer goes deeper, for the reconstruction of color images, the cluttered background is eliminated and the spatial structure of object gradually changes. It is proved that our model is able to extract the common features of neuromorphic events and color images at avgpool layer.