Vector-Symbolic Architecture for Event-Based Optical Flow

The second approach is the feature matching method, which calculates optical flow by evaluating the similarity or correlation of feature representations for individual pixels between consecutive event frames in the temporal domain. For instance, the model-based EDFLOW estimates optical flow by applying adaptive block matching [40, 41]. Meanwhile, this approach is frequently employed in the design of learning-based optical flow neural networks that incorporate cost volume modules capable of computing feature similarity or correlation, such as E-RAFT [21] and TMA [61]. In addition, treating auxiliary grayscale images as low-dimensional features, EV-FlowNet engages in self-supervised learning by minimizing the intensity difference between warped images based on the estimated optical flow [64].

The third approach, exclusive to event cameras, is the contrast maximization method. This method maximizes an objective function, often related to contrast, to quantify the alignment of events generated by the same scene edge [53, 16, 17]. The underlying idea is to estimate motion by reconstructing a clear motion-compensated image of the edge patterns that triggered the events. This approach can be applied not only to model-based optical flow estimation [52] but also frequently serves as a loss function for unsupervised and self-supervised optical flow learning [52, 60, 46, 22, 47].

In contrast to prior work, our proposed VSA-based framework for event-based optical flow adopts a classical feature matching approach to offer deeper insights into the problem. This framework is adaptable to both model-based and self-supervised learning methods, akin to the contrast maximization method [16, 52]. Particularly, the self-supervised learning method in the framework can achieve accurate optical flow solely from event-only VSA-based HD feature descriptors, eliminating the need for auxiliary grayscale images.

Accurate representations of feature descriptors that encompass individual pixels and their contextual features are crucial for optical flow estimation based on the feature matching method. In contrast to prior work, we adopt a specific type of VSA, Vector Function Architecture (VFA), which embodies continuous similarity characteristics to reduce the impact of randomness in events. This specific VSA is employed as a HD kernel to extract localized feature information from event frames. Meanwhile, optical flow estimation models commonly incorporate a multi-scale pyramid design to enhance their performance. Utilizing the binding capacity of structured features in VSA, we amalgamate HD feature representations from multiple scales and two event polarities into a unified feature descriptor.

VSAs constitute a family of computational models with vector representations that have two distinct properties [33, 14]. Firstly, symbols are represented by mutually orthogonal randomized $d$-dimensional vectors ($\in\mathbb{R}^{d}$), which facilitates a clear distinction between different symbols. Secondly, all computations within VSAs can be composed by a limited set of elementary vector algebraic operations, where the primary operations are the binding ($\circ$) and superposition ($+$) operations. The binding operation commonly signifies associations between symbols, such as a roll-filler pair [27], while the superposition operation is frequently used to represent sets of symbols. Both operations do not change the hypervector dimensionality. Through the combination of these operations and symbols, VSAs can effectively achieve structured symbolic representations. For instance, consider a scenario in which the character $\boldsymbol{1}$ is located at position $\boldsymbol{P_{A}}$ and $\boldsymbol{2}$ at position $\boldsymbol{P_{B}}$ in a given image. The hypervector symbolic representation of this image can be denoted as $\boldsymbol{I}=\boldsymbol{P_{A}}\circ\boldsymbol{One}+\boldsymbol{P_{B}}\circ\boldsymbol{Two}$, where $\boldsymbol{P_{A}}$, $\boldsymbol{P_{B}}$, $\boldsymbol{One}$ and $\boldsymbol{Two}$ $\in\mathbb{R}^{d}$ represent mutually orthogonal randomized hypervectors of corresponding concepts.

VSAs have various models that use different types of random vectors [33]. In this study, an improved Holographic Reduced Representation (HRR) is employed as the VSA model to ensure high concept retrieval efficacy [19]. For HRR, the binding operation is the circular convolution of two hypervectors, and the superposition operation the component-wise sum. Additionally, the similarity between two HRRs can be measured through the cosine similarity.

In this work, the feature extraction from event frames requires the VSA-based 2-D spatial representation. Here, we first introduce the fractional power encoding (FPE) method [48, 49] for representing integers along each coordinate axis in an image plane, and then the VSA-based spatial representation.

Knowing the per-pixel optical flow $\boldsymbol{u}\left(\boldsymbol{x}\right)\in U$, the matching point at time $\frac{k}{K}\varDelta t$ can be obtained through:

However, the matching point $\boldsymbol{x}_{i}^{\prime}$ may not correspond to an actual pixel. Thus, the similarity between HD feature descriptors of $\boldsymbol{x}_{i}$ in $F^{0}$ and the matching point $\boldsymbol{x}_{i}^{\prime}$ in $F^{k}$ is calculated by evaluating its similarity to the descriptors of $4$ neighboring pixels ${\boldsymbol{x}_{i}^{j}}^{\prime}$ ($j=0,...,3$) around the matching point $\boldsymbol{x}_{i}^{\prime}$ in $F^{k}$, with normalized weights ${w_{i}^{j}}^{\prime}$ ($j=0,...,3$) via bilinear interpolation (Figure 3b):

In this study, we use the similarity maximization proxy loss for feature matching to learn to estimate the event-based optical flow, as outlined in Equation 16. Building upon the principles of previous unsupervised learning methods that emphasize contrast maximization [52, 47], we build on the loss function $\mathcal{L}$ as follows:

which is a weighted combination of two terms: the similarity loss $\mathcal{L}_{similarity}$ and the smoothness $\mathcal{L}_{smooth}$. The computation of the similarity loss involves $N$ pixels, encompassing the most recent events occurring before time $0$, as well as pixels sampled at every 5th interval both horizontally and vertically across the image plane (Figure 3a and 3b). The formulation of the similarity loss is as follows:

Here, $\left<\mathrm{sim}\right>$ represents the average similarity encompassing all relevant pixels within $K$ pairs of descriptors, while $\alpha$ serves as a coefficient. A higher value of $\left<\mathrm{sim}\right>$ corresponds to more accurate optical flow estimation and a diminished similarity loss function. Additionally, the smoothness $\mathcal{L}_{\text{smooth}}$ adopts the Charbonnier smoothness prior [22, 65] or the first order edge-aware smoothness [55].

In this study, we train E-RAFT [21] in a self-supervised manner, utilizing the loss function described in Equation 17, to demonstrate the effectiveness of our self-supervised learning method based on similarity maximization of HD feature descriptors. In principle, this methodology holds applicability across various event-based optical flow networks. Meanwhile, we adopt the full-image warping technique [55] to improve flow quality near image boundaries.

Following previous works [21, 52], both VSA-Flow and VSA-SM are evaluated using well-established event-based datasets DSEC-Flow ($640\times 480$ pixel resolution) [21] and MVSEC ($346\times 260$ pixel resolution) [64].

For the model-based method (VSA-Flow), experiments are conducted on the official testing set of the public DSEC-Flow benchmark and on outdoor_day1 and three indoor_flying sequences with time intervals of $dt=1,4$ gray images on the MVSEC benchmark. For the self-supervised learning method (VSA-SM), E-RAFT is trained on the official training set of DSEC and on outdoor_day2 sequence on MVSEC, respectively. To increase the variation in the optical flow magnitude during training, the training sequences on MVSEC are extended with time intervals of $dt=0.5,1,2,4,8$ gray images. Following separate training, evaluations are performed on the same testing sets as VSA-Flow, respectively on DSEC and MVSEC. Both methods are implemented using Pytorch library. For the VSA-SM training, we set batch size to $1$, the optimizer is set to Adam [32] and learning rate to $1e{-2}$.

We evaluate the accuracy of our predictions using following metrics: (i) EPE, the endpoint error; (ii) $\%_{1\mathrm{PE}}$ and $\%_{3\mathrm{PE}}$, the percentage of points with EPE greater than $1$ and $3$ pixels; (iii) AE, angular error. For both DSEC-Flow [20, 21] and MVSEC [64] datasets, metrics are measured over pixels with valid ground-truth and at least one event in the evaluation intervals.

In this study, HD feature descriptors are derived from feature extractors utilizing VSA-based HD kernels. We explore the impact of different VSA types (basic VSA and VFA) on the descriptor similarity among flow-matching points within the DSEC and MVSEC datasets (Figure 4).

In the basic VSA HD kernel, all hypervectors are pseudo-orthogonal, implying that each pixel within the neighborhoods contributes independently to the feature descriptor. Feature descriptors obtained from the basic VSA HD kernel reflect the most fundamental image patterns. Hence, Figure 4 (blue curves) reveals that the similarity of flow-matching points in the MVSEC dataset is inferior to that in the DSEC dataset. This observation suggests that, in comparison to the DSEC dataset, the MVSEC dataset experiences greater randomness in event frames, leading to lower event frame quality.

Figure 4 (red curves) illustrates that VFA yields higher descriptor similarity for flow-matching points compared to basic VSA. In contrast to basic VSA, VFA exhibits an improved ability to encode the similarity of flow-matching points in event frames.

Table 1 presents the evaluation results on the DSEC-Flow benchmark [21]. The methods listed in different rows are classified into three types: model-based methods (MB), supervised learning methods (SL), and self-supervised learning methods (SSL). The notations ‘VFA’ and ‘Basic VSA’ within the parentheses for our methods represent the utilization of the VFA (Eq. 6) and the basic VSA (Eq. 2) HD kernels for feature descriptors. It’s important to note that the stochastic nature of generating spatial base vectors for the HD kernel impacts the evaluation of the VSA-Flow method, all evaluation metrics for the VSA-Flow method represent the statistical outcomes obtained from randomly producing 10 sets of HD kernels. This includes the mean and standard deviation for each metric. Regarding the VSA-SM method, due to its prolonged training duration, Table 1 showcases the evaluation results based on a single set of randomly generated HD kernel used during training.

The VSA-Flow (VFA) method provides the superior performance among all model-based methods in the DSEC-Flow dataset. In particular, the EPE and 3PE metrics slightly outperform other methods, whereas the 1PE and AE metrics display substantial improvements. Moreover, it is evident that employing VFA as the HD kernel in VSA-Flow leads to a significant performance improvement compared to utilizing the basic VSA, which is consistent with observations in Figure 4. In self-supervised training group, the proposed VSA-SM (VFA) method demonstrates the best results among all self-supervised learning methods. The extent of its improvement across the metrics aligns with the evaluation outcomes of the VSA-Flow (VFA).

Table 2 reports the evaluation results on the MVSEC benchmark [64]. Due to the small deviation of all metrics when $d=1024$ (Table 3), for the sake of simplicity, the evaluation results on MVSEC for our methods come from a single set of randomly generated HD kernel. Consistent with [64] and [52], Table 2 compares some primary methods using the same training and testing sequences. Many learning-based methods trained on alternate outdoor sequences or datasets are not used for testing.

The VSA-Flow method achieves the best results among all methods in indoor_flying sequences when $dt=4$ and the competitive results when $dt=1$. These results indicate that the model-based VSA-Flow method, based on HD feature descriptors, are well-suited for large optical flow estimation ($dt=4$) and maintain competitiveness for low optical flow ($dt=1$). In addition, in comparison to the indoor_flying sequences, the performance of VSA-Flow is less competitive in the outdoor_day sequence. This discrepancy may primarily stem from the fact that, compared to the indoor_flying scene, the smaller motion in the outdoor_day scene leads to sparser events [65], thereby impacting the representation of HD feature descriptors.

As mentioned earlier, the training sequences for VSA-SM on MVSEC are extended with time intervals of $dt=0.5,1,2,4,8$ grayscale images. Because VSA-Flow exhibits relatively weaker performance for low optical flow ($dt=1$, Table 3) compared to large optical flow ($dt=4$, Table 3), in the training strategy for VSA-SM, the optical flow predictions at time intervals $dt=0.5,1,2$ are scaled by factors of $8$, $4$, and $2$, respectively. Subsequently, self-supervised learning when $dt=0.5,1,2$ is conducted using high-dimensional feature descriptors of events frames for $dt=4$. Evaluation results indicate that the VSA-SM method achieves competitive performance compared to other self-supervised learning methods. Furthermore, it outperforms some semi-supervised learning methods that employ grayscale images for supervision, particularly on certain sequences.

It is noteworthy that many learning methods, including VSA-SM, exhibit lower performance in the indoor scenes compared to model-based methods. This discrepancy arises because training for MVSEC is exclusively conducted on the outdoor_day2 sequence, but indoor and outdoor sequences contain distinct scene information.

Qualitative results of both VSA-Flow and VSA-SM methods on multiple sequences from the test partition of DSEC-Flow dataset are shown in Figure 5. Given the unavailability of ground truth for the official testing set, a comparison with the state-of-the-art E-RAFT architecture [21] is performed. Our model-based and self-supervised learning methods can achieve high-quality event-based optical flow estimation from events without the need for additional sensory information. Several conclusions can be drawn from these results: (1) Both VSA-Flow and VSA-SM accurately estimate optical flow, particularly in regions containing events. Event-masked sparse optical flow estimation appears more accurate than dense flow estimation. (2) The optical flow estimation from VSA-SM appears smoother compared to VSA-Flow; (3) VSA-Flow exhibits inaccuracies in optical flow estimation near image boundaries, whereas the adoption of a full-image warping technique [55] for VSA-SM during self-supervised learning enhances its accuracy near image boundaries; (4) Due to both methods relying solely on event frames for flow estimation, accuracy diminishes in large areas devoid of events, sometimes resulting in zero flow estimation - a trend consistent with other self-supervised learning methods [22, 47]; (5) As model-based and self-supervised learning approaches relying on event-only local features, our methods predict optical flow less smoothly compared to supervised learning methods; Meanwhile, our methods exhibit less sharp optical flow estimation at the edges of objects, displaying a smoother transition.

Table 3 reports evaluation results for experiments on the VSA-Flow method with varying hypervector dimensions ($d$) and different multi-scale numbers ($S$) for the HD feature descriptor. When $d=1024$, the VSA-Flow exhibits better EPE and 3PE metrics with $S=1$, while for $S=2$, it demonstrates better 1PE and AE metrics. Moreover, as $S$ remains constant and $d$ is altered, all metrics are improved with $d$, indicating that a larger hypervector dimension leads to enhanced performance. This result is consistent with the understanding that, within VSA, increased hypervector dimensions contribute to heightened information encoding capabilities [33, 34].

The temporal information of the HD feature descriptor is primarily impacted by the exponential-decay rate $\tau_{TS}$ of the accumulative Time Surface (TS). Figure 6 illustrates the metrics EPE and 3PE for a single trial of the VSA-Flow method on DSEC and MVSEC datasets with varying $\tau_{TS}$. Both metrics exhibit a trend of initially decreasing and then increasing with $\tau_{TS}$. These results indicate that the performance of VSA-Flow’s optical flow estimation diminishes when $\tau_{TS}$ is either too small or too large. Optimal performance is observed within a suitable range of $\tau_{TS}$. It is because short $\tau_{TS}$ for TS emphasizes recent events, resulting in its sparsity and inadequacy. Conversely, an excessively long $\tau_{TS}$ causes TS to encompass events over an extended period, leading to a blurred representation. Hence, an appropriate $\tau_{TS}$ is essential. It is worthy noting that the optimal range for $\tau_{TS}$ in the VSA-Flow method differs between DSEC and MVSEC due to variations in the characteristics of the event cameras used. In contrast to DSEC, events in MVSEC are more sparse, requiring a larger $\tau_{TS}$. This indicates the necessity to accumulate events over a longer period for MVSEC to achieve more accurate information encoding in the TS.