Super-Resolving Blurry Images with Events

Due to the imperfection of image sensors, the captured image $B$ may suffer from non-negligible quality degeneration including motion blur and low spatial resolution, which can be related to the high-quality (sharp and high spatial resolution) latent image $L$ as follows:

where $\mathcal{T}\triangleq[0,T]$ denotes the exposure interval of $B$, $T$ is the duration of $\mathcal{T}$, and $D^{\downarrow}(\cdot)$ represents the down-sampled operator to obtain Low Resolution (LR) sharp image $I$ from $L$. Thus the Motion-blurred image Super Resolution (MSR) can be formulated as:

It is obvious that the task of recovering a High-Resolution (HR) sharp image $L$ from a single LR blurry image $B$ poses a severe ill-posed problem. While significant progress has been reported in MSR techniques [13, 18, 23, 23], these approaches often are specialized for specific domains (e.g., face or text) or rely heavily on the strong assumption of uniform motion, thereby limiting their applicability in natural scenarios with complex motions.

Recently, many methods [26, 28, 27] utilize events to tackle MSR task due to their low latency. These events are triggered whenever the log-scale brightness change exceeds the event threshold $c>0$, i.e.,

where $I(t,\mathbf{x})$ and $I(\tau,\mathbf{x})$ denote the instantaneous intensity at time $t$ and $\tau$ at the pixel position $\mathbf{x}$, and polarity $p\in\{+1,-1\}$ indicates the direction of brightness changes. Hence, the Event-based MSR (E-MSR) task can be represented as:

where $\mathcal{E}_{\mathcal{T}}\triangleq{(\mathbf{x}_{i},p_{i},t_{i})}_{t_{i}\in\mathcal{T}}$ denotes the emitted event stream during the exposure interval of $B$. However, existing approaches [28, 27] often suffer performance degradation on more complex scenarios owing to the limitations imposed by sparse coding [28], and accumulated errors in the multi-stage training procedure [27]. To address these problems, we introduce a novel Event-based Blurry Super Resolution Network (EBSR-Net), a one-stage architecture aimed at directly recovering $L$ from $B$ and concurrent event stream $\mathcal{E}_{\mathcal{T}}$ across various challenging scenarios.

The overall architecture of our proposed Event-based Blurry Super Resolution Network (EBSR-Net) is illustrated in Fig. 2 (a), which is a deep network mainly consisting of a Multi-scale Center-surround Event Representation (MCER) module, a Symmetric Cross-Modal Attention (SCMA) module, and an Intermodal Residual Group (IRG) module.

Unlike conventional self-attention blocks that typically compute queries, keys, and values exclusively from either the frame or event branch of the network, our SCMA leverages multimodal information between frames and events. Queries are calculated from both images and events, with keys and values obtained from the opposite modality. SCMA consists of two parallel self-attention structures and a combination operation, formulated as:

where $(\cdot)^{{}^{\prime}}$ represents the transpose operator. Note that $Q$, $K$, and $V$, are produced through operations involving normalization and 1$\times$1 convolutional layers. Specifically, $Q_{b}$ and $Q_{e}$ are derived from $F_{b}$ and $F_{e}$ respectively, similar to $K$ and $V$. Additionally, the adaptively symmetric fusion-based output $F_{s}$ can be calculated by

where $\tilde{F}_{b\rightarrow e}$ and $\tilde{F}_{e\rightarrow b}$ represent intermediate features from the $Q_{b}$ and $Q_{e}$ branches, respectively. These features are generated by reshaping the results that combine attention and original features followed by a Multi-Layer Perceptron (MLP) layer.

The final result $F_{i}$ of the IRG module is computed using three additional RDSTBs and two 3$\times$3 convolutional layers. Consequently, the recovery HR sharp image $\bar{L}$ can be estimated by

where $\operatorname{Dec}$ denotes a decoder operator, and $\operatorname{BL}$ represents the bilinear upsampling operation.

We supervise the overall architecture by using $L_{1}$ loss and perceptual similarity loss $\mathcal{L}_{per}$ [32] for better visual quality, which can be formulated as

where $\alpha$ and $\beta$ are the balancing parameters.

The proposed EBSR-Net model, implemented using PyTorch on an NVIDIA GeForce RTX 3090, is trained separately on the training sets of both the GoPro [33] and REDS [34] datasets, and we use simulated events and blurry image [29]. Subsequently, evaluations are conducted separately on the testing sets of these two datasets. Furthermore, we use the ADAM optimizer [35] with an initial learning rate of $10^{-4}$, and the exponential term decays by $0.98$ for every $5$ epoch. The weighting factors $\alpha$ and $\beta$ in Eq. 12 are set to 1 and 0.1, respectively. We use Structural SIMilarity (SSIM) [36] and Peak Signal to Noise Ratio (PSNR) as performance metrics.

We compare our EBSR-Net to the state-of-the-art Motion Deblurring (MD) methods including MPR [37], RED [29], and EF [38], and Image Super-Resolution (ISR) approaches, including SwIR [6], CAT [7], and DAT [8], on both GoPro [33] and REDS [34] datasets. These comparison methods are categorized into two stages: MD methods are evaluated first, followed by the ISR methods, denoted as MPR+DAT, and so on. Additionally, a one-stage architecture, e.g., eSL-Net [26], is directly utilized for comparison by utilizing its official code with default parameters. According to the quantitative results shown in Tab. I, our EFSR-Net outperforms state-of-the-art methods by a large margin, achieving an average improvement of 4.12/5.70 dB and 0.0788/0.1607 in PSNR and SSIM on the GoPro and REDS datasets respectively. Meanwhile, our one-stage model only contains 7.3M network parameters, which is much smaller than other methods except eSL-Net. Note that eSL-Net requires 122.8G FLOPs to infer a 160$\times$320 image due to its recursive structure, while our model only needs 41.2G FLOPs, maintaining the overall efficiency.

The qualitative comparisons are shown in Fig. 3. The results estimated by the two-stage cascade architecture, e.g., RED [29]+DAT [8], suffer artifacts and distortions owing to the accumulated errors, leading to significant degradation of the overall quality. Furthermore, one-stage methods like eSL-Net [26] exhibit performance degradation in complex scenarios, attributed to limitations imposed by sparse coding. In contrast, our EBSR-Net achieves accurate reconstructions closely resembling the HR ground-truth sharp images.

In order to verify the effectiveness of the key components in our EBSR-Net, ablation experiments of the MCER, SCMA, and IRG modules are conducted, as shown in Tab. II. All models are trained on the same experimental environment and equipment, and we replace the modules with corresponding convolutional layers for a fair comparison. Specifically, removal of the MCER (Case 1, 2, 5), SCMA (Case 0, 2, 4), and IRD (Case 0, 1, 3) modules respectively lead to a degradation of 3.12/3.28/2.91 dB in PSRN and 0.0795/0.0858/0.0722 in SSIM. Furthermore, comparing (b) with (e) and (f) with (h), (c) with (f) and (g) with (h), and (d) with (g) and (e) with (h) demonstrate that EBSR-Net with the SCMA, IRG, and MCER modules respectively give sharper results than network without them. The improvement in both quantitative and qualitative results validates the effectiveness of the proposed three modules.