Learning Local Implicit Fourier Representation
for Image Warping

In local neural representation [22, 10], a neural representation $g_{\theta}$ is parameterized by an MLP with trainable parameters $\theta$. A decoding function $g_{\theta}$ predicts RGB value for a query point $\mathbf{y}=f(\mathbf{x})\in Y\subseteq\mathbb{R}^{2}$ as

$\Theta=[\theta,\varphi]$, $\mathcal{J}$ is a set $\{j|j=[f^{-1}(\mathbf{y})+[\frac{m}{w},\frac{n}{h}],[m,n]\in[-1,1]\}$ [10, 21], $w_{j}$ is a local ensemble coefficient [22, 10, 27], $\mathbf{z}_{j}\in\mathbb{R}^{C}$ indicates a latent variable for a index $j$, and $\mathbf{x}_{j}\in X\subseteq\mathbb{R}^{2}$ is a coordinate of $\mathbf{z}_{j}$.

Recent works [37, 44] have shown that a standard MLP structure suffers from learning high-frequency content. Lee et al. [27] modified the local neural representation in Eq. (1) to overcome this spectral bias problem as

where $h_{\psi}$ is a Local Texture Estimator (LTE). LTE ($h_{\psi}(\cdot)$) contains two estimators;(1) an amplitude estimator ($h_{a}(\cdot):\mathbb{R}^{C}\mapsto\mathbb{R}^{2D}$) (2) a frequency estimator ($h_{f}(\cdot):\mathbb{R}^{C}\mapsto\mathbb{R}^{2\times D}$). Specifically, an estimating function $h_{\psi}(\cdot,\cdot):(\mathbb{R}^{C},\mathbb{R}^{2})\mapsto\mathbb{R}^{2D}$ is defined as

$\boldsymbol{\delta}_{\mathbf{y}}=\mathbf{y}-f(\mathbf{x}_{j})$ is a local grid, $\mathbf{A}_{j}\in\mathbb{R}^{2D}$ is an amplitude vector, $\mathbf{F}_{j}\in\mathbb{R}^{2\times D}$ indicates a frequency matrix for an index $j$, $<\cdot,\cdot>$ is an inner product, and $\odot$ denotes element-wise multiplication. However, this formulation fails to represent warped images since $\mathbf{F}_{j}$ is a frequency response of an input image $\mathbf{I^{IN}}$, which is different from that of a warped image $\mathbf{I^{WARP}}$ [6]. In the following, we generalize LTE by considering a spatially-varying property of coordinate transformations.

We linearize the given coordinate transformation $f$ into affine transformations. A linear approximation of the local grid $\boldsymbol{\delta}_{\mathbf{y}}$ near a point $\mathbf{x}_{j}$ is computed as

where $\mathbf{J}_{f}(\mathbf{x}_{j})\in\mathbb{R}^{2\times 2}$ is the Jacobian matrix of coordinate transformation $f$ at $\mathbf{x}_{j}$, $\mathcal{O}(\mathbf{x}^{2})$ means terms of order $\mathbf{x}^{2}$ and higher, and $\boldsymbol{\delta}_{\mathbf{x}}=\mathbf{x}-\mathbf{x}_{j}$ is a local grid in input space $X$. By the affine theorem [6] and Eq. (6), a frequency response $\mathbf{F}^{\prime}_{j}$ of a warped image $\mathbf{I^{WARP}}$ near a point $f(\mathbf{x}_{j})$ is approximated as follows:

When comparing Eq. (4) and Eq. (11), we see that LTEW representation is capable of extracting Fourier information for warped images by utilizing the local grid in input coordinate space $\boldsymbol{\delta}_{\mathbf{x}}$ instead of $\boldsymbol{\delta}_{\mathbf{y}}$.

For SISR tasks within symmetric scale factors, the pixel shape of upsampled images is square and spatially invariant. However, when it comes to image warping, pixels in resampled images are able to have arbitrary shapes and spatially-varying, as described in Fig. 3.(a). To address this issue, we represent the pixel shape $\mathbf{s}(\mathbf{y})\in\mathbb{R}^{12}$ ($\mathbb{R}^{4}$ is for pixel orientation, and $\mathbb{R}^{8}$ is for pixel curvature) with a gradient of coordinate transformation for a point $\mathbf{y}$ as

where $[\cdot,\cdot]$ refers to the concatenation after flattening, $\tilde{\mathbf{J}}_{f^{-1}}({\mathbf{y}})\in\mathbb{R}^{2\times 2}$ and $\tilde{\mathbf{H}}_{f^{-1}}({\mathbf{y}})\in\mathbb{R}^{2\times 2\times 2}$ denote the numerical Jacobian matrix, indicating an orientation of pixel, and the numerical Hessian tensor, specifying the degree of curvature, respectively. For shape representation, we apply an inverse coordinate transformation to a query point and its eight nearest points ($\mathbf{y}+[\frac{m}{W},\frac{n}{H}]$ with $[m,n]\in[-1,0,1]$) and compute the difference to calculate numerical derivatives as described in Fig. 3.(b). Let us assume that given coordinate transformation $f$ is in class $C^{2}$, which means $f_{x_{1}x_{2}}=f_{x_{2}x_{1}}$. Hence; we use only six elements in $\tilde{\mathbf{H}}_{f^{-1}}(\mathbf{y})$ for shape representation.

Phase in Eq. (13) includes the information of edge locations or the shape of pixels [27]. Therefore, we redefine the estimating function in Eq. (11) as:

where $h_{p}(\cdot):\mathbb{R}^{10}\mapsto\mathbb{R}^{D}$ is a phase estimator.

Lastly, we add a bilinear interpolated image $\mathbf{I}_{\boldsymbol{B}}[\mathbf{y}]=\boldsymbol{f}_{\boldsymbol{B}}(\mathbf{I^{IN}})\simeq\mathbf{I^{IN}}(f^{-1}(\mathbf{y}))$ to stabilize network convergence and aid LTEW in learning high-frequency details [25]. Thus, the local neural representation of a warped image $\mathbf{I^{WARP}}$ with the proposed LTEW is formulated as follows:

Our LTEW-based image warping network consists of an encoder ($E_{\varphi}$), the LTEW ($h_{\psi}$, a purple shaded region in Fig. 2), and a decoder ($g_{\theta}$). An encoder ($E_{\varphi}$) is designed with a deep SR network, such as EDSR [30], RCAN [51], RRDB [48], without upscaling modules. A decoder ($g_{\theta}$) is a 4-layer MLP with ReLUs, and its hidden dimension is 256. Our LTEW ($h_{\psi}$) takes a local grid ($\boldsymbol{\delta}_{\mathbf{x}}$), shape ($\mathbf{s}$), and feature map ($\mathbf{z}$) as input, and includes an amplitude estimator ($h_{a}$), a frequency estimator ($h_{f}$), and a phase estimator ($h_{p}$). An amplitude and a frequency estimator are implemented with $3\times 3$ convolution layer having 256 channels, and a phase estimator is a single linear layer with 128 channels. We assume that a warped image has the same texture near point $f(\mathbf{x}_{j})$. Hence, we find estimated Fourier information ($\boldsymbol{A}_{j},\boldsymbol{F}_{j}$) at $\mathbf{x}_{j}$ using the nearest-neighborhood interpolation. Then, estimated phase is added to an inner product between the local grid ($\boldsymbol{\delta}_{\mathbf{x}}$) and estimated frequencies, as in Eq. (13). Before the decoder ($g_{\theta}$) resamples images, we multiply amplitude and sinusoidal activation output.

We have two batches with a size $\boldsymbol{B}$: (1) Image batch $\{I_{\boldsymbol{1}},I_{\boldsymbol{2}},\dots,I_{\boldsymbol{B}}\}$, $I_{\boldsymbol{i}}[Y]\in\mathbb{R}^{H\times W\times 3}$. (2) Coordinate transformation batch $\{f_{\boldsymbol{1}},f_{\boldsymbol{2}},\dots,f_{\boldsymbol{B}}\}$, where each $f_{\boldsymbol{i}}:X\mapsto Y$ is differentiable and invertible. For an input image preparation, we first apply an inverse coordinate transformation as:

where $I_{\boldsymbol{i}}[f_{\boldsymbol{i}}^{-1}(Y)]\in\mathbb{R}^{h_{i}\times w_{i}\times 3}$. While avoiding void pixels, we crop input images $I_{\boldsymbol{i}}^{crop}[f_{\boldsymbol{i}}^{-1}(Y)]\in\mathbb{R}^{h\times w\times 3}$, where $h\leq\min(\{h_{\boldsymbol{i}}\}),w\leq\min(\{w_{\boldsymbol{i}}\})$. For a ground truth (GT) preparation, we randomly sample $\boldsymbol{M}$ query points among valid coordinates for an $\boldsymbol{i}$-th batch element as: $\{\mathbf{y}^{\boldsymbol{i}}_{\boldsymbol{1}},\mathbf{y}^{\boldsymbol{i}}_{\boldsymbol{2}},\dots,\mathbf{y}^{\boldsymbol{i}}_{\boldsymbol{M}}\}$, where $\mathbf{y}^{\boldsymbol{i}}_{\boldsymbol{j}}\in\mathbb{R}^{2}$. Then, we evaluate our LTEW for each query point with cropped input images to compute loss as in Fig. 4 by comparing with following GT batch:

where $I_{\boldsymbol{i}}[\mathbf{y}^{\boldsymbol{i}}_{\boldsymbol{j}}]$ is an RGB value.

For asymmetric-scale SR, each scale factor $s_{x},s_{y}$ is randomly sampled from $\mathcal{U}(0.25,1)$. For homography transform, we randomly sample inverse coordinate transformation from the following distribution, dubbed in-scale:

where $h_{x},h_{y}\sim\mathcal{U}(-0.25,0.25)$ are for sheering, $\theta\sim\mathcal{N}(0,0.15^{{\circ}^{2}})$ is for rotation, $s_{x},s_{y}\sim\mathcal{U}(0.35,0.5)$ are for scaling, $t_{x}\sim\mathcal{U}(-0.75W,0.125W)$, $t_{y}\sim\mathcal{U}(-0.75H,0.125H)$, $p_{x}\sim\mathcal{U}(-0.6W,0.6W)$, $p_{y}\sim\mathcal{U}(-0.6H,0.6H)$ are for projection. We evaluate our LTEW for unseen transformations to verify the generalization ability. For asymmetric-scale SR and homography transform, untrained coordinate transformations are sampled from $s_{x},s_{y}\sim\mathcal{U}(0.125,0.25)$, dubbed out-of-scale, other parameter distributions for $h_{x},h_{y},\theta,t_{x},t_{y},p_{x},p_{y}$ remain the same.

We use a DIV2K dataset [1] of an NTIRE 2017 challenge [45] for training. For optimization, we use an L1 loss [30] and an Adam [26] with $\beta_{1}=0.9$, $\beta_{2}=0.999$. Networks are trained for 1000 epochs with batch size 16. The learning rate is initialized as 1e-4 and reduced by 0.5 at [200, 400, 600, 800]. Due to the page limit, evaluation details are provided in the supplementary.

Asymmetric-scale SR We compare our LTEW for asymmetric-scale SR to RCAN [51], MetaSR [19], ArbSR [46] within in-scale in Table 1, Fig. 5 and out-of-scale in Table 2, Fig. 6. For RCAN [51], we first upsample LR images by a factor of 4 and resample using bicubic interpolation. For MetaSR [19], we first upsample input images by a factor of $\max(s_{x},s_{y})$ and downsample using a bicubic method as in [46]. Except for the case of Set14 $(\times 3.5,\times 2)$ and B100 $(\times 1.5,\times 3)$, LTEW outperforms existing methods within asymmetric scale factors in performance and visual quality for all scale factors and all datasets.

In Fig. 10, we project an HR ERP image with size $8190\times 2529$ to $4095\times 4095$ with a FOV $90^{\circ}$ on the Intel Xeon Gold [email protected] and RAM 512GB. Note that RRDB [48] and SRWarp [42] are not able to evaluate given HR ERP size even under CPU-computing. Both RRDB and SRWarp need to upsample an ERP image by a factor of 4, consuming a massive amount of memory. In contrast, our LTEW is memory-efficient since we are able to query evaluation points sequentially. We notice that our LTEW is capable of restoring sharper and clearer edges compared to bicubic interpolation.

In rows 6-8, we remove an amplitude estimator (row 6), long skip connection (row 7), and reduce the number of estimated frequencies (row 8). We see that each component consistently enhances mPSNR of LTEW for both in-scale and out-of-scale. In row 9, we test LTEW with spatially-invariant cell as [27], specifically $r_{\mathbf{x}}=2w/W,r_{\mathbf{y}}=2h/H$. It causes a significant mPSNR drop for both in-scale and out-of-scale. From rows 10-12, we observe that both the Jacobian and Hessian shapes are significant in improving mPSNR only for in-scale. Inspired by [49], we hypothesize that INR performs superior interpolating unseen coordinates but relatively poorly extrapolating untrained shapes. Extrapolation for untrained shapes will be investigated in future work.

In Fig. 11, we visualize estimated Fourier feature space from LTEW and the discrete Fourier transform (DFT) of a warped image for validation. We first scatter estimated frequencies to 2D space and set the color according to magnitude. We observe that our LTEW ($h_{\psi}$) extracts Fourier information ($\mathbf{F}_{j}$) for an input image at $\mathbf{x}_{j}$ ($\mathbf{I^{IN}}[\mathbf{x}_{j}]$). By observing pixels inside an encoder’s receptive field (RF), LTEW estimates dominant frequencies and corresponding Fourier coefficients for RF-sized local patches of an input image. Before our MLP ($g_{\theta}$) represents $\mathbf{I^{WARP}}[f(\mathbf{x}_{j})]$, the Fourier space ($\mathbf{F}_{j}$) is transformed by the Jacobian matrix $\mathbf{J}_{f}^{-T}(\mathbf{x}_{j})$ and matched to a frequency response of an output image at $f(\mathbf{x}_{j})$ ($\mathcal{F}(\mathbf{I^{WARP}}[f(\mathbf{x}_{j})])$). We observe that LTEW utilizing the local grid in output space ($\mathbf{\delta_{y}}$) instead of $\mathbf{\delta_{x}}$ diverges during training. We hypothesize that representing $\mathbf{I^{WARP}}[f(\mathbf{x}_{j})]$ with frequencies ($\mathbf{F}_{j}$) of an input image makes the overall training procedure unstable. This indicates that a spatially-varying Jacobian matrix $\mathbf{J}_{f}^{-T}(\cdot)$ for the given coordinate transformation $f$ is significant in predicting accurate frequency responses of warped images. By estimating Fourier response in latent space instead of directly applying DFT to input images, we avoid extracting undesirable frequencies due to aliasing, as discussed in [27].

In Table 5, we compare model complexity and symmetric-scale SR performance of our LTEW for both in-scale and out-of-scale to other warping methods: ArbSR [46] and SRWarp [42]. Note that ArbSR [46] and SRWarp [42] are learned to perform asymmetric-scale SR and homography transform. Following [27], we use $\max(\mathbf{s},\mathbf{s}_{tr})$ instead of $\mathbf{s}$ in Eq. (13) for phase estimation. We see that LTEW significantly outperforms exiting warping methods for out-of-scale, achieving competitive quality to [46, 42] for in-scale. A local ensemble [22, 10, 27] in LTEW, preventing blocky artifacts, makes the model more complex than ArbSR [46]. SRWarp [42] blends $\times 1$, $\times 2$, and $\times 4$ features, leading to increased model complexity than LTEW, which uses only an $\times 1$ feature map.