Local Texture Estimator for Implicit Representation Function

Single image super-resolution (SISR) is one of the most fundamental problems in computer vision and graphics. SISR aims to reconstruct high-resolution (HR) images from its degraded low-resolution (LR) counterpart. Dominant approaches [24, 15, 34, 35, 5, 18, 3, 14] are to extract feature maps using a deep vision architecture and then upsample to HR images at the end of a network. However, we need to train and store several models for each scale factor when an upsampler is implemented by sub-pixel convolution [24]. In contrast, arbitrary-scale SR methods [9, 4] are promising since such ideas pave the way to restore images in a continuous manner with only a single network.

Recently, implicit neural functions parameterized by a multi-layer perceptron (MLP) achieved remarkable performance in representing continuous-domain signals, such as images [4], occupancy [19], signed distance [21], shape representation [11], and view synthesis [26, 20]. Such MLPs take coordinates as inputs and are trained in a framework of gradient descent optimization and machine learning. Inspired by recent progress in implicit representation, LIIF [4] replaced sub-pixel convolution with MLPs to accomplish arbitrary-scale SR, even at substantial scale factors.

One limitation of implicit neural representations is that a standalone MLP is biased towards learning low-frequency components [23] and fails to capture fine details [29]. Such phenomenon is referred to as spectral bias, and recent lines of research in resolving this problem are projecting input coordinates into a high-dimensional Fourier feature space [20, 29] or substituting a ReLU with a sinusoidal activation [25]. Motivated from previous works, we study arbitrary-scale SISR problems through the lens of Fourier analysis.

In this paper, we propose a Local Texture Estimator (LTE), a dominant-frequency estimator for natural images, to allow an implicit function to learn fine details while restoring images in arbitrary resolution. We assume that an implicit function prioritizes learning image textures when utilizing dominant frequencies of images, as described in Fig. 1. Let us take an example from an image with vertical textures. Intuitively, the dominant frequencies of such an image are located on an $\mathbf{x}$-axis in 2D Fourier space. We observe that LTE is capable of extracting such dominant frequencies, characterizing image textures in 2D Fourier space, when jointly trained with a deep SR architecture, such as EDSR [15], RDN [35], and SwinIR [14]. In addition to extracting dominant frequencies, we show that estimating Fourier coefficients is also essential in improving a representational power of an implicit function in Sec. 5.3.

Short computation time is essential in SR applications. In addition, restoring larger than 2K-sized images is a memory-consuming task. Hence, we study the computation time of our LTE at various memory conditions and demonstrate that our approach is faster compared to previous works regardless of memory setting in Sec. 6.

In summary, our main contributions are as follows:

• •

  We point out that a deep SR network followed by LTE is capable of estimating dominant frequencies and corresponding Fourier coefficients for natural images.

• •

  We show that an implicit representation function for arbitrary resolution prioritizes learning high-frequency details when essential Fourier information is estimated by LTE.

• •

  We investigate the computation time of our LTE at several memory settings and demonstrate that our approach is faster compared to previous works.

Implicit neural representation Based on the fact that neural networks are universal function approximators [8], coordinate-based MLP is widely applied to represent continuous-domain signals in computer vision and graphics. Such MLPs are a memory-efficient framework for HR data since the amount of memory to store MLPs is independent of data resolution. However, pre-trained MLPs show limited performance in representing unseen data, requiring training per each signal. To overcome this generalization issue, [26, 25] trained representation with a hypernetwork, which maps latent variables to weights of an MLP as in [7]. In [21, 19], an MLP takes not only coordinates but also latent variables as inputs to enable representation to be a function of data. Recently, meta-learned initialization [28] has been proposed, which provides a strong prior for representing signals, leading to fast convergence. Our work is mainly related to local implicit neural representation [11, 4]. Such approaches assume that latent variables are evenly distributed over space, allowing an implicit function to focus on learning local features. Inspired by this, we hypothesize that the local region in HR representation shares the same image textures.

Spectral bias Recent works [23, 25, 20, 29] have shown that a standard MLP with ReLUs shows limited performance in representing high-frequency textures. Such a phenomenon is referred to as spectral bias, and various methods have been proposed to alleviate this problem. Recently, SIREN [25] used the sine layer as a non-linear activation instead of a ReLU, resulting in fast convergence and high data fidelity. Other approaches [20, 29] are to map input coordinates into high dimensional Fourier space by using position encoding or Fourier feature mapping before passing an MLP. Frequencies are fixed to the power of two [20] or randomly sampled from Gaussian distribution [29]. Unlike previous works, dominant frequencies from our LTE are data-driven and characterize high-frequency textures in 2D Fourier space.

Deep SR architecture After ESPCN [24] has proposed an efficient learnable upsampling module using pixel-shuffling, numerous CNN-based approaches [15, 34, 35, 5, 18] have been studied to improve the representational power of models. Such approaches have exploited more complicated neural network architecture designs, such as residual block [15], densely connected residual block [35], channel attention [34, 5], or non-local neural networks [18]. Inspired by the success of the self-attention mechanism in high-level vision tasks [6, 16], general-purpose image processing transformers, such as IPT [3] or SwinIR [14], have been proposed. Even though transformer-based SR architectures using a large dataset surpass CNN-based architectures in performance, these approaches nevertheless need to utilize a specific upscale module for each upsampling rate.

Arbitrary-scale SR Our work is highly related to SR tasks within an arbitrary-scale factor [9, 27, 31, 4], which is convenient and efficient for practical benefits. Training and storing models for each specific scale factor [24, 15, 35, 34, 3, 14] is unfeasible when considering limited memory resources. Since MetaSR [9] first proposed an arbitrary-scale SR method with a single model, various approaches have been explored. Recently, ArbSR [31] has been proposed as a general plug-in module using conditional convolutions. ArbSR conducts an SR with different scales along horizontal and vertical axes, respectively. More recently, SRWarp [27] successfully transformed an LR image into any shape in HR representation using a differentiable adaptive warping layer. Most related to ours is the model of [4]. Inspired by advancements in implicit neural representation, LIIF replaced sub-pixel convolution with an MLP, taking continuous coordinates and latent variables as inputs. Even though such a method outperformed previous works at large upsampling rates, structural distortion occurs at extreme scales. Instead of concatenating coordinates and latent variables, our LTE-based architecture transforms input coordinates into the Fourier domain using dominant frequencies extracted from latent variables.

In this section, we aim to represent $\mathbf{I^{HR}}\in\mathbb{R}^{r_{\mathbf{y}}H\times r_{\mathbf{x}}W\times 3}$ from $\mathbf{I^{LR}}\in\mathbb{R}^{H\times W\times 3}$ given any fraction number $r_{\mathbf{x}},r_{\mathbf{y}}$. We first review a continuous representation of an RGB image with a local implicit neural representation [11, 4]. Even though such approaches showed outstanding performance in representing continuous-domain signals, a standalone MLP fails to capture high-frequency details [23]. To overcome this spectral bias problem, we formulate a Local Texture Estimator (LTE), a dominant frequency estimator for natural images. Unlike prior arts [20, 29], estimated frequencies are data-driven and strongly correlated to image textures. Additionally, we introduce scale-dependent phase estimation and LR skip connection, which aid LTE in learning high-frequency textures.

Local implicit neural representation In local implicit neural representation [11, 4], a decoding function $f_{\theta}$ is shared by all images and is parameterized by an MLP with trainable weights $\theta$. A decoder $f_{\theta}$ maps both latent tensors and local coordinates into RGB values; $f_{\theta}(\mathbf{z},\mathbf{x}):(\mathcal{Z},\mathcal{X})\mapsto\mathcal{S}$. $\mathbf{z}\in\mathcal{Z}$ is a latent tensor from an encoder $E_{\varphi}$, $\mathbf{x}\in\mathcal{X}$ is a 2D coordinate in the continuous image domain, $\mathcal{S}$ is a space of predicted values from $f_{\theta}$. For simplicity, we assume that a latent tensor $\mathbf{z}\in\mathbb{R}^{H\times W\times C}$ has the same width and height as $\mathbf{I^{LR}}$. Then, predicted RGB values ($\mathbf{s}\in\mathbb{R}^{3}$) at a coordinate $\mathbf{x}\in\mathbb{R}^{2}$ are estimated as

$\Theta=[\vec{\theta};\vec{\varphi}]$, $\mathcal{J}\in\mathbb{Z}^{4}$ is a set of indices for four nearest (Euclidean distance) latent codes around $\mathbf{x}$, $w_{j}$ is the bilinear interpolation weight corresponding to the latent code $j$ (referred to as the local ensemble weight [4], $\sum_{j}w_{j}=1$), $\mathbf{z}_{j}\in\mathbb{R}^{C}$ is the $j-$th nearest latent feature vector from $\mathbf{x}$, and $\mathbf{x}_{j}\in\mathbb{R}^{2}$ is the coordinate of the latent code $j$. Given a series of $M$ data points from $N$ images such as $(\mathbf{x}_{m},\mathbf{I}^{\mathbf{HR}}_{n}(\mathbf{x}_{m}))$, $m=1,\dots,M$ and $n=1,\dots,N$, the learning problem is defined as follows:

In practice, $\mathcal{X}$ spans $[-H,H]$ and $[-W,W]$ for two dimensions. Note that a step size (Cell in Fig. 2) of an output grid ($\mathbb{R}^{r_{\mathbf{y}}H\times r_{\mathbf{x}}W}$) and an input grid ($\mathbb{R}^{H\times W}$) is different. In [11, 4], their decoding function ($f_{\theta}$) predicts continuous representation with a relative coordinate: $\mathbf{x}-\mathbf{x}_{j}$ $(|\mathbf{x}-\mathbf{x}_{j}|\leq\mathbf{1})$ known as Local grid. The same coordinate (Local grid) with [4, 11] is used for our work in order to represent a $r_{\mathbf{x}}\times r_{\mathbf{y}}$ local area in HR representation.

Learning dominant frequency component Recent works have shown that an MLP with ReLUs is biased towards learning low-frequency content [23]. To resolve this spectral bias problem of an implicit neural function, we propose a Local Texture Estimator (LTE), an essential Fourier information estimator for natural images. Inspired by position encoding [20] and Fourier feature mapping [29], LTE transforms input coordinates into the Fourier domain before passing an MLP. However, unlike [20, 29], estimated Fourier information is data-driven and reflects image textures in 2D Fourier space. The local implicit neural representation in Eq. 1 can be modified as follows:

where $h_{\psi}(\cdot,\cdot)$ denotes the LTE, which is shift-invariant. LTE ($h_{\psi}(\cdot,\cdot)$) consists of three elements;(1) an amplitude estimator ($h_{a}(\cdot):\mathbb{R}^{C}\mapsto\mathbb{R}^{2K}$), (2) a frequency estimator ($h_{f}(\cdot):\mathbb{R}^{C}\mapsto\mathbb{R}^{K\times 2}$), (3) a phase estimator ($h_{p}(\cdot):\mathbb{R}^{2}\mapsto\mathbb{R}^{K}$). Thus, given a local-grid coordinate $\boldsymbol{\delta}(=\mathbf{x}-\mathbf{x}_{j})\in\mathbb{R}^{2}$, the estimating function $h_{\psi}(\cdot,\cdot):(\mathbb{R}^{C},\mathbb{R}^{2})\mapsto\mathbb{R}^{2K}$ is defined as

$\mathbf{A}_{j}\in\mathbb{R}^{2K}$ is an amplitude vector for a latent code $j$, $\mathbf{F}_{j}\in\mathbb{R}^{K\times 2}$ denotes a frequency matrix for a latent code $j$, and $\odot$ represents element-wise multiplication. We believe that the amplitude vector and the frequency matrix are extracted from the latent code $j$ so as to represent $\mathbf{s}(\mathbf{x})$ as close as possible to original signals $\mathbf{I}^{\mathbf{HR}}(\mathbf{x})$. In this perspective, we understand that by observing pixels inside a receptive field (RF), LTE with the encoder ($h_{\psi}\circ E_{\varphi}$) estimates dominant frequencies and corresponding Fourier coefficients accurately. Here, the size of RF is decided by the encoder ($E_{\varphi}$). We visually demonstrate estimated frequencies and corresponding Fourier coefficients in Sec. 5.4.

In practice, to enrich the information in outputs of LTE, we apply the unfolding technique to $\mathbf{z}_{j}$ leading to a concatenation of the $3\times 3$ nearest latent variables in $\hat{\mathbf{z}}_{j}$ [4]. This is implemented with trainable convolutional filters ($h_{a}(\cdot):\mathbb{R}^{9C}\mapsto\mathbb{R}^{2K},h_{f}(\cdot):\mathbb{R}^{9C}\mapsto\mathbb{R}^{K\times 2}$).

Scale-dependent phase estimation Phase in Eq. 7 contains information about edge locations of features. For SR tasks, the location of edge changes within a small neighborhood in its HR domain when the scale factor changes. To address this issue, we redefine the estimating function as:

where $\mathbf{c}$ denotes the cell size. Inspired by the observation that MLPs with ReLUs are incapable of extrapolating unseen non-linear space [32], we use $\hat{\mathbf{c}}=\max(\mathbf{c},\mathbf{c}_{tr})$, where $\mathbf{c}_{tr}$ denotes the minimum cell size during training.

LR skip connection A long skip connection in local implicit representation enriches high-frequency components in residuals and stabilizes convergence [12]. In the Fourier domain, LTE tends to predict frequencies located near a low-frequency region (DC). To prevent LTE from learning the DC only, we add upscaled LR. Thus, local implicit neural representation with the proposed LTE can be formulated as follows:

Advantage of LTE over DFT In DFT, Fourier information is represented with a linear combination of image intensity. However, the DFT of an LR image with aliasing (middle) is limited in capturing dominant frequencies. In contrast, LTE with a deep neural encoder (bottom), which is a multi-chain of linear combinations and non-linear activations, is capable of estimating accurate Fourier information for an HR image (top).

Effect of aliasing Fig. 8 shows that SwinIR-LTE is capable of estimating dominant frequencies of natural images. In addition, the middle row in Fig. 12 demonstrate that SwinIR-LTE extracts essential Fourier information under mild aliasing. However, such capability of SwinIR-LTE is limited when an LR image has severe aliasing. From Fig. 12, we observe that dominant frequencies (bottom right) are inconsistent with a GT spectrum (top right) when harsh aliasing artifacts occur in an LR image (bottom left). We can resolve such limitations by extending the size of an encoder’s RF, followed by increased computation and memory costs. Cost-effective architectures achieving robust performance even under severe aliasing will be investigated in future work.

Gibbs phenomenon When representing continuous signals with a finite sum of Fourier basis, function overshoots at discontinuities. Such observation is referred to in the literature as the Gibbs phenomenon or ringing artifacts in 2D images. From Fig. 13, we notice that LTE might cause overshoot at large scale factors, e.g., $\times 12$. Further investigation of smoothing algorithms to alleviate such an issue is a promising direction for future work.

Computation time In practice, SR applications require short computation time. Moreover, reconstructing high-quality images, such as DIV2K, consumes extensive memory during evaluation. Tab. 4 compares the computation time of our LTE to other arbitrary-scale SR methods for both cases: memory-limited (top rows) and memory-consuming (bottom rows) on NVIDIA RTX 3090 24GB. To evaluate an HR image under a memory-limited condition, we compute $96\times 96$ output pixels per query [4]. From the top rows of Tab. 4, we observe that our LTE takes the shortest computation time while increasing memory usage. LTE has 4-layer MLP for querying, two convolution layers for estimators, and LIIF has 5-layer MLP only for querying. When querying evaluation points only once, estimators become more dominant than querying, resulting in more computation time, as in the bottom rows of Tab. 4. To overcome such a limitation, we design an LTE+, which utilizes $1\times 1$ convolution instead of a shared MLP for decoder implementation. Since $1\times 1$ convolution has a GPU-friendly data structure, our LTE+ takes a shorter computation time and consumes less memory compared to previous works when all output pixels are queried at once.

In this paper, we proposed the Local Texture Estimator (LTE) to overcome the spectral bias problem of an implicit neural function. Our LTE-based arbitrary-scale SR method consists of three components: (1) Deep SR encoder (2) LTE (3) Implicit representation function. Firstly, a deep SR encoder extracts feature maps whose height and width are the same as an LR image. Then, LTE takes feature maps from the encoder and estimates dominant frequencies with corresponding Fourier coefficients for natural images. Scale-dependent phase and LR skip connection are further provided to allow LTE to be biased in learning high-frequency textures. Finally, the implicit function reconstructs an image in arbitrary resolution using estimated Fourier information. We showed that our LTE-based neural function outperforms other arbitrary-scale SR methods in performance and visual quality with the shortest computation time.