Multiresolution Neural Networks for Imaging

The main breakthrough in deep learning for computer vision and imaging was due to the seminal work of LeCun, Bengio, and Hinton [LB98]. This paper proposes the Convolutional Neural Network (CNN) as a proper architecture for the analysis of visual imagery. The effectiveness of CNNs comes from the translation invariant properties of the convolution operator.

Deep multi-layer perceptron networks, such as CNNs, employ an array-based discrete representation of the underlying signal. In this case, the network input consists of a vector of pixel values (R,G,B) that represents the image directly by data samples. Therefore, we can also call this kind of network a data-based network.

In contrast, a coordinate-based neural network represents the image indirectly. This kind of network is a fully connected MLP (Multi Layer Perceptron) that takes as input a pixel coordinate $(x,y)$ and outputs the (R,G,B) color at that location.

While the data-based network is appropriate for analysis tasks, relying on a discrete description of the image, the coordinate-based network is suitable for synthesis, providing a continuous image description. For its characteristics, namely continuity and compactness, there is a growing interest in the research community to explore the potential of coordinate-based networks for imaging applications.

Coordinate-based networks provide a continuous functional description for images using an implicit neural representation [CLW21]. Since the coordinates are continuous, images can be presented in arbitrary resolution.

Spectral neural implicit architectures constitute a particular form of neural implicit representation in which the non-linear activation function is the periodic function $\sin(x)$. As such, it bridges the gap between the spatial and spectral domains, given the close relationship of the sine function with the Fourier basis.

However, these neural representations with periodic activation functions have been regarded as difficult to train [PHV17]. To overcome this problem, Sitzmann et al. [SMB⁺20] proposed a sinusoidal network for signal representation called SIREN. One of the key contributions of this work is the initialization scheme that guarantees stability and good convergence. Furthermore, it also allows modeling fine details in accordance with the signal’s frequency content.

A multiplicative filter network (MFN) [FSWK20] is a spectral neural implicit architecture simpler than SIREN which is equivalent to a shallow sinusoidal network. Lindell et al. [LVPW22] presented BACON (Band-limited Coordinate Network), an MFN that produces intermediate outputs with an analytical spectral bandwidth which can be specified at initialization and achieves multi-resolution decomposition of the output.

The control of frequency bands in the representation is closely related with the capability of adaptive reconstruction of the signal in multiple levels of detail. In that context, Mueller et al. [MESK22] developed a multiresolution neural network architecture based on hash encoding. Also, Martel et al. [MLL⁺21] designed an adaptive coordinate network for neural signal representation.

Another benefit of multiresolution image representations is the built-in support for antialiasing, which traditionally is implemented using image pyramids, such as in MIP Mapping [Wil83].

One of the important applications of imaging in both 2D and 3D Computer Graphics is texture synthesis. In that realm, besides antialiasing, the creation of visual patterns from exemplars has great relevance [TZN19]. Spectral neural implicit architectures are particularly suited to model stationary or quasi-stationary signals due to the periodic nature of its activation function [CLC⁺22].

In summary, we make the following contributions:

• •

  We introduce a family of multiresolution coordinate-based networks, with unified architecture, that provides a continuous representation spatially and in scale.

• •

  We develop a framework for imaging applications based on this architecture, leveraging classical multiresolution concepts such as pyramids.

• •

  We show that our architecture can represent images with good visual quality, outperforming related methods both in PSNR and number of parameters; we also demonstrate its use in applications of texture magnification and minification, and antialiasing.

Our proposal is a family of coordinate-based networks with unified architecture. We derive three main variants, namely S-Net, L-Net, and M-Net. As a whole, they provide different trade-offs with respect to control of frequencies in the representation.

The characteristics of the MR-Net Family are:

• •

  2 Types of Level of Detail – in this respect it can be based on network capacity or spectral projection.

• •

  3 Types of Sampling – the input signal can be given either by regular sampling (with or without subsampling) or by stochastic / stratified sampling.

• •

  Progressive Training – the network is trained progressively using a variety of schedule regimes.

• •

  Continuous Multiscale – the representation is continuous both in space and scale. Therefore it can reconstruct the signal at any desired resolution / level of detail.

The following subsections present the concepts involved in our approach, as well as, the technical details of the MR-Net architecture. For a more complete description see [VPNY22].

Based on considerations for learning and splitting frequencies of the input signal into levels of detail, we devised a general architecture for a family of neural networks.

The core idea is to structure the network into multiple stages. Each stage learns in a controlled way a level of detail corresponding to a frequency band.

A stage configuration is derived from a sub-network called MR-Module, which is composed of four blocks of layers: the first layer; the hidden layers; the linear layer; and the control layer. The three initial blocks form a fully connected sinusoidal MLP. The last block consists of one node (not trained) to control the stage output, given by the function $c_{\alpha}({x})=\alpha x$, with $\alpha\in[0,1]$, where ${x}$ comes from the linear layer. Figure 1 depicts the anatomy of the network, showing $N$ stages of MR-Modules.

We can say that the first layer performs a projection of the input signal into a dictionary of sine functions, the hidden layers correspond to correlations of order $n$ of signal frequencies, the linear layer reconstructs the signal as a combination of these frequency atoms, and the control layer is just a mechanism to provide a continuous blend of level of detail in the network.

The stages of the network are trained based on a predefined schedule (See Section 2.3). During training, the control layer is the identity function, i.e. $\alpha=1$.

The contribution of these $N$ stages is added together forming the network output. Assuming that the MR-Net is learning a function $f(x)$ that fits the input signal, then

where $g_{i}(x)$ is the detail function given by the stage $s_{i}$, for $i=1,\ldots,N$. The first stage $s_{0}$ corresponds to the coarsest approximation of the signal and the other subsequent stages add increasingly finer details to it.

Note that this architecture is very much in the spirit of the Multiresolution Analysis [Mal89]. Indeed, consider the base case with $f(x)=g_{0}(x)+g_{1}(x)$, then $g_{1}(x)=f(x)-g_{0}(x)$, i.e., $g_{1}$ are the details that need to be added to $g_{0}$ to increase the level of detail.

The network learns the decomposition of the signal as the projection into the coarse scale space and a sequence of finer detail spaces. The characteristics of the level of detail decomposition of each member of the MR-Net family will depend on the specific configuration of the network stages, as will be presented next.

The training of the MR-Net has to take into account the mechanisms for learning different levels of detail by each stage of the network. There are two basic mechanisms: i) level of detail filtering and ii) pre-processing of the input signal.

Level of detail filtering is achieved either with the frequency band filtering (through the initialization of the $w_{0}$ weights of the first layer); or with the capacity filtering (conditioned on the number of nodes and layers of the network).

Regarding the network input, we can either use the original signal, or pre-process the signal with a low-pass filter.

By incorporating the different aspects discussed in the previous sections we can define various schemes for learning level of detail representations using the family of Multiresolution Sinusoidal Neural Networks. The main ones are: Filtering with Gaussian Tower; and Filtering with Gaussian Pyramid. Here we highlight these two.

We now show an example of a multiresolution image representation using the setup described above. For this experiment we chose the ”Cameraman”, a standard test image used in the field of image processing and also in [SMB⁺20]. The source is a monochromatic picture with $512\times 512$ pixels of resolution.

Figure 5 depicts the levels 1, 3, 5 and 7 of the multiresolution hierarchy reconstructed with full resolution of $512\times 512$, as well as the corresponding Fourier spectra (intermediate levels skipped to fit the page).

The training times for each stage of the networks are as follows: 5s, 4s, 3s, 11s, 17s, 29s and 48s. The total training time is 117s. The machine was a Windows 10 laptop with a NVIDIA RTX A5000 Laptop GPU. Note that these times result from the adaptive training regime and the number of samples for each level of the Gaussian Pyramid.

The training evolution is depicted in the graph of Figure 6 that shows the convergence of the MSE loss with the number of epochs for each multiresolution stages 1, 3, 5, and 7. It is worth pointing out the qualitative behavior of the network, in that the base level (stage 1) takes more than 200 epochs to reach the limit, while detail levels (stages 3, 5, 7) take less than 150 epochs to converge. Also, the error decreases for each level of detail. It is like, there are two different modes, one to fit the base level and other for the detail levels.

The inference time for image reconstruction, both in CPU and GPU is sufficiently fast for interactive visualization.

The second imaging example is the usage of the MR-Net representation to model texture and patterns. Arguably, visual textures constitute one of the most important applications for images in diverse fields, ranging from photo-realistic simulations to interactive games. Currently, more and more image rendering relies on some kind of graphics acceleration, sometimes through GPUs integrated with Neural Engines. In that context, it is desirable to have a neural image representation that is compact and supports a level of detail.

For the experiment shown in this subsection we have chosen an image of woven fabric background with patterns. The characteristics of this texture allows us to explore the limits of visual patterns at different resolutions. The original image has a resolution of $1025\times 1025$ pixels and the corresponding model contains 5 levels of detail.

In Figure 7 we show our reconstruction at original resolution, Fig.7(b) (center); as well as a zoom-in, Fig.7(c) (right), and a zoom-out, Fig.7(a) (left).

The zoom-in is a detail with resolution of $562\times 562$. marked by the white rectangle in Fig.7(b) and scaled up to $1025\times 1025$. Thus a zoom-in factor of 1.8 times. It can be seen that the enlargement extrapolates the fine details of the image at this higher resolution beyond the original image.

The zoom-out is a reduction of the entire image to $118\times 118$ pixels (shown enlarged to $501\times 501$ in the image for better viewing). The top sub-image is the network reconstruction at the appropriate level of detail (approximately 0.92). The bottom sub-image is point-sample nearest neighbor reduction. It can be seen that our reconstruction is a proper anti-aliased rendition of the image, while the sampled reduction exhibits aliasing artifacts.

These two behaviors in the experiment are manifestations of “magnification” and “minification”, classical resampling regimes for respectively scaling up and down the image [Smi21]. In the first case it is necessary to interpolate the pixel values and in the second case it is required to integrate pixel values corresponding to the reconstructed pixel. The M-Net model accomplishes these tasks automatically. Note that we have chosen a fractional scaling factors in both cases to demonstrate the continuous properties in space and scale of the M-Net model.

In the previous subsection we resorted to level of detail control to guarantee an alias free rendering independently of the sampling resolution. However, this task was facilitated because we could use a constant level of detail for the entire image, due to the zooming in/out operation in 2D.

On the other hand, in texture mapping applications, this scenario is no longer the case. Typically, it requires to map a 2D texture onto a 3D surface that is rendered in perspective by a virtual camera. In such situation, the level of detail varies spatially depending on the distance of the 3D surface point from the camera. Here, proper anti-aliasing must compensate the foreshortening caused by a projective transformation. Next we present a simple example of anti-aliasing using the M-Net.

Let $I$ be a checkerboard image, $T$ be a homography mapping the pixel coordinates $(x,y)$ of the screen to the texture coordinates $(u,v)$ of $I$, and $f=g_{1}+\cdots+g_{N}:[-1,1]^{2}\to\mathbb{R}$ be a M-net with $N$ stages approximating $I$.

Figure 8(left) illustrates aliasing effects on the image $I$ after applying it to the inverse of $T$. We avoid such a problem using the multiresolution of the M-Net $f$. The result is presented in Figure 8(right). Observe that the procedure reduces aliasing at large distances. Specifically, we define the level of detail parameter $\lambda(x,y)$ for the M-Net $f$ at a pixel $(x,y)$ considering the formula proposed by Heckbert [H⁺83]:

Thus $\lambda(x,y)$ is the bigger length of the parallelogram generated by the vectors $\frac{\partial T}{\partial x}$ and $\frac{\partial T}{\partial y}$.

We define the resolution $\lambda$ of $f$ using the formula

where $\lambda_{i}$ are weights defined as follows. First, scale $\lambda$ such that $\lambda\big{(}[-1,1]^{2}\big{)}\subset[0,N]$. Thus, set $\lambda_{i}=1$ for $1\leq i\leq\lfloor\lambda\rfloor$, $\lambda_{\lfloor\lambda\rfloor}=\lambda-\lfloor\lambda\rfloor$, and $\lambda_{i}=0$ otherwise. Here $\lfloor\lambda\rfloor$ denotes the floor value of $\lambda$.

Parameters: Hidden Features = 256; Hidden Layers = 3; Number of Levels = 1; w0 parameter = 120; Model Size = 197376; Image Resolution = 512 x 512. The PSNR of Reconstruction = 61.8 db.

Parameters: Hidden Features = 256; Hidden Layers = 6; Learning Rate = 0.005; Number of Steps = 5001; Pe Scale = 3.0; Model size = 398343. The PSNR of Reconstruction = 82.1 db.

BACON controls the frequency band for Level of Detail by truncating the spectrum of the more detailed level (note that the center of the spectrum images for Levels 1 and 2 in Figure 10 are analogous to the center of the spectrum in Level 6). This approach is analogous to applying a low-pass filter with non-ideal shape in the frequency domain, which results in an image with ringing effect (notice how the silhouettes propagate all over the left image in Figure 10). M-Net does not present such artifacts. Figure 11 resembles more faithfully what would be expected to be the process of sequentially applying Gaussian filters in a high resolution image.

We found that choosing appropriate values for the $\omega_{0}$ parameter, which defines the spatial frequency of the first layer of the network, is important to achieve proper results. When using a shallow sinusoidal network such as the S-Net, we can use the Nyquist frequency as a direct reference to pick the frequency intervals at each stage. However, a deep sinusoidal network such as the M-Net can learn higher frequencies that were not present in the initialization. To the best of our knowledge, there is not (yet) theory to compute or bound these frequencies.

In our experiments, we determined the $\omega_{0}$ values for initialization empirically, testing values below the Nyquist frequency. To better harness the power of sinusoidal neural networks, it is important to develop mathematical theories to understand how the composition of sine functions introduces new frequencies based on the initialization of the network.

In terms of future work, we plan to expand this research in two main directions. On one hand, we would like to explore the MR-Net architecture for other image applications including super-resolution, operations in the gradient domain, generation of periodic and quasi-periodic patterns, as well as image compression. On the other hand, we would like to extend the MR-Net representation to other media signals in higher dimensions, such as video, volumes, and implicit surfaces.