JPEG INFORMATION REGULARIZED DEEP IMAGE PRIOR FOR DENOISING

Before describing our proposed ES method, we describe deep image prior (DIP). DIP is proposed for image restoration problems including denoising. Image restoration algorithms aim to recover an unknown clean image $\boldsymbol{x}$ given a corrupted image $\boldsymbol{x}_{0}$. DIP argued that a neural network architecture behaves as the general image prior, and the clean image $\boldsymbol{x}$ can be recovered from the corrupted image $\boldsymbol{x}_{0}$ without additional regularizations:

where $\boldsymbol{z}$ is randomly initialized and fixed noise, $f_{\boldsymbol{\theta}}$ is the neural network parameterized by $\boldsymbol{\theta}$, and $\mathcal{L}$ is the loss function such as mean squared error.

As described in Sec. 1, compressed image file size (CIFS) can be used as a proxy metric of degrees of noise. As the network parameters fit to the corrupted image, the loss function $\mathcal{L}$ decreases almost consistently. On the contrary, the network gradually outputs a noisy image which would have a larger compressed image file size. Based on this insight, our ES considers finding trade-off between the loss function $\mathcal{L}$ and CIFS based regularizer $\mathcal{R}$ formulated as:

where ${C}$ is a function which takes an image as input and outputs CIFS, $t\in\mathbb{N}$ is an epoch number during optimization, and $\lambda\in\mathbb{R}_{\geq 0}$ is a balancing weight between two terms. In our experiments, we adopt mean squared error as the loss function $\mathcal{L}$ same as DIP, JPEG file size output function as $C$, and squared JPEG file size averaged over image size as $\mathcal{R}$. Specifically, $\mathcal{R}$ is

where $L$ is CIFS and $(H,W)$ are image height and width, respectively. Since CIFS depends on its image size, $L$ is averaged over its image size $HW$. Moreover, we also use squared file size $L^{2}$ so that regularization works strongly as the file size increases.

The $\lambda$ is affected by degrees of noise. We show estimated $\lambda$ for additive gaussian noise with different standard deviation $\sigma$ in Fig. 2. The $\lambda$ estimation is based on our preliminary experiment on CBSD500 [7]. We assume that we can observe 400 clean images on CBSD500. Under this assumption, we could search $\lambda$ by the following steps: (1) adding a gaussian noise to a clean image to create a noisy image synthetically, (2) denoising the noisy image by DIP and monitoring values $\mathcal{L},\mathcal{R}$ in Eq. (2), (3) and finding the optimized $\lambda$ as maximizing the average PSNR between the clean image and the denoised image at the ES criterion minimized epoch over the observed 400 images. Specifically, the final step is formulated as:

where $U_{\boldsymbol{z}}$ is a pixel-wise uniform distribution on $[0,1]$, and $U_{\boldsymbol{x}}$ is a uniform distribution on the 400 observed images of CBSD500.

We adopted CBSD68 [8] and Kodak24 [9] as image denoising benchmark datasets. CBSD68 is a different split not overlapping the split used in $\lambda$ estimation described in Sec. 3.2.

Non-reference image quality assessment (NR-IQA) has tackled to score image quality without a reference image. Since NR-IQA methods are expected to provide good image criteria, these methods can be employed as ES. Following the recent ES method for DIP, ES-WMV [4], we compared to BRISQUE [10] and NIQE [11] as the NR-IQA methods and ES-WMV [4] as the ES method to validate whether our proposed method can provide a good ES criterion. We used models and code provided by OpenCV²²2https://github.com/opencv/opencv_contrib [12] and LIVE³³3https://github.com/utlive/live_python_qa for BRISQUE and NIQE, respectively.

Let $T$ denote the number of training epochs. First, we optimized the neural network parameters with respect to Eq. (1) reaching to $T$ epochs. Second, we obtained the ES detected epoch $t_{\ast}$ such that each ES criterion is satisfied. Following ES-WMV ES detection, our and other ES criteria find ES candidate epochs $\mathcal{T}$ and adopt the minimum epoch in $\mathcal{T}$ as the ES detected epoch $t_{\ast}$. The candidate epoch $t\in\mathcal{T}$ satisfies that the ES criterion outputs a smaller value than the next consecutive $S$ epochs. Specifically,

where $M$ is a specific criterion function for each ES method. If an ES method cannot find any candidate epochs (i.e., $\mathcal{T}=\emptyset$), $T$ is adopted as an alternative to the ES detected epoch. Finally, PSNR is computed between the clean image $\boldsymbol{x}$ and the denoised image $f_{\boldsymbol{\theta}_{t_{\ast}}}(\boldsymbol{z})$ as the ES criterion evaluation.

CIFS utilizes JPEG as image compression method. JPEG can specify a quality value $Q\in[0,100]$ (higher is better image quality) which affects the saved image file size. $Q$ is set to 95 which is the default value in OpenCV.

The following other implementation details are shared with CIFS and all comparative methods. Pixel-wise gaussian noise is independently sampled from $\mathcal{N}(0,\sigma^{2})$ where we set $\sigma\in\{5,10,\ldots,75\}$ for pixel value range $[0,255]$. The network architecture is same as DIP and Adam [13] with a learning rate $0.01$ is used. The input noise $\boldsymbol{z}\in\mathbb{R}^{D\times H\times W}$ to the network is randomly initialized and fixed during optimization where $D$ is set to $32$. We adopted the $\boldsymbol{z}$ perturbation technique from DIP [1, 2] where we perturb $\boldsymbol{z}$ at each iteration. The number of training epochs $T$ is set to 20k. $S$ is set to 1k, which is same as ES-WMV experimental setting.

Table 1 shows PSNR comparisons of our and other comparative ES methods for $\sigma\in\{15,25,50\}$ on CBSD68 and Kodak24. Fig. 3 depicts PSNR performance gap comparison for $\sigma\in\{5,10,\ldots,75\}$ on CBSD68 as the broader noise level range evaluation. DIP without ES (”No ES”) is significantly lower than the maximum achievable PSNR (”Peak”) regardless of gaussian noise levels. CIFS provides good ES criterion and almost outperforms other comparative methods for both datasets. Moreover, the standard deviations of PSNR for different noise levels are relatively small compared to other ES methods. ES criteria and qualitative comparison for $\sigma=25$ are depicted in Fig. 4. Note the ”Peak” PSNR is same for all ES methods since all ES metrics are computed in the same optimization process. BRISQUE and NIQE, which are NR-IQA methods, have an issue with the magnitude of variance of their metrics. ES-WMV is stable, however, it keeps a long variance sequence since the metric is windowed moving variance. CIFS can compute the metric independently at each iteration and still provide a stable metric.

We thank Sol Cummings for helpful feedback. This research work was financially supported by the Ministry of Internal Affairs and Communications of Japan with a scheme of ”Research and development of advanced technologies for a user-adaptive remote sensing data platform” (JPMI00316).