Utility-Oriented Underwater Image Quality Assessment Based on Transfer Learning

IQA can be divided into three categories according to its objective as described above. Fidelity-oriented IQA evaluates whether the image clearly conveys all visual information. The related studies can be classified into Natural Scene Statistics (NSS)-based methods and learning-based methods. [4] utilized an NSS model of DCT coefficients to predict image quality scores. [5] used locally normalized luminance coefficients to quantify possible losses of naturalness in images due to distortions. A small codebook was employed in [6] for a general-purpose blind IQA based on high-order statistics aggregation. [7] integrated the features of NSS derived from multiple cues to learn a multivariate Gaussian model of image patches. [8] combined feature learning and regression into one optimization process to estimate quality. A neural network-based pooling was presented in [9] to assess the global image quality with local patch qualities. To explore the relationship between fidelity and quality, [10, 11] proposed deep learning methods. Notwithstanding the prosperity of methods, the above-mentioned ones perform well in the quality estimation of Natural Scene Images (NSIs) but fail in that of utility-oriented IQA.

Unlike fidelity-oriented IQA, aesthetics-oriented IQA prefers visual contents with reasonable layouts and visual comforts. [12] relied on artificially designed functions to improve the image aesthetics reasoning of Convolutional Neural Network (CNN). [13] proposed a two-stream CNN which considers heterogeneous and complementary aesthetic perceptual abilities respectively. [14] proposed to quantify image aesthetics by distributing it across multiple quality levels. In [15], authors proposed a multi-reference eye inpainting Generative Adversarial Networks (GAN) approach based on an eye aesthetic dataset. [16] presented a deep multi-modal learning for aesthetic quality assessment of unmanned aerial vehicle videos. [17] employed a gated information fusion network to weight the roles of foveal vision and peripheral vision, which are key issues in image aesthetic evaluation. [18] proposed a semi-supervised deep active learning to explore the way humans perceive semantically important areas in images. Moreover, it developed a probabilistic model to incorporate the aesthetic experience of multiple users by encoding the experience of several professional photographers. These methods focus on various visual factors and composition to enhance users experience.

The utility-oriented IQA evaluates the task-aware utility based on the richness of useful information. High-utility images help complete subsequent tasks. At present, there is few work in utility quality assessment. Rouse et al. proposed the concept of utility assessment for natural images in [1], where the objective evaluation was expected to be consistent with subjective judgement of usefulness. In this paper, a subjective utility quality database for natural images, referred to as the CU-Nants database, was obtained. On basis of CU-Nants database, a series of full-reference utility quality assessment methods were proposed [2], [1], [19]. These methods are all built upon the hypothesis that contour degradations are consistent with decreased perceived utility. In [20], Edward et al. came up with a no-reference utility measurement utilizing the Oxford Visual Geometry Group’s (VGG) deep CNN for CU-Nant database. As an extent, Edward et al. also found that highly performing utility measurements can also predict saliency for object recognition [21], since the utility measured is related to the contour difference between test and reference images, and the distortion in contour impacts the ability of observer to recognize object. [22] considered the influence of blur, dramatic pose variations, and occlusion on face quality. The utility evaluation was guided by the vascular structure, rather than the perceived quality of the whole retinal image in [23]. Besides, video utility evaluation algorithms have also been discussed in the context of tasks such as compression [24], [25].

Despite of these great efforts, there are still lack of a deep analysis of utility-oriented IQA and a large-scale database for underwater image utility evaluation. Most of aforementioned contributions are designed for natural images or videos. Nevertheless, underwater images are different from natural images both in terms of the statistical properties, visual characteristics and distortion types. As for underwater images, although sonar images [26] can provide acoustic information that assists underwater detection, underwater optical imaging is still critical to show more intuitive visual information and its quality assessment is different from fidelity evaluation of sonar images [27]-[28]. Therefore, the fidelity-based sonar IQA is not suitable for the utility-oriented IQA discussed in this paper.

In response to these problems, we design a utility-based subjective experiment to construct a quality database. Then, an image quality evaluation method based on the fish detection task is proposed.

Due to the limited training samples in the existing database, IQA based on deep learning faces the problem of overfitting. Transfer learning [29] offers an effective solution by utilizing data from a related source task to improve the performance. However, there are few works on quality evaluation based on transfer learning. A transfer learning framework was described in [30], which learned an end-to-end image quality estimator in classification or regression. In [31], the features extracted from distorted images were transferred to the same feature space, in order to solve the problem of insufficient video contents. [32] developed an IQA architecture of multi-domain transitive transfer learning, which is associated with the ImageNet source domain, the IQA target domain, and their corresponding tasks.

Object detection is to find out the objects that people are interested in from images or videos, and detect their position and size. Different from the image classification task, object detection must not only solve the classification problem, but also the positioning problem. Until 2012 [33], the rise of CNNs pushed the field of object detection of natural scene images to a new level. With the development of ocean exploration and exploitation, detecting fish from underwater videos and images is of great significance for fishery resource assessment and ecological environment monitoring. In [34], authors proposed a novel composite fish detection framework based on a composite backbone and an enhanced path aggregation network called Composited FishNet. [35] presented a novel dataset with 400 images of fish in the wild. By using these dataset, the state-of-the-art detection models are trained with fine-tuning strategies. [36] proposed a deep but lightweight neural network to detect fish. However, due to the appalling underwater conditions, images or videos captured underwater are often with poor utility. Most fish targets are small and easily confused. It is difficult to achieve fully automatic machine recognition according to the current technology.

Inspired by these, we propose a UIUM metric based on transfer learning, which uses a trained object detection model to share the prior knowledge of key feature information for detection.

We have built a fish image quality database (i.e., the UIFD database) which contains 2675 fish images. This database employed the Fish4Knowledge video repository, which was taken to monitor coral reefs, to simulate the underwater environment and combine it with an underwater fish detection task. In this work, 145 images with clear fish characteristics were selected from the Fish4Knowledge video repository for the UIFD database, and each image includes 1 to 6 fishes. To simulate the complex underwater environment, five typical underwater distortions are devised, including channel distortion, contrast distortion, illumination distortion, motion blur, and ocean-snow distortion. Besides, the foreground/background distortion is also introduced to cover the impact of regional distortions on object detection. There are 4 to 5 distortion levels for most of the distortion types. Besides, we add 90 non-target images to conduct a tentative analysis. A high-fidelity image without targets is considered as a low-utility image. By introducing the foreground/background distortions as well as non-target images, the proposed database covers different utilities of underwater images.

In the proposed UIUD database, a total of 3,340 images are generated including original images and the corresponding distorted images. We present some examples of the UIUD database in Fig. 3. For convenience, the distortion type is marked as “Type1” to “Type6”. The detailed information corresponding to each type is given in Table I.

We invited 21 subjects to conduct subjective experiments on the supplementary images except for no-target images. The number of subjects can make the experimental results reach the saturation point. All subjects have been pretrained with sufficient knowledge of underwater fishes. The collected information includes the number of fishes, the number of fish species, the time for the subject to make a judgment, and the quality of this image. The first three kinds of information are analyzed in further research. In the UIUD database, only the subjective scores are used.

Particularly, the scoring of image quality is utility-based in this work. Traditional Absolute Category Rating (ACR) defined in ITU-R BT.500 utilizes five adjectives as categorical scales to assess image quality, which is called adjective categorical judgment methods as the right column of Table II shows. The UIUD database focuses on the utility of image, i.e., whether or not the fish can be detected. So we develop the rating scales as given in the left column of Table II. In Table II, the first adjective reflects the completeness of fish information, while the second adjective establishes the identification threshold of target fish. Apart from that, the experimental environment and monitor are calibrated according to the recommendations of ITU-R BT.500 [37].

Finally, to improve the reliability of results, we randomly sampled 5 images to form a verification set. If a subject presents significantly different scores between the verification set and those corresponding images in the subjective test, his or her score will be discarded. In detail, if the difference between two scores of the same image is more than 2, the image is regarded as a fluctuation image. In this subjective test, the average number of fluctuation images of each subject is less than 1. Thus no score has been discarded, which means that all subjects are reliable.

Follow the practices in [43], we further verify the reliability of user scores obtained in Section III-B. Outlier Coefficient (OC) is introduced to quantify the subjective agreement of the UIUD database:

where $N_{\text{total}}$ denotes the total number of labeled images, and $N_{\text{outlier}}$ denotes the number of the images whose interval between the 25th and 75th percentiles of subjective ratings is larger than 1. To visualize the results of OC, Fig. 4 shows the boxplot of subjective scores of 30 images. A subjective score is considered as an outlier when its blue rectangle is larger than 1. According to the analysis, our database achieves an OC of 5% and thus is considered to be with high subjective agreement. Then, we process the subject rating values given by each viewer into vectors and then compute the Normalized Cross Correlation (NCC) and the Euclidean Distance (EUD) between every two vectors. The final values of NCC and EUD achieve 0.91 and 0.08, respectively. The results demonstrate the high correlation between two subjective rating vectors.

After data processing, Mean Opinion Score (MOS) values were calculated and used as the image labels. The MOS histograms of the UIUD database and other IQA databases are shown in Fig. 5. Among them, the proposed database has its own characteristics in data distribution. The data often presents Gaussian distribution without task constraints as Fig. 5(a)(b)(c) show. However, there are discontinuities according to different requirements in subjective experiments with task backgrounds. The scores increase with more fish information but saturate when the image clarity increases to a certain level. In contrast, there are fewer mid-quality images. This is consistent with the facts reflected in Fig. 5(d). There are few images with mid-quality between 30 and 45 and also few images with high quality above 65. The different distributions of scores lead to different characteristics of UIUM compared with conventional IQA approaches.

Underwater images for fish detection have their particular features which are different from features extracted in fidelity- or aesthetic-oriented IQA. Traditional methods such as pixel-mapping-based measurements and textural features are not targeted in utility measurement. Moreover, there is still a lack of training data in real-world applications. To address these problems, the transfer learning technique, which requires less training data, is utilized in this paper to transfer utility-based features from fish detection networks to an IQA model. In addition, the introduction of transfer learning is also motivated by the top-down design of perception-based, in which IQA learns various perceptual characteristics of the human visual system. As shown in Fig. 6, we design a dual-stream output structure that is decomposed into main-task and sub-task. The sub-task is trained on a fish detection database to get a fish detection model. The main-task predicts the utility-oriented quality of images.

The proposed network includes a shared layer and a dual-stream output layer. The shared layer utilizes the YOLOv4 which consists of backbone and neck to transfer fish detection network to utility-oriented IQA. It will be used as a feature extractor for both the main-task and the sub-task. The dual-stream output layer completes object detection and quality prediction, which can guide the realization of transfer learning.

In the shared layer, the backbone and neck are critical detectors to extract basic features for object detection. We employ CSPDarknet53 as the backbone for its high accuracy and low complexity. CSPDarknet53 is a combination of Yolov3 backbone network and Cross Stage Partial Network (CSPNet) [44]. CSPNet mainly solves the problem of complex computation from the perspective of network structure design. The neck further processes the important features extracted by the Backbone. In order to better extract the features concerned by the fish detection task, Spatial Pyramid Pooling (SPP) [45] module, Feature Pyramid Networks (FPN) [46] and Path Aggregation Network (PAN) [47] module of YOLOv4 are implemented. The SPP module employs different kernels for pooling and then concatenates feature maps of different scales, thus it can effectively increase the receiving range of backbone features and significantly separates the most important context features. The FPN layer conveys strong semantic features from top to bottom, while PAN transports strong positioning features from bottom to top. They work together to aggregate parameters from different backbone layers to different detection layers.

After the shared layer, three feature maps under different scales and receptive fields are obtained. Fig. 6 (d) is used for sub-task, whose loss function is generally composed of classification loss and bounding box regression loss. The output of Fig. 6 (d) verifies the performance of object detection. Fig. 6 (e) is designed for main-task, i.e., predicting quality. The fully connected layer is exploited to achieve quality prediction. Experimental result shows that an optimal number of fully connected layer is 1. We connect feature maps into a one-dimensional vector. This vector will be inputted into this fully connected layer. The Mean Square Error (MSE) is employed as a loss function in the main-task, which is widely used in various regression tasks.

After building the network architecture, we exploit the shared layer and dual-stream output layer to transfer an object detection model to an IQA model. Our transfer learning process is performed in two steps: pre-training and fine-tune.

Throughout the training process, we utilize three databases, namely the UIUD database, Fish Object Detection Database (FODD) and Microsoft Common Objects in Context (COCO) [48]. The UIUD database is the image quality database established in this paper, and its label is a subjective utility quality score. FOOD and COCO are both object detection databases, and their labels are categories and coordinates. COCO is a large-scale object detection database that mainly obtains images from complex daily scenes. FODD is a database that we compiled to train the fish detection network. We denote these training datasets UIUD, FODD and COCO by $D_{\text{t}}$, $D_{\text{f}}$ and $D_{\text{c}}$, respectively.

Methods for comparison. To compare the performance comprehensively, we choose several state-of-the-art methods for comparison. They include 11 fidelity-oriented methods (BLIINDS-II [4], BRISQUE [5], HOSA [6], ILNIQE [7], WaDIQaM-NR [9], CNN-IQA [8], PSNR, SSIM [49], FSIM [50], UCIQE [51], UIQM [52]). Among them, the WaDIQaM-NR and CNN-IQA are deep learning-based NR-IQA methods. The PSNR, SSIM and FSIM are FR IQA algorithms. In addition, the last two are underwater IQA algorithms. In particular, underwater utility-oriented IQA (NRCDM [28]) and aesthetic-oriented IQA (NIMA [53]) have also been added for comparison.

Performance criteria. Three commonly used criteria are chosen to calculate the correlation between the subjective and objective quality scores, which indicates the performance of IQA methods. These criteria include Spearman Rank-Order Correlation Coefficient (SRCC), Pearson Linear Correlation Coefficient (PLCC), and Kendall Rank Correlation Coefficient (KRCC).

The three metrics mentioned above can effectively evaluate the monotonicity, accuracy and consistency between the prediction quality of the algorithm and the MOS. However, these performance metrics suffer from some drawbacks. On the one hand, they do not take into account the uncertainty of subjective ratings. On the other hand, they require mapping between predicted values and subjective scores. These defects not only cause the performance metrics to be vulnerable to the quality range of the stimuli in the experiments, but also result in that the performance comparison is not performed in the real scenarios. In order to overcome the above shortcomings, we utilize the method proposed in [54], [55], which is less dependent on the range effect and inspired by the real application without any mapping. First of all, the subjective scores need to be pre-processed as follows:

where $var(i)$ and $N(i)$ denote the variance of the subjective scores and the number of subjects of image $i$, respectively. $MOS(i)$ is the MOS value of image $i$. Then, the cumulative distribution function (cdf) of the normal distribution is employed to calculate the disparity between the qualities of image $i$ and $j$:

where the paired images are subjectively considered to be significantly different when $p_{z(i,j)}>0.95$. For significantly different pairs, the quality difference predicted by IQA model $m$ is defined as:

where $Q_{m}(i)$ represents the objective score predicted by IQA model $m$ for image $i$. When $\Delta_{m}(i,j)>0.95$, the quality of image $i$ is objectively identified as significantly better than the quality of image $j$.

After the above processing, the percentage of correct recognition of qualitatively better image from the pair, which is denoted by $C_{0}$ (the higher the better), is employed as one performance metric.

Performance on UIUD database. The UIUD database is divided into training and testing sets with an 80/20 split and no content overlapped. Besides, 10-fold cross-validation is utilized to evaluate the performance. Since the UIUD database is currently the first utility-oriented image quality database, we cannot perform cross-database verification. To ensure fair comparisons, the compared algorithms with deep learning are retrained and fine-tuned in the UIUD database. Table III shows the average results of all methods, where the best and 2nd-best performances are highlighted with bold and underline respectively. Our method outperforms all of the state-of-the-art methods for PLCC, SRCC and KRCC evaluations. As shown in Table III, only the proposed method and the WaDIQaM-NR method achieve SRCC and PLCC values above 0.8, while the performances of other algorithms are not ideal under the task of object detection. Deep network structure, feature fusion and spatial pooling methods make WaDIQaM-NR competitive. Especially, the WaDIQaM-NR network is significantly deeper than related IQA models. From this comparison, the UIUM is more relevant to utility. Although NRCDM evaluates images based on utility quality, its performance is less than satisfactory. This is owing to the big gap between imaging principles of acoustic image and optical image. The poor performance of NIMA results from the fact that it evaluates image content based on aesthetic appreciation.

Performance on individual distortion types. To further understand the effect of different distortion types, we test the performance of different algorithms on each individual distortion type. The results are shown in Table IV, where the best and second-best results are also shown in bold font and underline, respectively. Our algorithm has high correlations in all distortion types. The other algorithms also have higher correlations in some traditional distortion types, such as contrast distortion and illumination distortion. This is obvious because these types of distortions also appear in fidelity-oriented quality evaluations. Most algorithms perform well in transmission distortion, since the degree of distortion is highly correlated with visual perception. Furthermore, the traditional methods based on manual feature extraction fail on the remaining distortion types. The reason is that the extracted features are not suitable for underwater images, especially under task backgrounds.

From Table IV, three deep learning methods have relatively superior performances due to their strong learning abilities. CNN-IQA has no advantage compared with WaDIQaM-NR and UIUM algorithms, since its network structure is relatively simple and less targeted. Quite rightly, deep learning is data-driven, we will further verify the superiority of our algorithm from the perspective of data requirements in following subsection.

Discrimination ability for significantly different qualities. In order to test the discrimination ability, we calculate $C_{0}$ for each selected IQA methods and the UIUM. Since testing in the UIUD requires large-scale complicated computation, we randomly sampled 100 images from each distortion type and completed the test on this subset. For convenience, the IQA methods are numbered separately as shown in the penultimate column of Table III. The last column of Table III shows the exact values of $C_{0}$ for selected IQA methods. The results in the table indicate that UIUM (#14) has the best performance, and the WaDIQaM (#6) is second only to the UIUM in $C_{0}$. The same results can be obtained from the data bar of Fig. 9 (left), UIUM (#14) outperforms all other algorithms. The significance plot about $C_{0}$ is also shown in Fig. 9 (right). In this plot, black and white boxs indicate the performance of the method in the row is significantly lower and higher than the method in the column, respectively. The gray box reflects the similar performance. It can be concluded from the box plot and table that the UIUM is significantly better at distinguishing difference in utility than other IQA methods.

Intuitive Comparison. To visually illustrate the performance of UIUM, its prediction scores and the subjective MOSs are directly compared in Fig. 10. Fig. 10(a) to Fig. 10(c) are the reference images, while Fig. 10(d) to Fig. 10(f) are the corresponding distorted images. The predictions made by UIUM have a high correlation with MOS values. Another key difference between UIUM and traditional IQA is shown here is that the quality score of the original image is not necessarily higher than that of the severely degraded images, such as Fig. 10(b) and Fig. 10(e).

Computation Time Comparison. To measure the time complexity of each algorithm, UIUM and all other algorithms are run on the UIUD database for testing. These computational cost tests are conducted on the MATLAB R2019a and Pytorch software platform on a computer with a 3.98 GHz CPU, 16.00 GB of RAM and an RTX2070 graphics card. The average time consumption of each method is tabulated in Table V. The first two lines are traditional NR-IQA, the third line is traditional FR-IQA, and the last line is NR-IQA based on deep learning. SSIM and FSIMc have obvious advantages in speed, but they are both methods of manually extracting features, which are difficult to achieve robust performance due to the limitation of fixed features. In the deep learning methods, the calculation time of UIUM is almost the same as that of CNN-IQA, while UIUM has great advantages over CNN-IQA in performance. WaDIQaM-NR is 4% worse than UIUM in performance, which is the closest to UIUM among all methods. It also lags behind us in computational efficiency. In general, UIUM has great advantages in performance and calculation time.

To further evaluate the impact of transfer learning, we conduct an ablation experiment. In the UIUM, the pre-trained weights are firstly fed in COCO. Then the model is trained on FODD to obtain a fish detection model. Finally, we fix the parameters of backbone and neck, and fine-tunes the fully connected layer on UIUD database. In the ablation experiment, we define an Object Detection Utility-Oriented Quality Assessment (ODUQA). The framework of ODUQA is the same to UIUM except that the ODUQA is trained on a normal object detection model, not a fish detection model. These two methods are tested separately in the UIUD database. The results in Table VI shows that our method achieves higher performance. It demonstrates that the features extraction module is able to precisely capture the useful information for the target domain (i.e., utility-oriented IQA) from the source domain (i.e., fish detection).

In this work, transfer learning makes full use of the labeled data of source task, which enable better performance of a new task with less labeled data. However, data annotating is boring and expensive. Therefore, we discuss the performance of UIUM and WaDIQaM-NR on smaller data sets in this section. The reason for choosing WaDIQaM-NR is its high performance on the complete UIUD database, and it is also an excellent deep learning-based NR-IQA method in recent years. After randomly deleting part of the reference images and their corresponding distorted images, we got two databases with 1007 and 2015 images, respectively. In this section, we name the databases according to the number of images, which are UIUD-3340, UIUD-2015, and UIUD-1007. The proportion of testing and training sets is the same as above.

The performances of UIUM and WaDIQaM-NR on the UIUD database with different image numbers are shown in Fig. 11. When the number of images changes, the performance of UIUM remains the same. However, the performance of WaDIQaM-NR drops significantly. When the number of images becomes 1007, its SRCC and PLCC drop to 10% and 8%, respectively. This fully proves that our algorithm employing transfer learning achieves better performance with less training data. Furthermore, the performance of WaDIQaM-NR starts to saturate on UIUD-2015. Specifically, the optimal SRCC and PLCC that WaDIQaM-NR can achieve in this task is about 0.8, while the UIUM yields a 5% performance gain In consequence, UIUM achieves higher performance in a smaller database, and its final performance is also better than WaDIQaM-NR.

We have added 90 non-target images to UIUD. The reason is that the task cannot be completed with a distortion-free and clear but targetless image. Therefore, we tentatively verify the non-target images. The result is shown in the Fig. 12. It can be found that UIUM can define most targetless images as low-quality images below 40 points, and the prediction of WaDIQaM-NR for these images is more volatile, even with predictions of more than 60 points. In the future, we will add non-target images and further analyze the content in combination with the background of specific tasks.