Portuguese Man-of-War Image Classification with Convolutional Neural Networks

Jellyfish stings are coastal hazards being some of the most common envenomation encountered by humans in marine ecosystems. Among the most dangerous species, the jellyfish Portuguese man-of-war (PMW) (Physalia physalis) represents a public health issue, a threat to tourism in many parts of the world [11]. Their stings can cause intense immediate skin pain and elicit serious systemic effects, including headache, vomiting, abdominal pain, diarrhea, muscular, cardiac, neurological, and allergic reactions, causing even deaths occasionally [4].

Portuguese man-of-war has a gas-filled float that keeps it floating passively at the water surface with tentacles that can be more than 30 meters long containing the venom trailing below. Stings events are usually associated with wind conditions, local currents, and the consequent transport of these animals toward the coast. The abundance and occurrence of this species fluctuate inter-annually, but the highest risk of stings occurs during the warmer months, due to the peak of bathers in coastal environments during the holiday season [8].

Information about Portuguese man-of-war occurrence and distribution is essential to implement measures of prevention and manage stings risks, such as beach closures, warning flags, anti-jellyfish nets [18], and forecasting systems [12]. However, the Portuguese man-of-war is one of the least studied jellyfish. Gaps are caused partially due to the difficulty of studying this species, hampering its capture in traditional methodologies [10].

Scientists are increasingly utilizing information provided by citizens [13] to address knowledge gaps and acquire occurrence records of this species. [17] analyzed the feasibility of using data extracted from social media to complement the information on the distribution of Portuguese man-of-war on the Brazilian coast. For this, a methodology for collecting, filtering, and storing Instagram data was established. From specific hashtag searches, evidence such as photos and videos of the animal were sought in posts, as well as, information about the location and date of occurrence. However, the manual methodology of social media data filtering suffers the consequences of human limitations, resulting in a tedious and time-consuming process. Furthermore, the continuity and expansion of the method require a permanent human effort, making it a little scalable. Computational techniques, such as machine learning and artificial intelligence, are improving the ability to search, collect and process crowdsourcing ‘big data’ from social media in a less time-consuming way, also helping to create a centralized open-source data system [24].

The media treatment consists of verifying the presence of a Portuguese man-of-war in at least one of the images or videos of the social media post. Therefore, we propose to model the problem as a binary classification of images between a positive class for the presence of jellyfish, and a negative class represented by any other random image. This task is complex, and a suitable approach would be the use of Convolutional Neural Networks (CNN), a deep neural network that showed good previous results in complex image classification [20]. In the present study, we aim to implement and analyze some models of CNN applied to the image classification problem of the Portuguese man-of-war, to validate the use of this technology for the methodology defined by [17].

This paper is organized as follows. Section 2 covers the technical approach used for image classification. Related work is described in Section 3. Then, the methodology used for the development is described in Section 4. Section 5 presents and discusses the results obtained from the experiments. Finally, Section 6 concludes the paper with a discussion of the results and future directions.

There are several deep neural network architectures with a wide variety of applications. According to [1], one of the architectures with the most promising results in the area of image classification is the Convolutional Neural Network (CNN). CNNs gained popularity in 2012 for their use in the popular ImageNet Large Scale Visual Recognition Challenge (ILSVRC) object recognition competition [20], where they achieved very promising results compared to other technologies. One of the main factors for the success of CNNs is their ability to automatically extract features from the provided images, unlike other approaches that usually require manual feature extraction, such as support vector machines (SVM) [1]. The architecture of a CNN can have several formats, depending on its implementation. As a standard, the input of a CNN always goes through several processes sequentially, and each step of these processes is called a layer. The layers can be convolutional, pooling, normalization, fully-connected, loss, and other types [1]. A generic architecture of a CNN is shown in Figure 1.

The convolutional layers, which give the network its name, are meant to extract features from its input and form a feature map. Each input is convoluted with filters that can learn and passes through an activation function, generating an output feature map, which serves as input to the next layer[1].

CNNs, in general, require large image databases, such as ImageNet, to achieve good results [1]. So, transfer learning has become a popular technique. In transfer learning, a neural network is trained on problem $P1$, where there is a large image database. After then, this network is used to solve $P2$. This technique can achieve good results if both problems have related inputs. Moreover, it can help to improve the learning of useful features and generalization of the second problem [7]. Finally, a fine-tuning is applied, where adjustments are made to the pre-trained network to better adapt to the new problem. These adjustments usually include replacing the original output layer with an output layer that fits the new problem, as well as inserting new fully connected layers and regularization techniques. In addition to making changes to the network structure, a total or partial freezing of the pre-existing layers is performed. This procedure aims to prevent the network from having its old weights modified with the new training data, and consequently losing its existing robust learning.

Currently, there are several popular CNN architectures considered to be state-of-the-art, as they can achieve good performance on different classification tasks [1]. Performance comparison of architectures applied on large image databases has been fundamental in guiding the development of models applied on varied tasks [1]. A widely used competition in this regard is ILSVRC [20], which uses the ImageNet image database consisting of over 14 million images. Some architectures described here have gained prominence in this competition. The VGG (Visual Geometry Group) architecture defined in [22] is a trained model that was highlighted in the competition in 2014. It has six types of layers and a total of 13 convolutional layers and 3 fully-connected layers. Its variation, called VGG-16 and VGG-19, consists of two models that differ by the number of convolutional layers. The ResNet architecture was the winner of the ILSVRC challenge in 2015 in the categories of image classification, detection, and localization, in addition to other recognized challenges [9]. It can be created with various numbers of layers, but some of its most popular patterns are RestNet50, with 50 layers, and ResNet101, with 101 layers. Defined in [23], the Inception architecture was created to optimize the use of computational resources, so that the network can become more complex without becoming so computationally costly. The architecture has some variations, such as InceptionV1, InceptionV2, and InceptionV3.

This section will present the main related works that underpin the present work. First, the research related to Portuguese Man-of-war will be presented, focusing on their visual characteristics. Next, the researchers found focusing on deep learning applications of jellies will be described. For the task of image classification, it is important to perform an analysis of the main physical characteristics of the object of study and in which environments it is commonly photographed, to understand which images compose the database, and to identify the possible errors that can be made by the neural networks. The Portuguese Man-of-war, Physalia physalis, are organisms that live in a colony and belong to the cnidarian group, of the order siphonophores. The individuals that make up the colony are dependent on each other, and each has a specific function [16]. The greatest visual highlight of its composition is its pneumatophore, a balloon-shaped structure filled with the gas used in navigation, as it has vivid colors in pinkish and bluish tones [3]. Despite being quite characteristic, the colors and the shape of the caravela-portuguesa can be confused with another cnidarian: the Velella velella, popularly known as By-the-wind Sailor. Both species are oceanic and float freely on the sea surface, and eventually are carried to shore and stranded on the sand [2]. Analyzing Figure 2, it is possible to observe that the images of the Portuguese Man-of-war have a more bluish coloration and are in groups that can be easily confused with the images of the By-the-wind Sailor, even by humans.

Few studies were found that specifically focused on the classification of Portuguese Man-of-war images, therefore, we describe studies of any species of jellyfish or gelatinous organism, animals physically similar to the Portuguese Man-of-war. In [14] the Jellytoring system is presented, which aims to detect and quantify different jellyfish species through a CNN, allowing the generation of a history of the presence of these animals. The input images for this model come from underwater video recordings, and refer to three species: Pelagia noctiluca, Cotylorhiza tuberculata, and Rhizostoma pulmo. The architecture used was Inception-Resnet-v2 pre-trained with the COCO database, which was compared with the ResNet101 and InceptionV2 architectures, and achieved F1 Score metrics greater than 95%. The study presented in [6] analyzes the problem of detecting and classifying seven species of jellyfish and fish in underwater images, to promote the monitoring of these species. The architecture employed was YOLOv3, a CNN used to detect objects in an image. The authors make a series of improvements in the application of the CNN and manage to achieve the result of the accuracy metric at 95.53%. [15] presents JellyNet, a CNN model trained to detect jellyfish blooms from high-resolution images obtained by unmanned aerial vehicles (UAV), also known as drones. The architecture used was based on the VGG-16, and achieved 90% accuracy.

In [19], a system for predicting the occurrence of three jellyfish species, Cyanea capillata, Aurelia aurita, and Staurostoma mertensii, in the Gulf of Maine is tested. The system used data from satellites and also data submitted by citizens via email and the social network Twitter to train a prediction model. To engage citizens to collect and submit data for the research, announcements were made through social media and institutional networks asking for submissions. In total, 259 records were collected via email and 29 via Twitter. Overall, the model showed promising results. However, the authors point out that these results do not necessarily imply good actual forecasting performance, and that the generality of the model will become clearer as forecasts over longer periods are included. The study presented in [5] plans to develop a low-cost system for the detection, recognition, and classification of various types of incidents in coastal areas through UAVs. The incidents include everything from oil spills to the appearance of dangerous sea creatures, such as the Portuguese Man-of-war. The first part of the solution is composed of a CNN to process the images obtained by the UAVs, and the second part is a mobile application to manage the UAVs. The solution is still being developed by the authors.

In sum, despite the similarity between the present work and the cited works, due to the physical characteristics of Portuguese Man-of-war being similar to that of other jellyfish, almost none of the models could be applied in the classification problem presented here, because they have another function. In [14] and [15], the goal is to detect jellyfish, i.e., the intention is to find where and how many animals are in the image. In [6], the goal is also to detect jellyfish and classify them, but using underwater images and among different species than the one studied here. In [19], the function of the model is to predict the appearance of jellyfish in a region. In [5], the model has a broader objective than the one presented here, but still uses images of Portuguese Man-of-war. However, it is not possible to validate whether the model can serve as a basis for the problem of this study as it is still under development. Furthermore, the use of underwater images or aerial images for training is a very relevant factor, since most of the images of jellyfish obtained from Instagram are of the animals stranded on the sand, making them quite distinct. Moreover, data collection, in most of the works cited, was performed in an active manner, where a job is employed to perform this task. In the present work, the task is to analyze a large amount of data present on social media, specifically Instagram. Such data is not collected specifically for this purpose, but is a useful source of information about biodiversity, and is cheap and easy to collect.

This section describes the methods used for image collection and implementation of the neural networks used in this study.

The results obtained by the different CNNs are presented in the following tables. The tables have the same format: the first three rows represent the architectures without pre-training, and the last three rows represent the architectures pre-trained with the ImageNet database.

Analyzing the results presented in Table 2 and Table 3, it is possible to observe that the networks pre-trained with the ImageNet database obtained considerably better results than networks without any pre-training. Among the networks without any pre-training, the best performance was obtained by the ResNet50 network, with about 83% accuracy, followed by InceptionV3, with 82%. The VGG-16 network had the worst performance, with results well below the other networks, with only 74% accuracy. Among the pre-trained networks, all obtained very consistent metrics in both classes, demonstrating a generalized quality of the models. Although very close, the best performance was obtained by the ResNet50 network, with almost 95% of accuracy Therefore, its results will be analyzed in more depth below. The pre-trained ResNet50 obtained accuracy in the PMW class slightly better, with an error rate of about 4.30%, while the not-PMW class obtained an error rate of about 6.18%, i.e., almost 2% different from the previous one. There are 53 false negatives in the PMW class between them, 11 have a person in the photo, and 14 have the presence of a human hand, either holding the Portuguese Man-of-war or not. The presence of a human may be a factor impacting the classification since the not-PMW class is also composed of images of people. Besides, the author has difficulty classifying 12 of these images, because they are pictures taken far away from the animal, with low quality, or only of its tentacles. Figure 4 shows an example of a false negative.

There are 76 false positives, 46 are Velella velella images, 25 are jellyfish images, 5 images are illustrations, and one image is of a handicraft. Since the test base is approximately balanced among the image types, the highest error rate for the not-PMW class model is in the Velella velella and jellyfish images, which add up to more than 93% of the total errors. Figure 5 shows a sample of the false positives.

A possible attempt to improve the performance of the CNN is providing higher examples of images that are part of the groups mentioned with higher occurrence in the false positives and false negatives of the PMW class. However, it is necessary to observe if this change does not cause negative impacts on the classification of other images and consequently worse results.

This study presents the evaluation of the use of convolutional neural networks to automate a step of the methodology proposed by [17]. Three different types of architectures were evaluated to measure their performance in a binary image classification task for the identification of the Portuguese Man-of-war. The experiments performed show that CNNs present promising results for this task, reaching an accuracy of over 94% by the best performing network, the ResNet50 pre-trained on the ImageNet database. Therefore, these obtained results show that the application of CNNs is suitable and can be highly beneficial for the collection of evidence of images of Portuguese caravels on Instagram. Some interesting future directions of this research include the application of different techniques to further improve the results, such as other fine-tuning, data augmentation, and regularization approaches, as well as, the refinement and expansion of the database, focusing on the treatment of the types of image that represented the higher error rate. In addition, other reference architectures can also be evaluated on this problem. Another proposal for future work is to analyze the automation of other parts of the publication of the Istagram. A complete flow could consist of a system that periodically and automatically collects data, analyzes the data present in each publication, both images and text, and classifies it as a valid or invalid record for the Portuguese Man-of-war. For valid records, the system saves the relevant information in a database. It is also interesting to develop a dashboard that allows users to manage the data and the system.