Rapid Image Labeling via Neuro-Symbolic Learning

Reducing the human effort in image labeling has become increasingly important in recent years with the advent of deep learning, which requires a massive amount of labeled data to train a model. There has been a continuous effort to develop automated solutions for image labeling. The basic idea is to train a model with some labeled data and automatically assign labels to new data samples without further human involvement. Among these existing methods, fully-automated methods have attracted significant attention and achieved promising performance. Some methods exploit the similarity between unlabeled and label images (Das et al., 2020). Some methods resolve the problem of data scarcity by creating representations of images using auxiliary information such as corresponding captions (Quattoni et al., 2007), meta-data (McAuley and Leskovec, 2012), or pseudo-labels generated by other models (Titano et al., 2018). Despite the promising performance of these methods, they often lack generalizability to domains where data acquisition is challenging, as the models usually require a substantial amount of data to learn the knowledge. Thus, some semi-automated labeling approaches use human efforts in the training or inference process to provide the information needed for training (Suchi et al., 2019) or a coarse initial label (Grady and Funka-Lea, 2004). Compared with existing work, our approach uses inductive logic learning to infer logic rules from a small amount of human-annotated data to label images.

Active learning is widely adopted to get humans involved in the labeling process iteratively while minimizing the amount of data to be labeled by humans. Active learning methods select the data samples that can benefit the model most in each iteration and request humans to label them to push the usage of human efforts to the minimum. To determine which data sample to be labeled by humans, some approaches use probability models and prioritize the data samples with high prediction inconsistency (Zhang et al., 2008; Gao et al., 2020; Guo et al., 2021; Ning et al., 2022), some depend on the vectorized representation from deep learning models (Ho et al., 2020; Sinha et al., 2019; Zhang et al., 2022), and some calculate the low-rank matrix representation for both labeled and unlabeled data to calculate the informativeness (Wu et al., 2017).

Though the active learning strategy can optimize data selection to reduce human effort, it often requires many iterations to achieve a reasonable labeling accuracy. Thus, it remains too expensive for labeling tasks where time is precious for domain experts such as clinicians. Snorkel (Ratner et al., 2017) enables users to explicitly embed their domain knowledge by creating labeling functions. However, the labeling functions are either written in programming languages or special declarative functions defined by the author of Snorkel, causing huge overhead effort in learning the labeling function grammar. Our work combines interactive learning with an inductive logic learner, generating logic rules to classify images. The generated rules are represented with simple logic and descriptive predicates, which are easy for users to read and edit.

There has been a growing interest in combining neural networks with symbolic methods (Mao et al., 2019; Yi et al., 2018a; Hudson and Manning, 2019; Amizadeh et al., 2020; Murali et al., 2022). Here, the term neuro refers to artificial neural networks or connectionist systems, while the term symbolic refers to AI approaches that perform explicit symbol manipulation, such as term rewriting, graph algorithms, and formal logic. There are different ways of combining neural network modules with symbolic learning. Following the categorization in a recent survey (Sarker et al., 2021), our method belongs to a cascading neuro-symbolic paradigm that extracts latent patterns from input data using a neural system and then feeds them into a symbolic reasoner for final prediction.

Existing approaches in this category include NS-VQA (Yi et al., 2018b), NS-CL (Mao et al., 2019), and FO-SL (Murali et al., 2022). NS-VQA (Yi et al., 2018b) firstly parses an image to a structural scene representation with Mask R-CNN and ResNet-34 and converts a natural language question into a query program with an LSTM model. Then it uses a symbolic executor to run the program on the scene representation to obtain the answer to the given question. NS-CL (Mao et al., 2019) adopts a similar approach as NL-VQA but learns the feature vector representation of an object from question-answer pairs, instead of extracting them directly with pre-trained models. FO-SL (Murali et al., 2022) represents images in first-order logic and uses an SAT solver to solve visual discrimination puzzles.

Unlike existing neuro-symbolic learning approaches in this category, we are the first to use inductive logic learning as the symbolic method for rule inference. In this way, we can explicitly model the logic of labeling rules. Furthermore, our approach is also the first to apply neuro-symbolic learning to image labeling.

To the best of our knowledge, our work is the first that integrates active learning with Inductive Logic Learning (ILL). Existing research in ILL has focused on improving the learning algorithm for better efficiency and scalability. There are only a few that investigates the interactivity of ILL in the 1990s (De Raedt and Bruynooghe, 1992; De Raedt, 1992; Bergadano et al., 1993). Specifically, De Raedt et al. (De Raedt, 1992; De Raedt and Bruynooghe, 1992) propose an interactive paradigm in which the inductive learner asks a yes/no question about the correctness of a learned rule. If a user answers no, the learner will backtrack and learn a new rule. Bergadano et al. (Bergadano et al., 1993) propose to prompt users for new counter-examples to refute an incorrect logic program, but users need to manually design counter-examples from scratch. More recently, Sivaraman et al. (Sivaraman et al., 2019) present an interactive inductive logic programming approach to infer rule-based code patterns for code search. However, this approach only allows users to mark some search results as correct or incorrect for pattern refinement. None of these four approaches have an active learning component (i.e., a data selection algorithm) to carefully rank and select data samples for user inspection and correction.

A visual attribute extractor processes a given image and extracts the basic, low-level attributes of the image that are useful and relevant to the labeling task. The visual attributes can be object types in an image, the relationships between the objects, and an object’s properties (e.g., size, number). This work mainly uses pre-trained perception models as the visual attribute extractors, detailed in Section 4.3. But one can also use a traditional feature extractor such as SIFT (Lowe, 1999) to extract visual attributes. The visual attribute extractors are designed as pluggable components in our approach. For different labeling tasks, we use different pre-trained models to extract visual attributes related to the labeling task. This design increases the flexibility of our approach to be reused for new labeling tasks.

Given its capability to learn from a small amount of data, we use First Order Inductive Learner (FOIL) (Quinlan, 1990) to infer labeling rules. Furthermore, the declarative nature of logic rules makes them easy to be understood and refined by human labelers based on their domain knowledge. FOIL is initially designed to learn a logic rule with pre-defined predicates to distinguish a set of positive and negative examples. In our design, each predicate represents one trait of a visual attribute. The original FOIL algorithm can only infer logic rules with variables, which lacks the expressiveness for logic rules with constant values. Therefore, we extend FOIL to support the inference of constant values. As a consequence, this increases the search space exponentially. To address this challenge, we design several inductive biases, such as a TF-IDF-based heuristic, to improve search efficiency (detailed in Appendix A).

In our approach, a labeling rule is defined in a disjunctive normal form with $k$ clauses, as shown below.

$C_{1},...C_{m}$ denote clauses and $L$ denotes the label. If at least $1$ clause is satisfied, an image is labeled as class $L$. A clause is defined as,

where $p_{1},...p_{k}$ are logic predicates for visual attributes. A clause is a conjunctive normal form with $k$ predicates. Hence, a clause is satisfied if and only if all the predicates are satisfied. In this work, we design a set of primitive predicates for different kinds of visual attributes, as shown in Table 1. For example, in the traffic scenario labeling task, a target image class, “highway”, can be inferred based on the types of objects on the road, e.g., “trucks”. An example labeling rule for “highway” images can be:

where X is an input image, A and B are objects detected by a pre-trained object detection model. This rule means that if there are no pedestrians or there exist more than five trucks, the image is classified as “highway”. In practice, users can redesign the predicates, e.g., by removing irrelevant predicates and adding domain-specific predicates based on the characteristics of each labeling task to improve the efficiency of inferring logic rules.

Algorithm 1 describes how to infer labeling rules with inductive learning. For each image label, our algorithm takes the set of positive examples $T^{+}$ and the set of negative examples $T^{-}$ as input and infers a logic rule. It also allows human labelers to specify which clauses must be included ($I$) or excluded ($E$) based on their domain knowledge. It first adds must-include clauses into the rule $R$ (Line 2). Then, it keeps searching for possible clauses until $T^{+}$ is empty (Lines 3 to 15). If the clause does not need to be excluded (Line 11), the algorithm removes the positive examples which contain all visual attributes in the clause from $T^{+}$ (Line 12), and then adds the clause to the rule (Line 13). The algorithm initializes a negative set $T_{i}^{-}$ as $T^{-}$  (Line 5) and a set containing all possible predicates from the set of positive examples $S$ (Line 6) before the first iteration of the inner loop. To find a possible clause (Line 7 to 10), the algorithm constantly selects predicates with the maximum information gain from $S$ and adds into the clause (Line 8) until $T_{i}^{-}$ is empty (Line 7). The information gain of each predicate is defined below:

where $T_{i}^{+}$ and $T_{i}^{-}$ denote the set of positive examples and the set of negative examples before adding the new predicate $S_{i}$. $T_{i+1}^{+}$ and $T_{i+1}^{-}$ denote the set of positive examples and the set of negative examples after adding $S_{i}$. Then in each iteration in the inner loop, $T_{i}^{-}$ is redefined to a set that removes the negative examples which contain all visual attributes in the clause from $T^{-}$(Line 9). The loop continues until it finds a labeling rule that matches all labeled images in a given class (i.e., positive examples) while not matching labeled images in other classes (i.e., negative examples).

Due to the ambiguity and incompleteness of the small amount of training data, the inductive learning module may not learn the best labeling rules in one pass. We propose to use active learning to improve the performance of inductive learning by iteratively soliciting more human labels.

In each iteration of active learning, when selecting data samples, our goal is to choose the data with the most information and variety to reduce data usage and improve performance. We propose a multi-criteria data selection strategy to achieve this goal.

To answer RQ1, we measure the image labeling accuracy and efficiency of Rapid on four different image labeling tasks—two from highly specialized domains and two from common domains. Section 4.3 describes the four tasks and datasets. To represent the condition of learning from limited training data, for each task, we bootstrap Rapid with only $3$ randomly sampled data instances with labels to learn the initial set of rules. In each following iteration, the active learning module selects $3$ images and corrects their labels if wrong to refine the learned labeling rules. The choice of $3$ is to simulate the rapid, incremental feedback from users. This process continues until the $20$th iteration. Thus, for each task, Rapid is trained with in total $60$ images. We compare Rapid with $6$ baselines. Section 4.4 describes these baselines and their training procedures.

To answer RQ2, we measure the degradation of image labeling accuracy and efficiency when abating the human feedback mechanisms in Rapid. Rapid supports two kinds of human feedback—(1) directly editing the labeling rules generated by Rapid and (2) fixing incorrect labels inferred by Rapid and supplementing new labels. Thus, we create three variants of Rapid—Rapid without rule editing (Rapid${}_{no\text{-}edit}$), Rapid without any labeling correction (Rapid${}_{no\text{-}al}$), and Rapid without any kinds of feedback (Rapid${}_{no\text{-}feedback}$).

RQ3 aims to measure the effectiveness of adopting inductive logic learning for image labeling. To answer RQ3, we create four variants of Rapid by replacing the inductive logic learning module in Rapid with three statistical models—SVM, random forest, and XGBoost (Chen and Guestrin, 2016)—and a neural network. The design of these variants is inspired by existing frameworks such as Snorkel (Ratner et al., 2017) and Concept Bottle Network (Koh et al., 2020), which use statistical models to make the final prediction based on symbolic representations extracted from raw input data. For the neural network variant, we adopt the design of the fully connected layers in ResNet-18 (He et al., 2016).

To answer RQ4, we compare Rapid with three alternative data selection strategies—random selection, selection with only the informativeness criterion, and selection with only the diversity criterion. We use image labeling accuracy, as well as the hit rate of misclassified data, as the evaluation metrics. Specifically, the hit rate is defined as the percentage of selected data samples that Rapid mislabels in the current iteration and thus is worth fixing. A higher hit rate indicates better effectiveness of data selection.

Since Rapid is designed as a human-in-the-loop approach, Rapid needs to keep soliciting feedback from human experts to refine the labeling rules. It is expensive to recruit human participants to provide feedback, especially in the two highly specialized labeling tasks that require domain experts such as ophthalmologists and ornithologists. Therefore, we develop an automated script to simulate human feedback based on the ground truth data. To simulate label corrections, our script compares the ground truth label of each image with the labels inferred by Rapid and automatically fixes the incorrect labels. To simulate human edits to labeling rules, the authors first manually constructed a set of high-quality labeling rules based on their own knowledge and the information shared on professional websites. In each iteration of the training process, our script compares the labeling rule inferred by Rapid with the corresponding manually curated rule. Our script then replaces the first inconsistent clause with the clause in the manually curated rule. In all experiments, we restrict the simulation script to only edit one clause per iteration to simulate the incremental editing process of human labelers.

Rapid is designed for image labeling tasks in highly specialized domains. Therefore, we first select two datasets—Glaucoma Diagnosis and Bird Species Labeling—from highly specialized domains. To test the generalizability of Rapid, we construct the two datasets on general domains, including traffic scene labeling and occupation labeling, by searching on Google and Flickr.

Glaucoma Diagnosis. Given a color fundus image, this task requires labeling the eye in the image to be either normal or diseased. We combine the color fundus images from three datasets, Drishti-GS (Sivaswamy et al., 2015), RIM-ONE_r3 (Batista et al., 2020), and REFUGE (Orlando et al., 2020). We have $116$ images of glaucomatous eyes and $189$ images of normal eyes with both glaucoma diagnosis and structure segmentation. We use a pre-trained model called BEAL (Wang et al., 2019b) as the visual attribute extractor to obtain the segmentation of eye fundus structures in the images. The visual attributes designed for this task are the diameter, area, and cup-to-disk ratio calculated based on the segmentation results.

Bird Species Labeling. Given a bird image, this task requires labeling the bird species. We use the Caltech-UCSD Birds-200-2011 (CUB 200-2011) dataset (Wah et al., 2011), containing $11,788$ bird images annotated with $200$ bird species and $312$ attributes that describe each body part of a bird, e.g., wing color, tail shape, etc. Following the experiment settings in Koh et al. (Koh et al., 2020), we use $112$ out of the $312$ attributes, and randomly choose three bird species to label in our experiments. We use the pre-trained concept models from Koh et al. (Koh et al., 2020) to extract the visual attributes.

Occupation Labeling. Given an image of a person, this task requires labeling the occupation of the person. For this task, we build a dataset containing $300$ images of three occupations—chef, farmer, and teacher. Each occupation has $100$ labeled images. We use a pre-trained object detection model (Anderson et al., 2018) to detect objects in the images and use the type (glasses, long hair, kitchen, etc.), color, and overlapping relationship between them as visual attributes.

Traffic Scene Labeling. Given a road image, this task requires labeling the traffic scene of the image. For this task, we build a dataset containing $420$ images of three traffic scenes—mountain road, highway, and downtown. Each traffic scene has $140$ labeled images. We use a pre-trained object detection model called DETR (Carion et al., 2020) to detect the objects in the images and use the position, color, and type (e.g., pedestrian, truck, car, etc.) of the objects and the overlapping relationship between them as visual attributes.

For RQ1, we compare Rapid with four image classification neural network baselines—ResNet-18 (He et al., 2016), ResNet-34, ResNeXt-32 (Xie et al., 2017) and Inception-V3 (Szegedy et al., 2016))—and an active learning baseline called CEAL (Wang et al., 2017). We further compare Rapid with GARDNet (Mahrooqi et al., 2022) on the Glaucoma diagnosis task since GARDNet is specially designed for this task. To represent the condition of learning from limited training data, in each task, we randomly sample $30$ training data per class label to finetune the baseline models. That is a total of $60$ training samples for the Glaucoma Diagnosis task since it only has two class labels and $90$ for the other three tasks since they have three class labels. For the first five baselines, we first pre-train them on ImageNet (Deng et al., 2009) and then fine-tune them on the four datasets, with training sets in the same size as training Rapid. For GARDNet, we obtained the trained model from its original paper and then fine-tuned it on our Glaucoma diagnosis dataset.

For RQ2, we build three variants of Rapid—Rapid without editing rules by users (Rapid${}_{no\text{-}edit}$), Rapid without labeling correction or new labels (Rapid${}_{no\text{-}al}$), and Rapid${}_{no\text{-}feedback}$, Rapid without feedback when selecting training samples. Similar to the training setting of Rapid, both Rapid${}_{no\text{-}edit}$ and Rapid${}_{no\text{-}al}$ are initially trained with $3$ randomly selected images. In the following interactions, Rapid${}_{no\text{-}edit}$ only fixes incorrect labels in the $3$ images selected by the multi-criteria active learning algorithm per iteration but does not apply any direct edits to the inferred rules. By contrast, Rapid${}_{no\text{-}al}$ only makes direct edits to one clause of the inferred rules per iteration but does not fix any incorrect labels. Rapid${}_{no\text{-}feedback}$ does not use any active feedback from users. It is trained with randomly sampled images in various numbers (e.g., 3, 6, 9, etc.) ahead of time without soliciting further human feedback.

For RQ3, we create four variants of Rapid by replacing the inductive logic learner with other machine learning methods, including SVM, random forest, gradient boosting, and neural network. For these variants, we use a feature vector generated with the extracted visual attributes to represent each image. The feature vector is constructed in the same way as calculating the diversity criterion in Section 3.3.3. We repeat each training three times and compute the average performance of each baseline.

Table 2 shows the image labeling accuracy of Rapid in the four labeling tasks from different domains in comparison to the fine-tuned models. Rapid outperforms all baselines on the two highly specialized domains (Glaucoma Diagnosis and Bird Species Labeling) by $11.75\%$ to $12.03\%$. Specifically, Rapid achieves a high accuracy of $85.52\%$ and $86.11\%$ in these two tasks, respectively. The result shows Rapid can effectively infer accurate labeling rules in highly specialized domains with a small amount of training data. For more details about the labeling rules, such as examples and statistics of the optimal rules, and how the labeling rules change over iterations, please refer to Appendix B.

For the two common domains, Rapid achieves comparable or worse accuracy. Specifically, Rapid achieves an accuracy of $83.33\%$ in the traffic scene labeling task, while the best baseline model achieves $92.86\%$ accuracy. For the occupation labeling task, the accuracy of Rapid is $88.33\%$ while the best baseline model achieves $97.78\%$ accuracy. This result is not surprising since the baseline models are pre-trained on ImageNet, which includes a considerable number of images similar to the ones in these two common tasks. Thus, the baseline models have already learned from many similar cases during the pre-training process.

Figure 3 shows the image labeling accuracy of Rapid during the training process. At the $4$th iteration, with only $12$ training samples, Rapid has already achieved a reasonable accuracy—$85\%$ in Glaucoma diagnosis, $70\%$ in occupation labeling, $65\%$ in traffic scene labeling, and $64\%$ in bird species labeling. Within the $13$th iteration, Rapid has achieved the peak accuracy on all four tasks. Besides, during the training process, the performance of Rapid is stably improving. The result shows Rapid can effectively learn and refine labeling rules within a small number of iterations.

Table 3 shows the image labeling accuracy and the number of iterations to achieve the optimal accuracy of Rapid in comparison to its invariants after ablating each feedback mechanism. Overall, Rapid always achieves the highest accuracy with the smallest number of iterations ($10.25$ on average). Rapid${}_{no\text{-}feedback}$ has the worst performance with the largest number of iterations ($25.25$ on average). Without direct rule editing, Rapid${}_{no\text{-}edit}$ takes significantly 4X more iterations to achieve a comparable accuracy on the Glaucoma diagnosis task and has significantly lower accuracy on the other three tasks ($11.87\%$ accuracy decrease on average).

Though Rapid${}_{no\text{-}al}$ can achieve the same final accuracy as Rapid, it takes $6$ more iterations in the bird species labeling task. It is interesting to observe that Rapid${}_{no\text{-}al}$ takes the same or even fewer iterations in three tasks. This is because the user simulation script always makes the right edit based on the ground-truth labeling rule each time. When using active learning together with rule editing, Rapid will regenerate the label after receiving new labels in each iteration. These newly generated rules may deviate from the ground-truth rules, therefore leading to more iterations.

Compared with the three variants, Rapid achieves the largest performance gain in the bird species labeling task. This performance gain can be largely attributed to the inherent learning challenge of the dataset. For example, birds of the same species are of great variety (e.g., a Kentucky Warbler can exhibit eight distinct wing color variations.). Thus, this increases the difficulty of learning a proper labeling rule to define what a certain bird species look like from such a small training dataset. On the other hand, by editing the rules, experts can directly embed their expert knowledge into the labeling rules, leading to a huge performance improvement.

Figure 4 shows the image labeling accuracy of Rapid and its variants over iterations in the bird species labeling task. For ease of comparison, we show the accuracy of Rapid${}_{no\text{-}feedback}$ over the number of randomly sampled training samples it uses. In other words, for Rapid${}_{no\text{-}feedback}$ in Figure 4, $1$ in the x-axis means training with $3$ samples, $2$ means training with $6$ samples, $3$ means training with $9$ samples, so on and so forth. Overall, Rapid achieves the highest image labeling accuracy ($86.11\%$) with only $12$ iterations and $36$ images in total. While Rapid${}_{no\text{-}al}$ achieves the same accuracy, it takes $7$ more iterations and thus $21$ more images to reach this accuracy. In this task, both Rapid${}_{no\text{-}edit}$ and Rapid${}_{no\text{-}feedback}$ takes significantly more iterations to achieve the final accuracy.

Table 4 shows the comparison of accuracy between Rapid and variants replacing the inductive logic learner with statistical and neural network models. Rapid outperforms all variant models in all four tasks by $6.02\%$, $15.74\%$, $8.89\%$ and $3.57\%$, respectively. The results show the great capability of the inductive logic learning model in learning image labeling tasks under a low resource setting.

Though Rapid_RandomForst does not achieve an accuracy as high as Rapid, it is worth mentioning that Rapid_RandomForst outperforms all the baseline models in Table 2, in the two highly specialized domain tasks. This implies that our pipeline, similar to Concept Bottleneck Models (Koh et al., 2020), which disentangles the perception process (Visual Attribute Extraction) and the learning process, has great capability in highly specialized image labeling tasks.

Figure 5 shows the comparison of accuracy and hit rate between four data selection strategies in active learning in the traffic scene labeling task. Recall that the hit rate is defined as the percentage of selected data samples that Rapid mislabels in the current iteration and thus is worth fixing. A higher hit rate indicates better effectiveness of data selection. Due to the page limit, we only showcase the experiment results on this task. Figures for the other three tasks have been provided in Appendix C.

Among the four data selection strategies, the multi-criteria strategy achieves the highest image labeling accuracy ($73.81\%$) with the smallest number of iterations ($3$). By contrast, using the diversity criterion alone takes $19$ iterations ($6$X more) to achieve similar accuracy. Both using the informativeness criterion alone and random selection achieve lower accuracy, $50.00\%$ and $67.86\%$. Yet compared with random selection, the informative selection still helps, achieving significantly higher accuracy within fewer iterations.

In the traffic scene labeling task, random selection, single diversity criterion, single informativeness criterion, and multi-criteria achieve $42.22\%$, $40.00\%$, $71.11\%$, and $76.67\%$ average hit rate over the iterations. Combining informativeness and diversity, our multi-criteria strategy achieves the highest hit rate in this task. The single informativeness strategy maintains the highest hit rate at the beginning of the training process, which proves the effectiveness of our informativeness metric designed in Section 3.3.2. The results on accuracy and hit rate imply that the two metrics are complementary to each other. With the informativeness metric selecting training samples with high information and the diversity metric choosing representatives from the selected samples, Rapid can achieve high accuracy in a small number of iterations.