Concept Induction using LLMs: A user experiment for assessment

In recent years, Explainable AI (XAI) has emerged as a critical area of research and development within the field of artificial intelligence [5]. The importance of XAI lies in its ability to enhance transparency and understanding of AI systems, enabling users and stakeholders to comprehend how AI arrives at its decisions. This transparency is crucial for establishing trust in AI systems that may otherwise appear as black boxes or opaque entities [28], particularly in domains where AI decisions directly impact human lives. Additionally, XAI serves as a means to debug and enhance network architectures to achieve improved outcomes.

Currently, significant research in XAI is centered on model-agnostic post-hoc algorithms [6]. These algorithms aim to provide understandable insights into how a pre-existing model generates predictions for any given input without compromising the model’s accuracy. They address the challenge of making complex AI models, such as deep neural networks, more interpretable by humans. Techniques like gradient-based saliency maps [32] and perturbation-based methods [16] have been used extensively to identify important features within AI models. Gradient-based methods highlight which parts of the input data influence the model’s predictions the most, whereas perturbation-based methods involve modifying the input data to observe changes in the model’s output. These approaches employ visual explanation maps to understand the decisions made by a deep learning (DL) model. However, due to challenges in visualizing these explanation maps and considering their susceptibility to adversarial attacks, there has been a shift away from these models [9].

One promising approach within XAI is the use of concept-based models for generating explanations [17, 26]. Concept-based models incorporate explicit representations of concepts or knowledge units, making them inherently more interpretable than traditional black-box models. By leveraging these explicit concepts, XAI techniques can provide explanations that align with human intuition and reasoning. However, generating meaningful concepts from input data remains a challenge, as it requires context-specific explanations that can accurately depict the model’s behavior while remaining understandable to humans.

In previous work [31], a post-hoc explainability strategy was proposed using concept induction [20, 21], and in [34] it was investigated how well explanations created via concept induction ”make sense” for humans. Concept induction involves creating complex Description Logic class descriptions (TBox axioms) based on instance examples (ABox data) using deductive reasoning algorithms over Description Logic knowledge bases. The authors in [34] demonstrated that concept induction can be utilized to explain data differentials in machine learning classifications, although human generated explanations are still better. Their method utilizes the Wikipedia category hierarchy [30] as the background knowledge, and the ECII (Efficient Concept Induction from Individuals) heuristic concept induction system [29] was used for explanation generation. A survey was conducted through Amazon Mechanical Turk to assess how meaningful the generated explanations are to humans.

Expanding upon this framework, our goal is to explore the feasibility of replacing the ECII model with a Large Language Model (LLM) to produce explanations that remain meaningful and coherent. The objective is to identify ”good” concepts that make sense to humans and can later be validated by mapping them with a Deep Neural Network (DNN) to accurately describe what neurons perceive. We utilized the GPT-4 [2] model to generate meaningful explanations for a specific scene classification task, which was done using a logistic regression algorithm that classified images into scene categories based on semantic tags of objects present in each image. The explanations are generated using Prompt Engineering [10] via the OpenAI API. Unlike logical-deduction-based systems such as ECII, which are limited by background knowledge, an LLM like GPT-4 can leverage its common-sense reasoning capability and vast domain knowledge to produce more comprehensive concepts. In the mentioned [34], the quality of explanations generated by concept induction was assessed and found to be more meaningful than semi-random explanations but less accurate than human-generated (gold standard) ones. Our objective is to evaluate the extent to which explanations generated by LLMs align with human-generated explanations and potentially surpass the concept induction system in terms of accuracy and comprehensibility.

Concept induction is a symbolic reasoning task that can be done using provably correct [20] or heuristic [29] deduction algorithms over description logic knowledge bases. In this paper, we are attempting to make use of pre-trained LLMs to produce results that are comparable to or even better than those obtained from a concept induction system. In other words, we are making use of an LLM to do better than a symbolic-reasoning-based algorithm, at least in a specific setting. As such, our work contributes to research on neurosymbolic artificial intelligence.

The rest of the paper is structured as follows. In Section 2 we review related literature. In Section 3, we give a detailed outline of our approach. In Sections 4 and 5, we discuss the evaluation method and the resulting outcomes. In Section 6, we discuss future work and conclude.

With the increasing need for explainable AI systems, various methods have been proposed to achieve explainability, each with its strengths and limitations. Interpretable models such as decision trees [33] and linear regression [15] inherently offer transparency by design, enabling straightforward explanations based on the model structure. However, these models may lack the complexity and performance of advanced techniques like deep neural networks. With growing demands for more explainable machine learning (ML) [14], there is a rising need for post-hoc methods that can be applied without retraining or modifying the network.

Post-hoc explanation techniques like LIME [27] and SHAP [23] provide ways to explain complex black-box models by approximating local behavior. They generate explanations at the instance level, offering insights into model predictions for specific input features or data points. Another local explanation method uses gradient-based saliency maps [3, 32] to highlight the importance of each pixel in an image for the output result. However, these methods can lack trustworthiness and exhibit random biases [12, 4], as explanations are often valid only for a specific data point that may vary significantly across datasets.

Methods like TCAV [17] focus on global explanations by utilizing high-level concepts to estimate their importance for predictions, requiring human-provided concepts. Alternatively, ACE [13] leverages image segmentation and clustering to curate automated concepts that may lead to some information loss. Other approaches such as Concept Bottleneck Models (CBM) [19] and Post-hoc CBM [35] map DNN models to human-understandable concepts but often rely on hand-picked concepts, highlighting the need for automated methods to generate higher-level concepts.

One approach involves using background or domain knowledge to generate human-understandable explanations via concept induction [34, 8]. However, these methods are constrained by their background knowledge and fail to capture common-sense interpretations evident to humans during explanation generation. Leveraging LLMs has the potential to bridge this gap by automating concept generation, utilizing minimal text-based object information. Another study [25] employing a similar approach utilizes GPT-3 with a few-shot method to produce automated concepts. Although aiming to reduce human involvement, this method requires a filtering process to refine initial concepts, based on numerical analysis.

In the field of XAI, explanations cater to end-users or system developers, but in all cases, it remains crucial that explanations make sense to humans. Our study explores the potential of LLMs in generating explanations through human evaluation, aiming to bridge the gap between complex AI systems and human understanding.

Our approach and evaluation setting is essentially the same as in [34], however instead of their comparison of explanations generated by (1) humans, (2) concept induction, and (3) a semi-random process, we compare (1) human, (2) concept induction, and (3) GPT-4 prompting. We went into the study with the hypothesis that explanations produced by GPT-4 would outperform those produced by concept induction in terms of ”meaningfulness to humans,” but that they would still not be as good as the human-generated gold standard.

To evaluate the concepts generated from GPT-4 we ran a study through Amazon Mechanical Turk using the Cloud Research platform. Our goal was to assess the quality of LLM explanations (i.e., GPT-4 explanations) compared to both human-generated (”gold standard”) explanations and ECII explanations. The study protocol was reviewed and approved by the Institutional Review Board (IRB) at Kansas State University and was deemed exempt under the criteria outlined in the Federal Policy for the Protection of Human Subjects, 45 CFR §104(d), category: Exempt Category 2 Subsection ii 7.

We recruited 300 participants through Mechanical Turk, compensating each participant with $5 for completing the task, which was estimated to take approximately 40 minutes (equivalent to $7.50 per hour based on the minimum legal wage in the USA). Based on the previous study [34], we aimed for a sample size of at least 89 unique participant judgments per trial to estimate the parameters (medium effect size of f2 = 0.15 and 95% power) of the Bradley-Terry model [7], which is used to evaluate the survey results. This required collecting data from 300 participants, resulting in a total of 100 observations per trial after accounting for potential exclusions.

Across all questions, each participant encountered three types of explanations, although only two explanation types were compared in any given question. Each participant was asked to choose the more accurate explanation using a two-alternative forced choice design. For each pair of image sets, participants answered three questions comparing (1) Human versus ECII explanation; (2) Human versus LLM (GPT-4) explanation; and (3) LLM versus ECII explanation. For each pair of image sets (A and B), a given participant completed all three comparisons.

The 45 pairs of image sets in this study resulted in a total of 135 unique target questions. Participants were randomly assigned to 15 image sets (45 questions in total), ensuring that image sets were counterbalanced across participants to receive an equal number of responses.

For all image sets, ECII explanations and Human ”gold standard” explanations were created in a previous study [34]. In this work, we generated LLM (GPT-4) explanations following the method described in Section 3. To form the ECII explanations, the object tags of the images were provided to the ECII algorithm, then the seven highest-rated unique concepts were selected based on the ranking of the F1 score. Human ”gold standard” explanations were crafted by presenting image sets (without object or scene category tags) to three human raters, selecting concepts unanimously mentioned by all three, then by two raters, and finally filling in randomly selected concepts until seven unique concepts were reached.

In addition to the 45 image sets, five ”catch trial” image sets were used to verify participant attention. These catch-trial image sets included two types of explanations: human explanations generated similarly to other human gold standard explanations, and explanations consisting of completely random concepts generated from a word generator to serve as obviously inaccurate explanations.

After providing consent, participants received brief training on the task, including instructions on how concepts and explanations were defined in the study. They then began answering questions, with the 50 questions (45 assigned targets and 5 catch trials) presented in random order. Figure 2 illustrates the stimuli presentation and response options shown to participants.

Prior to analysis, participant responses to catch trials were evaluated, and participants who failed more than one catch trial were excluded from further analysis. Among the 300 participants, 253 did not fail any catch trials, while 22 participants failed exactly one trial, and 35 participants failed more than one trial. The 35 participants who failed multiple trials were excluded from all subsequent analyses, resulting in a total of 265 participants included in the analyses.

Across all image sets, human explanations were overwhelmingly preferred over ECII explanations (chosen 3282 times versus 693 times; 83% preference) and over LLM (GPT-4) explanations (chosen 2762 times versus 1213 times; 69% preference). Additionally, LLM explanations were preferred over ECII explanations (chosen 2514 times compared to 1461 times; 63% preference). See Figure 3.

Participants’ pairwise judgments were utilized in a Bradley-Terry analysis [11] to derive ”ability scores” for each type of explanation, reflecting the extent to which each explanation type was preferred by participants. The Bradley-Terry model uses data where entities are compared pairwise, and the outcome (win/loss, preference ranking, etc.) is observed. From these comparisons, the model estimates the abilities $\theta_{i}$ such that the observed outcomes are statistically likely. The estimation process typically involves fitting the model to the pairwise comparison data to find the best-fitting values of $\theta_{i}$ for each entity. These estimates reflect the latent abilities or strengths of the entities relative to each other. Ability scores were calculated for each of the 45 image set pairs based on the pairwise comparison data (win/loss) for each type of explanation. The analysis of these ability scores demonstrated that human explanations had the highest scores (M = 1.77, SD = 0.978), followed by LLM explanations (M = 0.724, SD = 1.16), with a significant overall difference (F(2) = 46.28, $p<0.001$, $\eta^{2}=0.41$). Here, ECII explanations served as the reference point and were set to 0, with the ability scores for human and LLM explanations indicating their preference over ECII explanations.

A post hoc analysis using Tukey’s Honestly Significant Difference (HSD) test [1] was conducted to determine which specific group means are significantly different from each other. When comparing multiple group means, the Tukey post hoc test is preferred over multiple t-tests [18] because it adjusts for multiple comparisons, controlling the overall Type I error rate [24]. Conducting multiple t-tests increases the risk of false positives, while the Tukey test maintains the integrity of statistical conclusions by adjusting the significance levels appropriately. This test confirmed significant differences in ability scores between human vs. ECII explanations and human vs. LLM explanations (both $p<0.0001$), as well as between LLM vs. ECII explanations ($p=0.0004$) (Table 2). These low p-values indicate that the observed differences in ability scores are highly significant and unlikely to have occurred by random chance alone.

The individual ability scores for human and LLM explanations for each image set pair are detailed in Table 1.

The source code, input data, and raw result files related to the evaluation tasks (i.e., survey questionnaires, and collected responses) are available online here.

Based on the findings from our human assessment study of concepts generated by Large Language Models (e.g., GPT-4), it is evident that LLMs hold significant promise in automating concept discovery for complex AI systems. Our study demonstrated that LLMs can produce high-level explanations that are comprehensive and understandable to humans, showcasing their potential to enhance explainability in AI.

The advantage of LLMs over traditional logical concept induction systems lies in their ability to overcome limitations associated with limited background knowledge, algorithmic constraints of heuristic search, and the integration of common sense into explanations. However, we acknowledge the limitations in the current approach of prompting LLMs like GPT-4, particularly in generating irrelevant explanations due to their uncontrolled nature(e.g., hallucination) and reliance on annotated object information in the dataset. These shortcomings can be addressed through improved prompts with varied hyper-parameters and the integration of vision-based models for automatic object identification. Additionally, we thank a reviewer for suggesting the use of a larger sample of images to strengthen the analysis. Further analysis of the similarity between ECII- and LLM-induced concepts, for example, using a similarity model, would also be interesting to see. We want to explore further research on fine-tuning open-source LLMs with a symbolic approach using description logic and few-shot training can create a more controlled environment for meaningful concept generation, aligning closely with a neurosymbolic approach to Explainable AI.

While it is crucial to ensure that LLM-generated concepts are meaningful to human users, further development is needed to assess their mapping with network activations and performance under controlled conditions. We aim to advance the development of an automated system that harnesses LLMs to offer descriptive explanations that are both human-understandable and verifiable through deep neural network activations.

This study serves as a testament to the efficient utilization of LLMs in the domain of Concept Induction and lays the groundwork for future research in leveraging these models to enhance the explainability of AI systems.

The IRB approval for the study is attached here.