User Decision Guidance with Selective Explanation Presentation
from Explainable-AI

We implemented X-Selector in a stock trading simulator in which users get support from an XAI-based IDSS. Figure 1 shows screenshots of the simulator. The simulation was conducted on a website. Participants were virtually given three million JPY and traded stocks for 60 days with a stock price chart, AI prediction of the future stock price, and explanations for the prediction.

In the simulation, participants checked the opening price and a price chart for each day and decided whether to buy stocks with the funds they had, sell stocks they had, or hold their position. In accordance with Japan’s general stock trading system, participants could trade stocks in units of 100 shares. Participants were asked to show their decision twice a day to clarify the influence of the explanations. They were first asked to decide an initial order $d^{\prime}$, that is, the amount of trade only with chart information and without the support of the IDSS. Then, the IDSS showed a bar graph that indicated the output of a stock price prediction model and its explanations. We did not explicitly show $d_{\mathrm{AI}}$ to enhance the autonomy of users’ decision-making, which is inspired by libertarian paternalism [11], the idea of affecting behavior while respecting freedom of choice as well. However, we can easily extend X-Selector to a setting in which $d_{\mathrm{AI}}$ is given to users by including it with $\bm{c}$ (when you always show $d_{\mathrm{AI}}$) or $\bm{x}$ (when you want to selectively show $d_{\mathrm{AI}}$). Finally, they input their final order $d$. After this, the simulator immediately transited to the next day. The positions carried over from the final day were converted into cash on the basis of the average stock price over the next five days to calculate the participants’ total performance.

In the task, an IDSS provides a prediction of a stock price prediction model (StockAI) as user support. StockAI is a machine-learning model that is designed to predict the average stock price in the next five business days, and we used its prediction as the target of the explanation provided to users. StockAI predicts future stock prices on the basis of an image of a candlestick chart. Although using a candlestick chart as an input does not necessarily lead to better performance than modern approaches proposed in the latest studies [22], we chose this because of the better understandability of saliency maps generated with the model as an explanation of AI predictions. Note that the aim of this research is not building a high-performance prediction model but investigating the interaction between a human and an AI whose performance is not necessarily perfect.

For the implementation of StockAI, we used ResNet-18 [23], a deep-learning visual information processing model, using the PyTorch library ¹¹1https://pytorch.org/. The StockAI is trained in a supervised manner; it classifies the ratio of the future stock price to the opening price of the day into three classes: BULL (over +2%), NEUTRAL (from -2 to +2%), and BEAR (under -2%). The prediction results are presented as a bar graph of the probability distribution for each class, which hereafter denoted as $p$. For the training, we collected the historical stock data (from 2018/5/18 to 2023/5/16) of companies that are included in the Japanese stock index Nikkei225. We split the data by stock code, with three-quarters of the data as the training dataset and the remainder as the test. The accuracy with which the model was able to predict the correct class among the three classes was 0.474, and the accuracy for binary classification, or the matching rate of the sign of the expected value of the model’s prediction and that of actual fluctuations, was 0.63 for the test dataset.

We prepared two types of explanations: saliency maps and free-texts. We applied CAM-based methods available in the pytorch-grad-cam package ²²2https://github.com/frgfm/torch-cam to StockAI and adopted Score-CAM [24] because it most clearly visualizes saliency maps of StockAI. Because CAM-based methods can generate a saliency map for each prediction class, three maps were acquired for each prediction. Let $\bm{x}_{\mathrm{map}}$ be the set of the acquired maps.

In addition, we created a set of free-text explanations based on the GPT-4V model in the Open-AI API [25], which allows images as input. We input a chart with a prompt that asked GPT-4V to generate two explanation sentences that justify each prediction class (BULL, NEUTRAL, and BEAR). Therefore, we acquired six sentences in total for each chart. Let us denote the set of them by $\bm{x}_{\mathrm{text}}$.

As a result, three saliency maps and six free-text explanations were available for each trading day, and X-Selector considered $2^{9}=512$ combinations of the selected explanations ($\hat{\bm{x}}\subseteq\bm{x}_{\mathrm{map}}\cup\bm{x}_{\mathrm{text}}$).

We implemented UserModel with a deep learning model (Figure 2). The input of UserModel is a tuple $(\bm{c},\bm{x})$. $\bm{c}$ includes four variables: date $i$, StockAI’s prediction $p$, total rate $\delta$, and initial order $d^{\prime}$. $i$ is a categorical variable that embeds the context of the day such as the stock price. $p$ is a three-dimensional vector that corresponds to the values in the bar graph (Fig. 1). $\delta$ is the percentage increase or decrease of the user’s total asset from the initial amount. $i$ and the other variables are encoded in 2048-dimensional vectors $(h_{i},h_{p},h_{r},h_{d^{\prime}})$ with the Embedding and Linear modules implemented in PyTorch, respectively.

Let us denote $\bm{x}_{\mathrm{map}}$ and $\bm{x}_{\mathrm{text}}$ as a set $\{(x,cls,flag)\}$, where $x$ is the raw data of an explanation, and $cls\in\{\textrm{BULL},\textrm{NEUTRAL},\textrm{BEAR}\}$. $flag=1$ if $x$ is to be presented, and $0$ when hidden. $\bm{x}_{\mathrm{map}}$ and $\bm{x}_{\mathrm{text}}$ are also encoded in 2048-dimensional vectors $(h_{\mathrm{map}},h_{\mathrm{text}})$:

where $\odot$ denotes an element-wise product. $CNN$ is a three-layer CNN model. For $TextEncoder$, we used the E5 (embeddings from bidirectional encoder representations) model [26] with pretrained parameters³³3https://huggingface.co/intfloat/multilingual-e5-large.

All embedding vectors ($h_{i},h_{p},h_{r},h_{d^{\prime}},h_{\textrm{map}},h_{\textrm{text}}$) are concatenated and input to a three-layer linear model. To extract the influence of explanations, the model was trained to predict not $d$ but the difference $d-d^{\prime}$. In our initial trial, UserModel always predicted $d-d^{\prime}$ to be nearly 0 due to the distributional bias, so we added an auxiliary task of predicting whether $d=d^{\prime}$ and trained the model for predicting $d-d^{\prime}$ only when $d\neq d^{\prime}$. The expected value of $d_{u}$ in Equation 3 is $P(d\neq d^{\prime})\cdot(d-d^{\prime})+d^{\prime}$.

$\pi$ was acquired with the deep deterministic policy gradient, a deep reinforcement learning method [27]. We simply trained $\pi$ to decide $d$ to maximize assets on the basis of $p$ for the training dataset. The reward for the policy is calculated as the difference in total assets between the current day and the previous day.

We conducted a preliminary experiment to investigate the performance of users who were provided explanations with two naive strategies (ALL and ARGMAX). ALL shows all the nine explanations available for each day, and ARGMAX selects explanations for StockAI’s most probable prediction. To ensure the quality of the explanations, we also prepared two baselines: ONLY_PRED shows $p$ but does not provide any explanations. In PLAIN, participants received no support from the IDSS and acted on their own.

For simulation, we chose a Japanese general trading company (code: 2768) from the test dataset on the basis of the common stock price range (1,000 - 3,000 JPY) and its high volatility compared with the other Nikkei225 companies.

Because we had anticipated that the accuracy of StockAI would affect the result, we prepared two scenarios: high-accuracy and low-accuracy. We calculated the moving average of the accuracy of StockAI with a window size of 60 and chose two sections for them. The accuracy of StockAI for high-accuracy was 0.750, which was the highest, and that for low-accuracy was 0.333, the chance level of three-class classification.

We recruited participants to join the simulation with compensation of 220 JPY through Lancers⁴⁴4https://lancers.jp/, a Japanese crowdsourcing platform, and got 336 participants. The participants were first provided pertinent information, and 325 consented to the participation. We gave them instructions on the task and gave basic explanations about stock charts and the price prediction AI. We instructed the participants to increase the given three million JPY as much as possible by trading with the IDSS’s support. To motivate them, we told them that additional rewards would be given to the top performers. We did not notify them of the amount of the additional rewards and the number of participants who got them. We asked six questions to check their comprehension of the task. 34 participants who failed to answer correctly were excluded from the task. After familiarization with the trading simulator, the participants traded for 60 virtual days successively. 242 participants completed the task (152 males, 88 females, and 2 did not answer; aged 14-77, $M=42.8,SD=10.1$). Table I gives details on the sample sizes.

Figure 3 shows the changes in the participants’ performance. A conspicuous result is the underperformance of PLAIN, particularly in the high-accuracy scenario. ONLY_PRED performed well for high-accuracy, but could not outperform PLAIN for low-accuracy. This suggests that presenting $p$ alone contributes to improving performance only when it has enough accuracy.

ALL and ARGMAX showed different results between the scenarios. For high-accuracy, ARGMAX outperformed ALL. ALL slightly underperformed the ONLY_PRED baseline as well. This suggests that ARGMAX explanations successfully guided users to follow the prediction of StockAI while ALL toned down the guidance, which worked negatively in this scenario. On the other hand, ALL outperformed ARGMAX and the baselines for low-accuracy. Interestingly, ARGMAX also outperformed the baselines, which suggests that explanations successfully provide users with insights into situations and AI accuracy and can contribute to better decision-making. ALL positively worked for low-accuracy by providing multiple perspectives.

To evaluate X-Selector, we conducted a simulation with its selected explanations. To train UserModel, we used the data of the preliminary experiment and additional data acquired in another experiment in which explanations were randomly selected. We added the data to broaden the variety of explanation combinations in the dataset. The numbers of the additional participants were 54 and 45 for high- and low-accuracy, respectively. We conducted a 4-fold cross validation for UserModel, and the correlation coefficient between the model’s predictions and the ground truths was 0.429 on average ($SD=0.056$).

We obtained a participation of 97 participants. Finally, 39 and 35 participants completed the task for high-accuracy and low-accuracy, respectively (46 males, 26 females and 2 did not answer; aged 23-64, $M=39.6,SD=10.0$).

To analyze the results, we compared the correlation coefficient between $d_{\mathrm{AI}}$ and $d_{u}$ for each participant as a measure of whether X-Selector could successfully guided users to $d_{\mathrm{AI}}$, as well as the comparison of user performance that we did in the preliminary experiment.

Figure 4 shows the correlation coefficient distribution between $d_{\mathrm{AI}}$ and $d_{u}$ in the high-accuracy condition. The results for ALL and ARGMAX are also shown for comparison. Notably, while the peaks for ALL and ARGMAX are centered around zero, X-Selector shifted this peak to the right, indicating a stronger correlation between $d_{u}$ and $d_{\mathrm{AI}}$ for a greater number of participants. This means that X-Selector effectively guides users to $d_{\mathrm{AI}}$ without coercing but presenting explanations selectively.

Figure 4 illustrates the user performance. X-Selector generally outperformed ALL and ARGMAX, meaning that X-Selector enabled users to trade better with selective explanations. In more detail, X-Selector first underperformed ARGMAX, but the score reversed on day 16. The gap once narrowed near day 39, but it broadened again until the end.

A possible reason for X-Selector’s better performance is that it can predict which combination of explanations guides participants to sell or buy shares more. For example, the stock price around day 16 dropped steeply, so the IDSS needed to guide participants to reverse their position. Here, whereas ARGMAX showed explanations for BEAR, X-Selector showed explanations for NEUTRAL as well as BEAR, which may have helped users sell their shares more. Similarly, X-Selector also attempted to guide users to buy a moderate amount when $d_{\mathrm{AI}}$ was positive but not high by, for example, showing only a saliency map for BULL and no text explanations. Another reason is that X-Selector can overcome the ambiguity in the interpretation of $p$. $p$ reflects a momentum of stock price in the high-accuracy scenario and must provide some insight for trading, but it was up to the participants how to use this to actually decide their order. $\pi$ sometimes suggested that they buy shares even though NEUTRAL or BEAR was the most likely in $p$. Thus, we can say that $p$ was poorly calibrated, but by referring to $\pi$, X-Selector can avoid misleading participants and instead lead them to more promising decisions.

On the other hand, X-Selector underperformed ARGMAX until day 16. The stock price was in an uptrend until day 14, and ARGMAX continuously presented explanations for BULL for 12 days in a row, which may have strongly guided participants to buy stocks and lead to large benefits. In our implementation, UserModel considers only the explanations of the day and does not consider the history of what explanations were previously provided, which can be a next target for future work.

X-Selector could not improve user performance in the low-accuracy scenario (Figure 5). Overall, the result was similar to ARGMAX and underperformed ALL. We further focused on the correlation coefficient between $d_{u}$ and $d_{\mathrm{AI}}$. Figure 5 shows that, contrary to the high-accuracy scenario, X-Selector did not increase the score. The different results between the high- and low-accuracy scenarios indicate the possibility that participants actively assessed the reliability of the AI and autonomously decided whether to follow X-Selector’s guidance. This itself highlights a positive aspect of introducing libertarian paternalism to human-AI interaction in that users can potentially avoid AI failure depending on its reliability. However, this did not result in improving their performance in this scenario. The lack of correlation between the score and the final asset amounts in the X-Selector condition ($r=0.048$) suggests that merely disregarding the AI’s guidance does not guarantee performance improvement. A future direction for this problem can be developing a mechanism to control the strength of AI guidance and provide explanations in more neutral way depending on AI accuracy.