Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction

To fix ideas, a bidimensional toy example with four input clusters is given in figure 2. We have illustrated a solution with two univariate experts, i.e. $\mathcal{S}=\{S_{1},S_{2}\}=\{\{X_{1}\},\{X_{2}\}\}$: each predicts two separable clusters and their respective selection function have low values where the remaining clusters are mixed. A bivariate expert ($S_{3}=\{X_{1},X_{2}\}$) can also solve the task (fig.  4), but in order to have the sparsest $M$, we would favour the first solution and have a zero-selection on the bivariate expert. In figure 4, we add four outer clusters to example (a) that are not separable when projected on $X_{1}$ nor $X_{2}$. In this case, the new clusters have to be attributed to the bivariate expert. A single bivariate expert could solve the whole task alone, but in order to get the sparsest mask $M$, denoting the minimal required input features to make a correct prediction in a specific part of the input space, using univariate experts is sufficient for the central clusters.

In the general case, $\mathcal{S}$ is composed of many potentially overlapping subsets. We can represent subsets and experts with a directed acyclic graph (DAG) defined as the Hasse diagram of the subsets partially ordered by inclusions (Bach, 2009): simple experts with few variables are children of parent experts of growing input support. The same way we select univariate experts over the bivariate expert in the toy example, the sparsity constraint means that the selection of a child subset should induce the deselection of parent subset it is included in. We ensure this property by design in a boosting-like manner: for each sample, we try to select an expert restricted to the smallest set of features, then, if it is not selected, we move to a parent expert.

To allow training with a gradient-descent method, we consider a stochastic relaxation of the selection of the expert. We introduce the parametric functions $g^{i}_{\theta}:S_{i}\rightarrow[0,1]$. We then define the selection functions recursively:

1. (1)

   Atomic subsets: For all $S_{i}\in\mathcal{S}$ such that $\forall S_{j}\in\mathcal{S}\setminus S_{i},S_{j}\not\subset S_{i}$, then $\alpha^{i}_{\theta}(x)=g^{i}_{\theta}(x\circ m_{i})$ ;

2. (2)

   Mixed subsets: For the remaining subsets, we introduce the notation $\omega(i)=\{j|S_{j}\subsetneq S_{i}\}$ for the set of strictly included subsets and $\alpha^{i}_{\theta}(x)=g^{i}_{\theta}(x\circ m_{i})\prod_{j\in\omega(i)}(1-g^{j}_{\theta}(x\circ m_{j}))$

By induction, the functions $\alpha^{i}$ we design satisfy the following properties:

1. (1)

   Probability: $\alpha^{i}\in[0,1]$

2. (2)

   Input restriction dependence: $\alpha^{i}$ is entirely conditioned on $S_{i}$, and is thus blind to eventual parent subsets

3. (3)

   Deselection induced by children: $\alpha^{i}\leq\prod_{j\in\omega(i)}(1-g^{j}_{\theta})$

There are many possible choices for $f^{i}_{\theta}$ and $g^{i}_{\theta}$. In the rest of the paper, we study the use of deep neural networks, that have good generalisation capabilities, and their specific adaptation to our interpretation framework. Let us denote $F_{\theta}^{i}$ a deep neural network function restricted on $S_{i}$.

For the binary problems, we propose to use a tanh function on the output layer. Then, we use the joint definition:

The neural network output simultaneously makes a prediction for $Y$ and its absolute value indicates a confidence value for selection as an expert. We experimentally found that for inference, using $g^{i}_{\theta}(x)=F_{\theta}^{i}(x)^{2}$ or $g^{i}_{\theta}(x)=\frac{2|F_{\theta}^{i}(x)|^{p}}{1+|F_{\theta}^{i}(x)|^{p-1}}$ also worked well to dampen noisy values around 0 then smoothly increasing selection importance for stronger predictions.

We could sequentially maximise the likelihood for each expert with subsets of increasing cardinality, or even group independent subsets for fewer training phases, as in (Lou et al., 2013) where all univariate functions are trained in parallel before bivariate functions. This approach can be time-consuming, especially with neural networks as experts.

Instead, we train all experts in parallel using EM. However, with neural networks, we do not have a tractable solution for $\arg\max_{\theta}\mathcal{Q}(\theta,\theta_{\textrm{old}})$. This issue is addressed by GEM by substituting the maximisation with an incremental update of $\mathcal{Q}$. The training of our models follows two alternating steps:

1. (1)

   E-step Evaluate $p_{\theta^{\textrm{old}}}(m_{i}|x)=\alpha^{i}_{\theta^{\textrm{old}}}(x\circ m_{i})=|F^{i}_{\theta^{\textrm{old}}}(x\circ m_{i})|\prod_{j\in\omega(i)}(1-|F^{j}_{\theta^{\textrm{old}}}(x\circ m_{j})|)$, which weights the sample $x$ differently for each expert with a deselection for parents ;

2. (2)

   Generalised M-step Perform a gradient-step update: $\theta_{\textrm{new}}=\theta_{\textrm{old}}+\eta\frac{\partial\mathcal{Q}}{\partial\theta}(\theta,\theta_{\textrm{old}})$, with $\eta$ the learning rate.

Following the derivation of $\mathcal{Q}$ in equation (4), and using equation (5), we have:

The M-step can be easily implemented in modern deep learning libraries to propagate the updates through the layers of the experts $F_{\theta}^{m}$. In the next section, we detail their architecture.

We assume that functions $(F^{i}_{\theta})$ have a single linear layer with an activation function $\sigma$:

We suppose we have added a scalar $1$ to the input $x$ to account for bias when multiplying by matrix $(W^{i}_{0})$ to simplify notations. $\tilde{W}^{i}_{0}$ corresponds to matrix $W^{i}_{0}$ with null columns at the indices where $m_{i}$ is null, i.e. to $W^{i}_{0}\circ m_{i}$ with the Hadamard product applied row by row. Then, we can stack the matrices and define $F_{\theta}(x)$:

We identify $F_{\theta}$ as an overarching single-layer neural network with activation $\sigma$ and matrix parameter $\tilde{W}_{0}$. The latter matrix is typically sparse because of the masks $(m_{i})$ successively applied on each row and can be efficiently implemented using a weight constraint in standard deep learning libraries. We have shown the base case: using masks, we can create one-layer networks for which the output dependencies to interpretation subsets can be traced. We must now prove by induction that it can be extended to several stacked layers.

When functions $(F^{i}_{\theta})$ are multi-layered neural networks with $K$ layers, activation functions $\sigma_{k}$, matrices parameters $W_{k}^{i}$ and hidden layer output $h^{i}_{k}$ on layer $k$, where $h^{i}_{K}\triangleq F^{i}_{\theta}$, we have by definition:

Our goal is to define overarching hidden layers $h_{k}$ that are conditioned on the corresponding restriction $S_{i}$ for $k>0$:

Let us assume we have already built interpretable hidden layers up to layer $k$. As in the case of the first layer, we would like to define a masked matrix $W_{k}$ such that $h_{k+1}(x)=\sigma_{k}(W_{k}h_{k}(x))$, while preserving the correct input dependencies.

A first approach is to define $W_{k}$ as a diagonal by blocks with submatrices $W_{k}^{i}$:

We would again use a masked matrix with sparse parameters and it would be equivalent to having $H$ neural networks trained in parallel but yet remaining independent from one another because the hidden features of each expert is only used in the computation of its corresponding upper hidden layer features.

However, we do not change the desired property of dependencies by also allowing to compute the hidden layer $h_{k+1}^{i}$ using $h_{k}^{j}$ for all $j\in\omega(i)$. We thus rather define $W_{k}$ with non-null blocks $W_{k}^{i,j}$ everywhere $S_{j}\subset S_{i}$:

An example of such added links is given in the overview figure 1.

The last step to be able to train this network using a gradient-descent-based algorithm is to prevent back-propagation before matrix blocks $W^{i,j}_{k}$ for $i\neq j$. Otherwise, the dependency is not preserved since child classifiers indirectly depend from their parents during training. This can be easily implemented in Tensorflow using a copy function as stop_gradient. We have then recursively defined an interpretable multi-layered neural network.

In previous sections, we have formulated a simple way to create interpretable linear layers using masked matrices. Then, applying additional activations or element-wise operations (e.g. skip connection, normalisation, …) does not change the dependency of each sub-network $F^{i}_{\theta}$ on its restriction $S_{i}$. We can also apply functions along a time dimension to extend our method to sequences of inputs: as long as we process together hidden features computed using the same expert ($i$) or its child experts ($j\in\omega(i)$), we do not change the restricted input dependency.

An interesting case is the Transformer model (Vaswani et al., 2017) or its variants (Devlin et al., 2019), that have been recently popularised across many fields, often achieving state-of-the-art results. Those models leverage multi-head attention modules taking as input a query ($q$), key ($k$) and value ($v$):

Defining $P$ as a multiple of $H$, we partition the heads into several groups that act as experts. With the same procedure as before, we constraint parameters $W^{q},W^{k},W^{v},W^{o}$ to be masked matrices to obtain an interpretable multi-head attention module. The remaining building blocks of the Transformer are element-wise functions or can be made interpretable, allowing to formulate an interpretable Transformer model.

Other deep architectures can be made interpretable using the same principle. We have derived an interpretable gated recurrent units network during our experiments. A bit of thinking can be required to correctly link each sub-function correctly in complex cases, for instance when using multiple heterogeneous inputs. Several implementation examples can be found on this paper code repository for more details ²²2 Our code repository: https://github.com/deezer/interpretable_nn_attribution.

We simulate a mixture of eight Gaussian distributions according to toy task (b) we have introduced in section 3.3.1. With $X=(X_{1},X_{2})$ and $Y\in\{-1,1\}$, we define three interpretation subsets $S_{1},S_{2},S_{3}=\{X_{1}\},\{X_{2}\},\{X_{1},X_{2}\}$, i.e. two univariate experts, $f^{1}$ and $f^{2}$, and one residual bivariate expert, $f^{3}$. We instantiate an interpretable two-layers feed-forward network with $3\times 16$ neurons on each hidden layer and ReLU activations and train it for a few minutes until convergence using Adam (Kingma and Ba, 2014) with default parameters².

Except for a few misclassified edges points, this task is simple enough to be almost perfectly solved by the network, as shown in figure 5 (RQ1). We see that the expected attribution (fig. 4) is obtained, the central clusters are attributed to each univariate expert instead of using the expert with all input features (RQ2).

We evaluate our method on a more realistic collaborative filtering task (CF). We reproduce the setup of NCF (He et al., 2017) and compare their method with an equivalent interpretable network. Specifically, we use the MovieLens 1M dataset ³³3https://grouplens.org/datasets/movielens/1m/, containing one million movie ratings from around six thousands users and four thousands items. All ratings are binarised as implicit feedbacks to mark a positive user-item interaction, while non-interacted items are considered negative feedbacks. The performance is evaluated with the leave-one-out procedure, and judged by Hit rate (HR) and Normalised Discounted Cumulative Gain (NDCG). Validation is done by isolating a random training item for each user. More details can be found in the original paper  (He et al., 2017) and (Dacrema et al., 2019).

In CF, a user $u$ (resp. item $i$) is typically embedded into a latent vector $p_{u}$ (resp. $q_{i}$), and the observed interactions are estimated via a similarity function. In NCF, the authors propose to replace the traditional inner product by a neural network to compute similarities. Because of the projection on a latent space, CF is more difficult to interpret than a content-based method that would only leverage the provided descriptive features for users - age range, gender, occupation ($c_{u}$) - and movies - year, genres ($c_{i}$). A model merely treating user and item using generic ranges (i.e. clusters) instead of personalised embeddings is however too coarse and underperforms.

Here, our method can be used to mix content-based and CF experts to discriminate interactions that can be predicted based on content, from the one that need an additional CF treatment to model users and items particularities. This way, we can trace if an item is recommended because of its similarity among a generic item range (eg. similar to horror-movies from the 90’s), a user range (eg. also liked by male viewers in their twenties), the combination of both, or beyond using CF. To this end, we define four experts with $\mathcal{S}=S_{1},\dots S_{4}=\{c_{u},c_{i}\},\{c_{u},p_{u},c_{i}\},\{c_{u},c_{i},q_{i}\},\{c_{u},p_{u},c_{i},q_{i}\}$.

We use the multi-layered network version of NCF named MLP in (He et al., 2017), and parametrise $p_{u},q_{i}\in\mathbb{R}^{64}$, $c_{u},c_{i}\in\mathbb{R}^{16}$, and four hidden layer of sizes $[512,256,128,64]$. The interpretable counterpart we dub Intrp-MLP, is build with the same architecture but with masked weights², which is equivalent to having four experts with hidden layer sizes $[128,64,32,16]$.

As presented in table 1, our interpretable version of NCF-MLP achieves close performances to the control non-interpretable model, with a 4% difference in HR (RQ1). This latter control model has a better HR and NDCG than reported in (He et al., 2017), which can be explained by the addition of contextual features $c_{u},c_{i}$ and bigger hidden layers.

Contrary to section 4.1, we do not have access to ground-truth attributions to check our model interpretability. As a simple proxy, we can check the attribution distribution on the test set for RQ2 (fig. 6). A first sanity check is that attributions do not collapse to an unique expert and have a relative diversity. We also see that the pure content-based expert ($S_{1}$) is hardly selected, which is coherent with the underperformance of content-based models on this task.

Overall, we must underline that 66% of the items are predicted using the three first experts, for which an interpretation can be provided as either or both item and user will be described by generic features instead of a CF embedding: e.g. the selection of $S_{2}$ (resp. $S_{3}$) indicates a similarity to the item cluster (resp. user), as movies from a specific year and genre. The residual 34% are left for the CF expert ($S_{4}$) when further personalisation is needed.

We study the task of skip prediction with two music sessions datasets. First, the Music Streaming Sessions Dataset (Brost et al., 2019). This public dataset contains anonymised listening logs of users from the Spotify streaming service over an 8 week period. Listening logs are sequenced together for each user, forming roughly $150$ million listening sessions of length ranging from size 10 to 20. Sessions including unpopular tracks were excluded, limiting the overall track set to approximately 3.7 millions tracks.

For the sequential skip prediction task¹, a session $(i)$ of length $2l^{(i)}$ is cut in half, with the first half (referred to as $A^{(i)}=A_{1}^{(i)}\dots A_{l^{(i)}}^{(i)}$ $\in\mathbb{R}^{l^{(i)}\times f}$) containing sessions logs, user interactions logs and track metadata, while the second half ($B^{(i)}\in\mathbb{R}^{l^{(i)}\times g}$) contains only the track metadata. The goal is to predict the boolean value labelled as skip_2 ($Y^{(i)}\in\{-1,1\}^{l^{(i)}}$) among the missing interaction features of $B^{(i)}$. We omit the indices $(i)$ to simplify notations.

In addition to this dataset, we also use a private streaming sessions dataset provided by the music streaming service Deezer. This dataset contains $10$ million listening sessions of range 20 to 50 from a week of streaming logs. To have a similar setup to Spotify, we extract random session slices of size 20 at each epoch, enabling to have virtually more listening sessions. Without the anonymity constraint, this private dataset enables to use more features and to better evaluate interpretations with tangible data that can be streamed and manually checked. This dataset notably includes user metadata, including favourite and banned tracks, recently listened tracks, mean skip rate, user embedding, …

The difficulty of the challenge lies in the multitude of origins a skip can have. For instance, we can check that users counterintuitively skip their favourite tracks with the almost same ratio as other tracks. Skips do not just signal a music a user does not like, they also happen when a track is liked by a user but streamed at the wrong time, or when a user quickly browses the catalog, looking for a specific song, or just fresh content, or with connection errors or misclicks. Conversely, non-skip are also implicit, sometimes the user is simply not there to change an unwanted track.

We compute the accuracy (Acc) of correctly predicted skip interaction in each half-session $B$. We also make use of the evaluation metrics introduced during the challenge, the Mean Average Accuracy (MAA), to allow for comparison. Average accuracy is defined by $AA=\sum_{j=1}^{T}\textrm{acc}(j)L(j)/T$, where $T$ is the number of track to be predicted in $B$, $\textrm{acc}(j)$ the accuracy at position $j$, and $L(j)$ a boolean indicating if the prediction at position $j$ was correct. This metric puts more weights on the first track of $B$ than the last ones.

It is argued that in the context of a session-based recommender system, this unbalance is due to the fact that it is more important to know if the next immediate track to be streamed will be skipped or not given preceding interactions and prevent a bad recommendation in the nearest future. However, as we will see, this latter argument can be flawed as first track prediction strongly depends on the blind continuation of interactions more than interesting underlying mechanisms that combine multiple features, hence not always providing much information to improve recommendation. We use the accuracy on the first immediate track (Acc@1) to highlight this effect. This question of relative relevance of a skip to a context needs to be addressed to allow retroactive improvements of a recommender system, which could be provided by the task we are trying to solve, skip interpretation.

As we suggested, skips strongly depend on the persistence of skip behaviours. This can be interpreted as users being active by blocks: once a skip is performed, there is a higher chance that a user will also skip the next track while still on its app, and the other round, while not on the app, users may be more likely to tolerate an unwanted song. Because of this effect, it is relevant to use a persistence model as baseline, returning the last known interaction in $A$ for all the elements of $B$. We additionally use a mean skip measure from $A$. We thus have two baseline predictors:

We use a standard Transformer architecture (Vaswani et al., 2017), with 3 stacked identical self-attentive layer blocks for the encoder architecture with key-values input $A$, as in the original paper, and 3 cross-attentive layer blocks for the decoder with query input $B$.

For interpretation, we define $8$ experts for the Spotify dataset (fig. 8), and $10$ for Deezer (fig. 10). It must be underlined that $A$ and $B$ do not have the same interpretation subsets on the first layer as some features are missing for $B$, the session to be predicted without logged interactions. The method however remains the same in this heterogeneous case to preserve dependencies: a link can be added between features based on $S_{x}$ and $S_{y}$ if and only if $S_{y}\subset S_{x}$.

We train all models using Adam with default parameters and learning rate set to $10^{-4}$. The learning rate is automatically reduced on plateau up to $10^{-6}$. The two datasets are split in a 80-10-10% fashion for training/validation/test. Because of the huge number of sessions, the models reach convergence on the training loss in around roughly two epochs. We did not observe any overfitting effect, making it easy to control the optimisation.

Results for both datasets are given in table 2 and 3. Baselines models, though parameter-free, are performing strikingly well on both datasets, especially on the first track prediction where the continuation effect is the strongest. In both cases, our interpretable models have close performances to their non-interpretable architecture counterparts, though losing around a point of accuracy.

In the WSDM Challenge¹, the evaluation was performed on a private and still inaccessible test set. However, the results of our baselines and control Transformer model seem to be coherent with the reported results, our control model would have been ranked to the fifth place of the challenge leaderboard.

As in 4.2, we inspect the attribution distribution on the test sessions of Spotify (fig. 8) and Deezer (fig. 10). In both cases, there is a strong unbalance toward the simplest expert containing the interaction data of $A$. This results allows to confirms our initial intuition that most skips result from pure interaction patterns and do not depend on other data. Those skips are not interesting for a recommender system as they do not tell much about user preferences. For the Spotify dataset, subsets $S_{3}$, $S_{4}$ and $S_{6}$ reveal that 25% of skips can be predicted from the overall track metadata coherence, while being agnostic to the given skips in $A$, which hints at simple ways to filter tracks in a candidate recommended session continuation $B$. For the Deezer dataset, the second most attributed expert ($S_{4}$) leverages a discounted skip rate measure that indicates a recent user-track affinity. We can conclude from the attribution levels that this relative measure is a stronger indicator than a favourite track signal ($S_{2}$) or overall popularity ($S_{3}$) to predict skips.

To illustrate the kind of interpretation we can get, an example of predicted session from Deezer is given in figure 11. Quite typically, the $S_{1}$ expert that only contains the given skip in $A$ will have the strongest attribution for the first track of $B$, corresponding to the continuation of the last two non-skips in $A$, but will vanish rapidly for the next tracks in favour of more complex experts. In the middle of $B$, there is a Malaysian pop music, this rupture from the other rock songs can be observed in the sudden attribution to $S_{9}$, an expert based on musical data. Beyond simple cases of interaction continuation or favourite tracks, our method can handle this kind of multifactorial session and provide an insight on their nature.