A shared neural encoding model for the prediction of subject-specific fMRI response

Naturalistic imaging paradigms, such as movies and stories, emulate the diversity and complexity of real-life sensory experiences, thereby opening a novel window into the brain. The last decade has seen an increased foothold of naturalistic paradigms in cognitive neuroimaging, fueled by the remarkable discovery of inter-subject synchrony during naturalistic viewing [1]. Naturalistic stimuli also demonstrate increased test-retest reliability and more active subject engagement in comparison to alternate paradigms such as resting-state fMRI [2]. Furthermore, experiments have shown that naturalistic stimuli can induce stronger neural response than task-based stimuli [3], suggesting that the brain is intrinsically more attuned to the former. Taken together, these benefits suggest an exciting future for naturalistic stimulation protocols in fMRI.

With large-scale compilation of multi-subject neural data through open-source initiatives such as the Human Connectome Project (HCP) [4], the development of approaches that can handle this enormous data is becoming imperative. Two approaches, namely inter-subject correlation (ISC) analysis [1, 5] and shared response model (SRM) [6], have dominated the analysis of multi-subject fMRI data under naturalistic conditions. The former approach exploits similarity in activation patterns across subjects to isolate stimulus-induced processing. The latter technique, SRM, decomposes neural activity into a shared response component and subject-specific spatial bases, and has been used for inter-subject knowledge transfer through functional alignment. While simple and efficient, both these approaches rely on a common time-locked stimulus across subjects and cannot, by design, model responses to completely unseen stimuli. On the other hand, predictive modelling of neural activity through encoding models is based upon generalization to arbitrary stimuli and can thus offer more holistic descriptions of sensory processing in an individual [7].

Neural encoding models map stimuli to fine-grained voxel-level response patterns via complex feature transformations. Previously, neural encoding models have yielded several novel insights into the functional organization of auditory and visual cortices [8, 9, 10, 11]. Encoding models encapsulating different hypothesis about neural information processing can be pitted against each other to shed new light on how information is represented in the brain. In this manner, neural encoding models have been largely used for making group-level inferences. The potential to extract meaningful individual differences from naturalistic paradigms remains largely untapped. Understanding inter-subject variability in behavior-to-brain representations is of key interest to neuroscience and can potentially even help identify atypical response patterns [12]. Modelling individual brain function in response to naturalistic stimuli is one step in this direction; however, building accurate individual-level models of brain function often requires large amounts of data per subject for good generalization. The problem is further exacerbated by the variability in anatomy and functional topographies across individuals, making inter-subject knowledge transfer difficult. There is limited work in leveraging multi-subject data for more robust and accurate individualized neural encoding. To our knowledge, this problem has been studied only in the context of natural vision with a handful subjects using a Bayesian framework [13]. Further, the proposed method in [13] transfers knowledge from one subject’s encoding model into another through a two-stage procedure and does not allow simultaneous optimization of encoding models across multiple subjects.

In this paper, we attempt to fill this gap; to this effect, we propose a deep-learning based framework to build more powerful individual-level encoding models by leveraging multi-subject data. Recent studies have revealed that coarse-grained response topographies are highly similar across subjects, suggesting that individual idiosyncrasies manifest in more fine-grained response patterns [14, 6]. This hints to the idea that encoding models could share representational spaces across subjects to overcome the challenges imposed by a limited quantity of per-subject data. We exploit this intuition to develop a neural encoding model with a common backbone architecture for capturing shared response and subject-specific projections that account for individual response biases, as demonstrated in Figure 1. Our proposed approach has several merits: (i) It allows us to combine data from multiple subjects watching same or different movies to build a global model of the brain. At the same time, it can capture meaningful individual-level deviations from the global model which can potentially be related to individual-specific traits. (ii) It is amenable to incremental learning with diverse, varying stimuli across seen or novel subjects with less constraints on data collection from single subjects. (iii) It poses minimal memory overhead with additional subjects and can thus handle fMRI datasets with a large number of subjects.

Our proposed methodology is illustrated in Figure 1. Neural encoding models comprise two components: (a) a feature extractor, which pulls out relevant features from raw images or audio waveforms and (b) a response model, which maps these stimuli features into brain responses. In contrast to existing works that employ a linear response model [11, 9], we propose a CNN-based response model where the coarse 3D feature maps are shared across subjects and fine-grained feature maps are individual-specific. Previous studies have reported a cortical processing hierarchy where low-level features from early layers of a CNN-based feature extractor best predict responses in early sensory areas while semantically-rich deeper layers best predict higher sensory regions [8, 9]. To account for this effect, we employ a hierarchical feature extractor based on feature pyramid networks [15] that combines features from early, intermediate and later layers simultaneously. The output of the feature extractor is fed into the convolutional response model to predict the evoked fMRI activation. This enables us to train both components of the network simultaneously in an end-to-end fashion.

Formally, let $\mathcal{D}=\{\textbf{X}_{i},\textbf{Y}_{i}\}_{i=1}^{N}$ denote the training data pairs for N subjects, where X_i denotes the stimuli presented to subject $i$ and $\textbf{Y}_{i}$ denotes the corresponding fMRI measurements. We represent X_i as RGB images or grayscale spectrograms for the visual and auditory models, respectively. The feature model maps the 2D input into a vector representation s and is parameterized using a deep neural network F(X${}_{i};\mathrm{\phi})$ that is common across subjects. In our experiments, this model is a feature pyramid network built upon pre-trained recognition networks as DNNs optimized for image or sound recognition tasks have proven to provide powerful feature representations for encoding brain response. We define a differentiable function G(s$;\mathrm{\theta})$ that maps the features into a shared latent volumetric space z, whose first 3 axes represent the 3D voxel space and the last axis captures the latent dimensionality. The predicted response for each subject is then defined using subject-specific differentiable functions H_i(z$;\psi_{i})$ that project the coarse feature maps z into an individualized brain response. We represent G and H_i’s using convolutional neural networks to have a sufficiently expressive model. Thus, $\theta$ and $\{\psi_{i}\}$ represent a mix of convolutional kernels or dense weight matrices. The number of shared parameters, $|\theta|+|\phi|$ is kept much greater than the cardinality of subject-specific parameters $|\psi_{i}|$ to accurately estimate the shared latent space. All parameters $\{\phi,\theta,\psi_{i}\}$ are trained jointly to minimize the mean squared error between the predicted and true response. The proposed method allows us to propagate errors through the shared network even if the subjects are not exposed to common stimuli since we can always backpropagate errors for subjects independently within each batch. Furthermore, using individualized layers to account for subject-specific biases enables the model to weigh gradients coming from losses of each subject differently according to their signal-to-noise ratio. This makes the model less susceptible to noisy measurements when responses for the same stimuli are available from multiple subjects.

Figure 2 shows prediction accuracy of the proposed (‘Shared’) and baseline methods for each subject. The performance improvement is striking between proposed and individual subject models, suggesting that a shared backbone architecture can significantly boost generalization. Comparative boxplots further show that the proposed method predicts a much higher percentage of the synchronous cortex than individual subject models. Further, the difference between ‘Shared’ and ‘Shared (mean)’ as well as the dominant diagonal structure in correlation matrices suggest that the proposed method is indeed capturing subject idiosyncrasies rather than predicting a group-averaged response. Further, while the CNN response model performs slightly better in visual encoding, it incurs a performance drop compared to linear regression in auditory encoding. This perhaps suggests that the boost in accuracy seen for shared models is largely due to inter-subject knowledge transfer rather than the convolutional response model itself.

In Figure 3(A) & 3(B), we visualize the un-normalized correlations between the predicted and measured fMRI response for the proposed models, averaged across subjects. For the auditory model, we see significant correlations in the parabelt auditory cortex, extending into the superior temporal sulcus and some other language areas (55b) as well. For the visual model, while we see significant correlations across the entire visual cortex (V1-V8), the performance is much better in higher-order visual regions, presumably because of the semantically rich features. The lower performance in early visual regions could also result from the dynamic nature of visual stimulation in movies.

Figure 3(C) & 3(D) illustrate the ability of our proposed model to characterize individual differences even beyond the experimental paradigm it was trained on. The diagonal dominance in the dice matrix for both contrasts suggests that predicted contrasts are most similar to the same subject’s true contrast. No prominent diagonal structure was observed for individual subject models, presumably because of their poor generalization to out-of-domain stimuli from the HCP task battery. Further, predicted contrasts consistently highlight known areas for face and scene processing, namely the fusiform face area[26] and parahippocampal areas[27] respectively.

In this paper, we presented a framework for utilizing multi-subject fMRI data to improve individual-level neural encoding. We showcased our approach on both auditory and visual stimuli and demonstrated consistent improvement over competing approaches. Our experiments further suggest that a single experiment (free-viewing of movies) can characterize a multitude of brain processes at once. This has important implications for brain mapping which traditionally relies on a battery of carefully-constructed stimuli administered within block-designs. Inter-subject variability in response patterns induced by the complexity of naturalistic viewing can facilitate the development of novel imaging-based biomarkers. Neural encoding models are not constrained to modeling the response to a limited set of experimental stimuli; their good generalization performance suggests that they can capture broad theories of cognitive processing. Accurate, individualized neural encoding models can thus bring us one step closer to achieving the goal of biomarker discovery.

This work was supported by NIH grants R01LM012719 (MS), R01AG053949 (MS), R21NS10463401 (AK), R01NS10264601A1 (AK), the NSF NeuroNex grant 1707312 (MS), the NSF CAREER 1748377 grant (MS) and Anna-Maria and Stephen Kellen Foundation Junior Faculty Fellowship (AK).