A Neural Network Model of the Entorhinal Cortex and Hippocampus for Event-order Memory Processing

Place cells fire when a mouse enters a particular place in an environment. In this study, the figure-eight maze includes $N_{p}$ place cells, and their receptive fields were predetermined. $N_{p}=x_{max}\times y_{max}$, where $x_{max}$ and $y_{max}$ are the maximum values in the $x$-axis and $y$-axis directions, respectively. A receptive field of a place cell is expressed by an activity bump, which is approximated by a 2-dimensional Gaussian distribution of firing rate.

where $(x_{c},y_{c})$ is the centroid of $i$-th place cell ($i=1,2,\cdots,N_{p}$), and $\sigma^{2}$ is the variance of the distribution. The velocity $v$ and the head direction $\theta$ drives the location of the activity bump;

where $\delta t$ is the unit time.

Cue cells fire in response to a non-spatial input derived by the signal of local view, which is called an environmental cue. Once an environmental cue signal is recognized, the firing rate of a $j$-th cue cell ($j=1,2,\cdots,N_{c}$) is made, and this rate at a constant frequency.

where $N_{c}$ is the total number of an environmental cue. For simplification, one cue cell is associated with one environmental cue information.

Cue-place cells integrate the neuronal activity of place and cue cells. The outputs of the place and cue cell corresponding to the $i$-th position and $j$-th cue stimulus, respectively, are combined in the cue-place cells. The firing rate of the cue-place cells $\nu_{k}^{cp}(t)\in\mathbb{R}^{N_{e}}$ is clipped at the certain value as follows:

where $N_{e}=N_{p}\times N_{c}$ is the total number of events, and $k=1,2,\cdots,N_{e}$. This means that the cue-place cell fires where the cue signal occurs.

The heteroassociative network is a matrix $W(t)$ in which components $w_{pre,post}(t)$ are modulated between related cue-place cells. Firings of cue-place cells are seconds apart, even in related events. The STDP rule is only tied together limited to a postsynaptic firing that occurs within a few tens of milliseconds after a presynaptic firing [21, 22]. If a presynaptic firing is held several seconds, two temporarily distant firings can be linked. Neural activity is potentially held several tens of seconds by cue-holding neurons [23] or time cells [7]. Here we assume that the neural activity of the cue-place cell is sustained by the cue-holding neuron-like mechanism. Thus, even if the postsynaptic cell fires a few seconds after the presynaptic cell fires, the synaptic connections are strengthened. However, we introduce an exponential decay function for synaptic potentiation. The synaptic weight modulation $F(\Delta t)$ between presynaptic and postsynaptic cells is as follows:

where $\Delta t$ is the time difference between presynaptic and postsynaptic firing timings, and $A^{+}$ is the learning rate.

It would be mismanagement of resources if all neuron firing times were recorded to store the order in which events occur. So, we propose a method to extract event-order information. The synaptic weight matrix $W(t)$ of the heteroassociative memory network is modulated based on correlations between presynaptic and postsynaptic neurons. The temporal proximity between a featured event (post) and events that precede it (pre) is retained as the magnitude of the synaptic weight $w_{pre,post}$ ($pre=1,2,\cdots,N_{e},post=1,2,\cdots,N_{e}$). The component of $W(t)$ is described below:

where $T_{s}$ is the period for seeking event pairs. $\nu^{cp}_{post}$ is the neural activity of the post event, and $\nu^{cp}_{pre}$ is the neural activity of the pre event.

All the synaptic weights expressed by the matrix $W(t)$ of the heteroassociative memory network are fixed in the EM matrix via the EH matrix. If all events throughout the learning period are stored, a huge amount of computational resources are consumed to express the order of the events. To solve this problem, we proposed the EH matrix using a synaptic eligibility trace [24].

The sequence information for each event, which decays over time, represented by a heteroassociative memory network, is summed up in the EH matrix. The matrix $C(t)=(c_{pre,post}(t))\in\mathbb{R}^{N_{e}\times N_{e}}$ of the eligibility variable is stimulated by $W$.

The eligibility variable $c_{pre,post}(t)$ exponentially decays with the time constant $\tau_{c}$. The initial value $C(0)$ is an $N_{e}\times N_{e}$ zero matrix.

The order information of past events is transferred from the EH matrix to the EM matrix $M=(m_{pre,post})\in\mathbb{R}^{N_{e}\times N_{e}}$. We introduce a mechanism to store events synchronized with the reward timing in the memory. The component $m_{pre,post}$ represents the strength of the relationship between the events $E_{pre}$ and $E_{post}$.

When a reward is given at $t_{reward}$, the eligibility variable $C$ allows a change of $M(t)$. The initial value $M(0)$ is an $N_{e}\times N_{e}$ zero matrix. This change is gated by the reward-associated value $d$ of dopamine.

where $d(t)$ exponentially reduces with time constant $\tau_{d}$, and $\delta(t)$ is the delta function. If $t=t_{reward}$, $\delta=1$; otherwise $\delta=0$.

Herein, we describe how to recall event-order information from the EM matrix $M(t)$. Let us consider recall procedures to extract events associated with impetus inputs event according to the order of stored events. A recall matrix $A^{k}$ is prepared to recall the event-sequence information stored in the EM matrix $M(t)$, where $k$ is an iteration number of recall procedures. The recall matrix $A^{k}$ is formed by operating $M$ from the right on the initial matrix $A^{0}$. By repeatedly applying the EM matrix $M(t)$ to the recall matrix, the events are recalled in the order starting from the impetus input $I_{s}$. Recall procedures are shown in the following equations:

where $s(=1,2,\cdots,N_{e}$) is a component of the impetus input to recall $I_{s}$. If the input value exceeds the threshold value $\mu_{th}$, $a^{k}_{q,r}$ equals the input value; otherwise, $a^{k}_{q,r}$ equals $0$. Repeating this recall process starting from the impetus input $I_{s}$, temporally distant events can be recalled.

In this study, three tasks were conducted. The first experiment was a path recall task. We verified whether it was possible to recall the moved routes in the order in which they were experienced using the proposed model. The second experiment was a reward recognition task. We verified whether the rewarded route could be recognized if the reward was given differently depending on the route traveled. The third task was a constructed-route retrieval task. We tested whether the mouse could integrate the traveled routes to reach the reward when it traveled the first route, took a break, and then traveled the next route.

Fig. 4 depicts the value of $A^{k}$ displayed as a heat map. The matrix $A^{k}$ is obtained by repeatedly applying the recall procedures starting from the impetus input $I_{s}$, which is $I_{1}(25,25,\text{UP})$ or $I_{2}(20,10,\text{RIGHT})$. The heat map represents the recalled spatial receptive field. The recalled procedure was iterated up to three times at each event. The number of events acquired by recalls increased with an increase in the number of recalls. Starting from the impetus input $I_{s}$, the routes taken by the $I_{s}$ associated events were recalled. The length of the recalled routes increases with an increase in the number of recalls $k$. The target event location was recalled by tracing the events in the order of temporal proximity to the trigger event.

In the reward recognition task, a reward was given only during either the counterclockwise or clockwise excursion. For recalling the reward location, the recall steps were repeated after the excursions. The following two inputs: $I_{2}$ and $I_{3}$, were prepared as trigger events, as shown by the green and yellow circles in Figs. 5a and 5b, respectively. Further, $I_{2}(20,10,\text{RIGHT})$ occurred in a counterclockwise route, whereas $I_{3}(30,10,\text{RIGHT})$ occurred in a clockwise route. Although the reward was not located near $I_{2}$, by repeating the recall procedures from the trigger event, the reward location ans route to the reward appeared. Thus, when $I_{2}$ is given, the reward and route to the reward were successfully retrieved after a couple of recall steps $k=2$ and $3$, as shown in Fig. 5a. However, when $k=1$, the reward was not recalled.

Fig. 7 depicts the heat map in the constructed-route retrieval task. This heat map was obtained by repeatedly applying the recall procedures starting from the impetus input $I_{2}(20,10,\text{RIGHT})$. The clockwise route was learned with a break after the counterclockwise excursion was conducted. For $k=2$, the central aisle has a bright color. Due to the waiting time of 100 s, the sequence information of the events related to the counterclockwise route expressed on the EH matrix was attenuated with time and then accumulated in the EM matrix. This process is considered to have caused this color difference in the heat map. This result indicates that the experience can be integrated to indirectly associate the trigger event with the reward, even if there is a time lag before obtaining the reward.

A virtual mouse could reach the point where the reward was given in the clockwise route when $k=3$, even when recalling from the impetus input $I_{2}$ in the counterclockwise route. For $k=3$, the heat map of the right route at the T-junction on the central stem is brighter than that of the left route. The difference in brightness is because the reward was given at the right corner on the clockwise route, and the event order during the clockwise route was less attenuated. It is considered that this phenomenon occurred because it was judged that the behavior that was close in time to the timing of receiving the reward in this model was more valuable.

We can expect to integrate the routes that the virtual mouse had traveled so far and associate it with the position where the reward was given, even if the virtual mouse recalls from positions on counterclockwise route that was not rewarded.