MUFM: A Mamba-Enhanced Feedback Model for Micro Video Popularity Prediction

The widespread adoption of portable devices has significantly contributed to the success of micro video platforms like TikTok. These devices make it easy for users to share their experiences, opinions, and thoughts in various formats, such as text, images, audio, and video. The resulting increase in user participation has led to the emergence of an important research area: micro video popularity prediction (MVPP).

We recognize that current approaches to predicting short video popularity often fall short by underutilizing the wealth of available data, particularly in overlooking the role of event-driven propagation within social networks. To address these shortcomings, we introduce the Mamba-Enhanced User Feedback Capture Model for Micro Video Popularity Prediction (MUFM), which advances popularity prediction by integrating refined recommendation systems with models that account for social network dynamics.

MUFM is designed to make better use of the diverse data linked to short videos. It starts by using a retrieval system to find relevant micro videos from a multi-modal database, filtering content based on all available information including visual, audio, and text features. To better understand user responses, we use the Mamba Hawkes process, training on a dataset of 19,000 scraped comments to show how recommendation systems and user behavior influence video popularity. Specifically, the model processes video, audio, and related text data using the Mamba Hawkes process to recreate the video’s spread and produce a recommendation index. Additionally, we use a cross-attention mechanism to detect connections between the target video and similar content, further boosting the model’s prediction performance.

Contributions: Overall, the main contributions of this work are summarized as follows:

• •

  We present a Mamba-based framework that captures long-term dependencies and dynamic event interactions driven by user feedback, aimed at predicting micro video popularity. Designed for robustness against noise and uncertainties, it ensures more accurate and reliable predictions.

• •

  We use a Hawkes Process Model, built on the Mamba architecture, to quantify the impact of user reactions. Additionally, we integrate this model with a user-focused SASRec model to analyze and infer the influence of TikTok’s recommendation system on the popularity of micro videos.

• •

  Our evaluation of the Microlens-100k dataset, the largest available for this task, shows that our MUFM model outperforms the current state-of-the-art by 23.2%.

Our methodological approach encompasses two modules: the video information extraction module founded on multi-modal information processing and database detection, and the social network dissemination simulation module for the video. In the multi-modal processing module of the video, we introduce LLM to retrieve related videos and process various modal information of the video via architectures such as ViT. Subsequently, we introduce a cross-attention mechanism to explore the interaction relationships between the retrieved videos and the target video. In our social network dissemination simulation module, we establish an event-focused Hawkes process model based on the Mamba architecture to reproduce the dissemination of the video within the group. A prediction network integrates all outcomes of the two modules and offers a prediction of the dissemination intensity of the video.

In this work, we propose MUFM, a multi-modal model for MVPP. MUFM introduces a retrieval-augmented framework and a Hawkes process model for the MVPP task. We align the visual audio and textual modalities to find similar videos. Cross-modal bipolar interactions are implied to address the presence of inconsistent information between models, as well as a retrieval interaction enhancement method to capture meaningful knowledge from relevant instances. The mamba-based Hawkes process model provides info on user behavior and improves our model precision. Experiments on the real-world micro-video dataset show our method is effective and outperforms the current state-of-the-art in MVPP.

The micro video memory repository is a collection of reference pairs, denoted as $\left\langle frames,text\right\langle$, with each element encoded for efficient retrieval. To generate a retrieval vector for a given micro video $V_{i}$, we begin by using BLIP-2, a pre-trained large language model, to analyze the video’s content and produce descriptive captions for its frames, resulting in the set $C_{i}=\{c^{i}_{1},\dots,c^{i}_{L}\}$, where $L$ represents the total number of frames in video $V_{i}$.

For the audio component, we apply a similar process using the CLAP (Cross-Modal Language-Audio Pre-training) model, which generates descriptive textual annotations for the audio, denoted as $a_{i}$. These annotations are derived from a predefined set of audio descriptions. The resulting synthetic captions $C_{i}$ are then combined with the audio descriptions $A_{i}$ and the original textual descriptions $T_{i}$ to form a comprehensive text prompt. This is achieved using the concatenation operator $\oplus$, leading to

This synthesized text prompt $P_{i}$ is then processed through a pre-trained semantic extraction model, specifically the UAE-Large architecture, to generate a retrieval vector $R_{i}$ that encapsulates the key attributes of video $V_{i}$. This process is repeated for each micro video in the memory repository, assigning each one a unique retrieval vector that serves as its identifier within the system.

This method unifies the visual, audio, and textual modalities into a cohesive representation, enhancing the model’s ability to predict micro video popularity. Additionally, it improves the accuracy and efficiency of content retrieval and analysis in the domain of micro form video dissemination.

The Bank Retriever mechanism is designed to identify the Top-$S$ most similar videos from a memory bank by evaluating the similarity scores between the target video and stored instances. The process begins by receiving a query video $V_{q}=\{v_{1}^{q},v_{2}^{q},\dots,v_{S}^{q}\}$ and searching through the memory bank, which holds a collection of frame-text pairs. The retriever employs a maximum inner product search algorithm to scan the memory bank, selecting the most relevant videos based on their similarity to the query.

To capture the key features of a micro video $V_{i}$, we extract visual, audio, and textual information. We start by selecting a series of key frames, denoted as $F_{1}^{i},F_{2}^{i},\dots,F_{K}^{i}$, where $K$ represents the total number of frames.

For visual feature extraction, we apply a pre-trained visual model. Each frame $F_{j}^{i}\in\mathbb{R}^{H\times W\times C}$ is divided into fixed-size patches and reshaped into a sequence of flattened 2D patches ${F_{j}^{i}}^{\ast}\in\mathbb{R}^{N\times P^{2}\times C}$, where $(H,W)$ is the resolution of the original frame, $C$ is the number of channels, $(P,P)$ is the patch size, and $N$ is the number of patches, which serves as the input sequence length for the Vision Transformer (ViT).

For audio features, we follow a similar approach using the Audio Spectrogram Transformer (AST). We first align the timestamps of the sampled frames with corresponding audio segments $W=\{w_{1},w_{2},\dots,w_{K}\}$ extracted from the video’s soundtrack. These audio segments are processed using a Fast Fourier Transform (FFT) to generate Mel-frequency spectral coefficients (MFCC), resulting in a spectrum matrix $M_{S}^{n_{mel}\times n_{a}}$, where $n_{mel}$ is the number of mel-filters, and $n_{a}$ is the number of time frames, determined by the audio length and stride. This spectrum matrix, treated as a single-channel image, is then processed using the same patching method applied to visual frames.

Next, we linearly embed each visual and audio patch, adding sequential position embeddings to capture positional relationships within the frames. The resulting $K$ patch sequences are then processed by ViT and AST, producing visual and audio representations $E_{i}^{v}$ and $E_{i}^{a}$, respectively. These feature matrices are concatenated to emphasize the relationship between visual and audio information:

where $\oplus$ denotes the concatenation operation, combining visual and audio features. Here, $d_{v}$ and $d_{a}$ represent the dimensions of the visual and audio features, respectively. The concatenated features $E_{i}^{v,a}$ are then passed through a linear layer $W_{c}\in\mathbb{R}^{(d_{v}+d_{a})\times d}$ with a ReLU activation function, generating an audio-visual input token $X^{c}$:

For textual information, we process micro video textual description into pre-trained language model AngLE to produce a sequence of textual embedding $E^{t}_{i}\in\mathbb{R}^{n_{w}\times d_{t}}$, where $n_{w}$ is the word length in the textual descriptions. The textual embedding $E^{t}_{i}$ are then passed through a linear layer $W_{t}\in\mathbb{R}^{d_{t}\times d}$, generating textual input token $X^{t}$:

Aligning audio-visual and textual modalities in micro videos presents challenges due to potential inconsistencies between textual descriptions and video content. To address this, we implement a cross-attention network comprising both positive and negative attention modules, designed to capture the similarities and differences between multi-modal information. The positive attention module focuses on identifying the most consistent features across different modalities, while the negative attention module highlights any inconsistent or contradictory information.

Within the positive attention module, the most aligned features between modalities are calculated using cross-modal attention vectors. For a given video $v_{i}$, the audio-visually guided positive textual features $T_{i}^{\mathcal{P}}$ and the textually guided positive audio-visual features $C_{i}^{\mathcal{P}}$ are derived as follows:

where $W^{\mathcal{Q}}_{\mathcal{P}},W^{\mathcal{K}}_{\mathcal{P}},W^{\mathcal{C}}_{\mathcal{P}},W^{\mathcal{T}}_{\mathcal{P}}$ denote the query, key, and two value projection matrices. The parameter $\alpha$ is used to balance the influence of positive and negative attention. Similarly, the positive audio-visual features guided by textual information can be obtained using the same method.

Negative attention is designed to highlight inconsistencies across different modalities. These negative attention scores are calculated using scaled dot-product attention, with a negative constant applied before the Softmax function. The process can be summarized as follows:

where $W^{\mathcal{Q}}_{\mathcal{P}},W^{\mathcal{K}}_{\mathcal{P}},W^{\mathcal{C}}_{\mathcal{P}},W^{\mathcal{T}}_{\mathcal{P}}$ denote the query, key, and two value projection matrices respectively. Moreover, $\gamma$ is denoted as $\gamma=-(1-\alpha)$. After this, we referred to the integration processing of hidden states in the MMRA model to generate a comprehensive textual modal representation in FFN layers, incorporate audio-visual hidden states into textual hidden states to generate a comprehensive textual and audio-visual modal representation $\widetilde{T_{i}}$ and $\widetilde{C_{i}}$, which modify the calculation of the FFN process as follows:

where $W^{t}_{1},W^{c}_{1}\in\mathbb{R}^{2d\times d}$, $W^{t}_{2},W^{c}_{2}\in\mathbb{R}^{d\times d}$ represent learnable weights matrix via the attentive pooling strategy, $\oplus$ denotes the concatenation operation. Finally, we exploit expressive representations $T_{i},C_{i}$ by the attentive pooling strategy.

This dual-path approach ensures a synthesis of audio-visual and textual modalities, each subjected to a consistent and rigorous extraction process. This method not only captures the intrinsic characteristics of micro videos but also lays a solid foundation for subsequent multifaceted analyses and applications, thereby enhancing the depth and breadth of our understanding of video content.

We focus on extracting valuable insights from relevant instances retrieved to improve micro video popularity prediction (MVPP). To achieve this, we use an aggregation function that combines the comprehensive representations of these instances from the memory bank. This function assigns attention scores based on the normalized similarity scores obtained during retrieval, using them as weights to construct the retrieved embeddings $X_{i}^{r,c}$ and $X_{i}^{r,t}$. Instances with higher similarity scores are prioritized, highlighting their relevance to the target micro video.

Similarly, we calculate $\widetilde{T_{i}^{r}}$ and $\widetilde{C_{i}^{r}}$, and then derive the final representations $T_{i}^{r}$ and $C_{i}^{r}$ using an attentive pooling strategy. To capture the popularity trends of the retrieved instances, we encode the label information through linear layers and aggregation, resulting in the retrieved label embedding $L_{i}^{r}$. Finally, AUG-MMRA integrates all features from $T_{i}^{r}$ and $C_{i}^{r}$ to model cross-sample interactions. These feature interactions are constructed as

where $\Phi$ denotes the process of inner products.

In our micro-video popularity prediction model, we adopt SASRec (Self-Attention Sequence Recommender), a Transformer-based architecture, to capture users’ sequential behavior patterns. SASRec is crucial for modeling both short-term and long-term user preferences, incorporating multimodal item features to enhance recommendation accuracy.

Our model processes various modalities, including video frames ($F_{i}$), cover images ($I_{i}$), and titles ($T_{i}$), which are extracted using pre-trained encoders. The user interaction histories are fed into the SASRec module, where self-attention layers encode sequential user behaviors, producing a sequence-based representation of user-item interactions. Simultaneously, the multimodal features are integrated to construct the item scoring representation item_scoring. The user embedding $U_{i}$, which encapsulates the user’s behavior sequence, is then combined with this multimodal item representation.

For each user, we calculate a score for each item by computing the dot product between the user embedding $U_{i}$ and the multimodal item scoring item_scoring:

The final score for an item across all users is obtained by averaging these individual scores:

where $N$ represents the total number of users. These averaged scores are converted into probabilities using a sigmoid function to predict the likelihood of interaction:

The model is trained by minimizing the cross-entropy loss between the predicted scores and actual user interactions (positive and negative samples). By integrating multimodal item features (video, image, text), SASRec provides a rich representation of user-item interactions. The model optimizes its predictions through backpropagation, adjusting parameters based on user behavior sequences.

Denote the sequence $\mathcal{S}=\{(t_{1},r_{1}),(t_{2},r_{2}),...,(t_{n},r_{n})\}$ as the temporal users’ reaction sequence, where $R$ is the number of events. Let $\Delta_{i}=t_{i}-t_{i-1}$ represent the temporal differences, with $\Delta_{1}=t_{1}$ for convention, hence the temporal sequence is represented by

Additionally, let $\mathbf{r}_{i}$ be the one-hot vector of the events. Inspired by the discretization

the hidden states differ by time gap $\Delta t$ are related by the formula, we put the temporal differences into the equation directly to construct our Mamba Hawkes Process (MHP) structure. Now we state our construction.

Let $W^{e}$ be the event embedding matrix with dimensions $D\times R$, where $D$ is the dimension of the hidden layers of Mamba blocks. The event embedding is defined as $x_{t_{i}}=\mathbf{r}_{i}(W^{e})^{T}$

In the Mamba architecture, the matrices $\Delta,\mathbf{B}$ and $\mathbf{C}$ are time-dependent and are obtained by linear projection from $x_{t}$. However, for the Hawkes Process, the approach is different, as it requires the use of temporal features. Specifically, we make $\mathbf{B}$ and $\mathbf{C}$ time-dependent, and $\Delta=(\Delta_{1},\Delta_{2},...,\Delta_{n})$ is defined by the temporal differences as above. Following Equation 41, we have $\Delta_{i}=t_{i}-t_{i-1}$, thus we replace $t$ and $t+\Delta t$ by $t_{i-1}$ and $t_{i}$. We define

are obtained by a linear transformation of the vector $x_{t_{i}}$. We have the transition formulas for the Mamba Hawkes process:

where

is the temporal-dependent coefficients. Hence, the temporal information is incorporated into our recurrence process.

Then we got the final output, which is denoted as $\mathbf{O}=(\mathbf{o}_{1},\mathbf{o}_{2},...,\mathbf{o}_{n})$, The attention output $\mathbf{O}$ is then fed through a neural network, generating hidden representations $\mathbf{h}(t)$ of the input event sequence:

where $W_{i},b_{i}(i=1,2)$ are the parameters for the neural network. The resulting matrix $\mathbf{H}$ contains hidden representations of all the events in the input sequence, where each row corresponds to a particular event.

To avoid “peeking into the future”, our algorithm is equipped with masks. That is, when computing the attention output $\mathbf{h}(j,:)$ (the $j$-th row of $\mathbf{h}$), we mask all the future positions, i.e. we only choose the first 5 user-item interactions to train. This will preventing the function from assigning dependency to events in the future.

The intensity function of Mamba Hawkes process is given by

where $\lambda_{r}$ is the intensity function of the $r$-th event, and

where $t$ is defined on interval $t\in\left[t_{j},t_{j+1}\right)$, and $f_{r}(x)=\beta_{r}\log\left(1+\exp\left(x/\beta_{r}\right)\right)$ is the Softplus function. For the log-likelihood based on a sequence $S$, it is given by

To evaluate the effectiveness of RecMMR, we conduct experiments on the micro-video dataset: MicroLens-100k. Its descriptive statistics are summarized in Table 5. MicroLens-100k consists of 19,738 unique micro-videos viewed by 100,000 users from various online video platforms.

We attempt to search for other datasets to enrich our experiment, but unfortunately by now among all acknowledged datasets, it is difficult to find a well-structured multi-modal micro-video dataset with a scale like MicroLens-100k dataset. A detailed comparison of the datasets is provided in Table 6.

Here we elaborate how base models works.

• •

  SVR: Uses Gaussian kernel-based Support Vector Regression (SVR) to predict micro-video popularity.

• •

  HyFea: Employs the CatBoost tree model, leveraging multiple features (image, category, space-time, user profile, tags) for accurate prediction.

• •

  Contextual-LSTM: Integrates contextual features into the LSTM model, capturing long-range context to improve prediction accuracy.

• •

  TMALL: Introduces a common space to handle modality relatedness and limitations, enhancing popularity prediction.

• •

  MASSL: Utilizes a multi-modal variational auto-encoder model, capturing cross-modal correlations to predict popularity.

• •

  MTFM: Combines fuzzy trend matching and Informer in a multi-step prediction model, forecasting popularity trends.

• •

  HMMVED: Extracts and fuses multi-modal features through a variational information bottleneck for better prediction.

• •

  CBAN: Applies cross-modal bipolar attention mechanisms, effectively capturing correlations in multi-modal data to enhance prediction.

• •

  MMRA: Enhances prediction by retrieving relevant instances from a multi-modal memory bank and augmenting the prediction process.