Curriculum Recommendations Using Transformer Base Model with InfoNCE Loss and Language Switching Method

In this section, we will delve into our points of innovation. By implementing a Transformer Base Model with a limited sequence length, utilizing InfoNCE Loss, and incorporating a language switching strategy, our approach aims to create a more equitable and effective learning environment for diverse learners.

To address challenges in topic-content matching, we employ a Transformer Base Model with the InfoNCE Loss, as detailed in Vaswani et al.[10] and Feng et al.[11]. Illustrated in Fig.1, this architecture enables nuanced learning of topic-content relationships. Leveraging Transformer Base Models to encode both topics and content, the design empowers the model to capture intricate patterns and dependencies within the input data.

By utilizing the InfoNCE Loss[12], our model aims to reduce noise in the training process, particularly when dealing with topics that share similar content. Unlike traditional cross-entropy loss, InfoNCE Loss emphasizes correct matches on the diagonal of the similarity matrix during loss calculation. This innovative loss function contributes to the model’s ability to precisely identify and match topics with their corresponding content. The InfoNCE Loss is formulated as follows:

Where:

• •

  $\text{sim}(x,y)$ is a similarity measure between samples $x$ and $y$.

• •

  $y^{+}$ is the positive sample similar to $x$.

• •

  $y_{k}^{-}$ is the $k$-th negative sample dissimilar to $x$.

• •

  $K$ is the number of negative samples.

This loss function guarantees that, during the loss calculation, emphasis is placed solely on the diagonal of the similarity matrix. This is vital to prevent misinterpretation of elevated similarities between topics and content within the same batch. By focusing exclusively on the diagonal, the loss function ensures a more accurate assessment of matching within the context of the specific batch, mitigating the risk of erroneous interpretations of similarities between disparate elements in the dataset.

Moreover, we integrate a specialized shuffle function to construct batches that prevent the co-occurrence of topics and related content. This approach is essential for minimizing noise and conflicts in the training process. Additionally, we employ a sampling strategy for missing and incorrect content, ensuring that the model effectively learns from both positive and challenging negative samples. This comprehensive approach enhances the model’s robustness by systematically addressing potential biases and difficulties inherent in the training data, promoting a more accurate and well-rounded learning experience.

We adopted a translation approach covering prevalent languages called language switching method. However, directly incorporating translated data into our training set presents challenges with the InfoNCE loss. For instance, translating an English topic and content item into French, where a corresponding topic and content already exist in the original French data, would introduce noise in loss calculation. To address this, we implement a careful filtering mechanism to exclude translated pairs with similar counterparts in the target language, ensuring that the augmented data contributes meaningfully to the training process without introducing redundancies or misleading information during loss calculation.

As shown in Fig.2, the switching strategy occurs every second epoch and is confined to languages like English (en), Spanish (es), Portuguese (pt), and French (fr). Despite anticipated noise in training, language switching contributes to a score increase of approximately 0.01–0.02. This modest gain implies that the introduced noise might influence training dynamics positively. The periodicity of the switching strategy, coupled with its language limitations, strategically introduces variability during training, and the observed score boost suggests a nuanced impact on the model’s ability to adapt and generalize, showcasing the strategy’s subtle yet meaningful role in enhancing the overall training process.

The dataset originates from the Kolibri Studio curricular alignment tool, allowing users to create channels, construct topic trees representing curriculum taxonomies, and organize content items. Users can upload their content or import materials from the Kolibri Content Library. The challenge is to predict the optimal alignment of content items with topics, mimicking the choices made by curricular experts and Kolibri Studio users. The widespread integration of AI is evident[13, 14, 15]. The goal is to streamline the curator’s task of discovering pertinent materials for each topic by recommending content items. The comprehensive test set comprises 10,000 additional topics (absent in the training set) and numerous extra content items exclusively correlated with the test set topics. This expansion aims to assess the model’s ability to generalize and recommend relevant content beyond the training data, simulating real-world scenarios where novel topics and content continuously emerge.

The $F_{2}$ score[16], also known as the F-beta score, serves as a valuable metric for assessing the accuracy of a test. It represents the weighted harmonic mean of precision and recall, with an emphasis on recall. The general formula for the F-beta score is expressed as:

Here, precision is defined as $\frac{TP}{TP+FP}$, and recall is defined as $\frac{TP}{TP+FN}$, where TP is the number of true positives, FP is the number of false positives, and FN is the number of false negatives.

For the $F_{2}$ score, $\beta=2$, indicating that recall is considered twice as important as precision. Substituting these values into the F-beta formula, we derive the following formula for the $F_{2}$ score:

To obtain the mean $F_{2}$ score across all predictions in a dataset, calculate the $F_{2}$ score for each row and then compute the average. Additionally, a 10-fold cross-validation (CV)[17] split strategy is employed, aiming to minimize the overlap of content relations between different folds. This involves creating 10 buckets and assigning topics to the bucket where all attached content results in the least overlap with the same attached content in other buckets. While achieving a perfect split aligned with the leader board is challenging due to the n x m relation of Topic x Content, efforts have been made to minimize the overlap during CV-split creation.

In this section, we evaluate the performance of our model using a diverse set of pre-trained models to predict content alignments for given topics. Notable models such as sentence-transformers/LaBSE[11], facebook/mcontriever-msmarco[18], sentence-transformers/stsb-xlm-r-multilingual, and sentence-transformers/paraphrase-multilingual-mpnet-base-v2[19] were employed. Training involved fold 0 of 10 Folds, with a max sequence length of 96 and a batch size of 768 pairs of [topic, content]. Utilizing polynomial decay with warmup (2 epochs) for 40 epochs and a learning rate of 0.0003. Table I presents the cross-validation scores, showcasing competitive performance across the models.

The competitive cross-validation scores highlight the efficacy of our sentence-transformers/LaBSE employing the InfoNCE loss and Language Switching leading with a score of 0.66314. These outcomes underscore the effectiveness of our methodology in capturing diverse linguistic nuances for content alignment prediction. The ensemble approach, incorporating innovative techniques, proves to be a robust solution for the task, showcasing its ability to yield competitive results in comprehensively addressing the challenges associated with content alignment prediction in the context of diverse linguistic nuances.