Investigating LLM Applications in E-Commerce

Instruction fine-tuning represented a pivotal advancement in the optimization of large language models (LLMs), such as GPT-4, for enhanced task-specific performance, especially in domain-specific applications (Hu et al., 2023; Zhang et al., 2024). This method involved the supplementary training of a pre-trained based model such as GPT (OpenAI et al., 2023), Llama (Touvron et al., 2023), or Falcon (Almazrouei et al., 2023) on a task specific dataset consisting of prompts paired with their optimal responses. The objective was to refine the model’s capacity to comprehend and execute instructions with increased accuracy and contextual relevance. Instruction fine-tuning has emerged as an invaluable technique for augmenting the proficiency of LLMs across various specialized domains, ensuring their outputs align more closely with user expectations and requirements.

Low-Rank Adaptation (LoRA) (Hu et al., 2021) is an innovative technique designed to fine-tune (LLMs) in a resource-efficient manner. This method addresses the challenge of adapting pre-trained models to specialized tasks without the extensive computational costs associated with traditional full-model fine-tuning. At the heart of LoRA is the strategic introduction of trainable low-rank matrices that target specific components of the LLM’s architecture, namely the attention and feed-forward neural network layers inherent to the transformer model. Specifically, it freezes the pre-trained layers of the LLM, and for each layer, it trains a rank-decomposition matrix and injects them into each layer of the pre-trained model to accomplish the LLM fine-tuning.

LoRA involves the addition of low-rank matrices $\mathbf{A}$ and $\mathbf{B}$ to the existing weight matrices $\mathbf{W}$ of the model. The model’s original parameters are kept frozen during fine-tuning while $\mathbf{A}$ and $\mathbf{B}$ are updated. These matrices $\mathbf{A}$ and $\mathbf{B}$ are much smaller in size compared to $\mathbf{W}$, enabling significant reductions in the number of trainable parameters. The adaptation occurs through the equation. $\mathbf{W}^{\prime}=\mathbf{W}+\mathbf{A}*\mathbf{B}^{T}$, where $\mathbf{W}^{\prime}$ represents the adapted weights. This process selectively fine-tunes the model, allowing it to acquire new capabilities or improve performance on specific tasks with minimal adjustments to its pre-trained parameters. This selective updating is particularly beneficial for domain-specific applications, where only certain aspects of the model’s knowledge need refinement.

With the rise of text generation models that are seemingly capable of performing large numbers of tasks and able to answer many questions, a number of evaluation strategies have been proposed. Evaluation leaderboards often consist of evaluation tasks like Hellaswag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021b), and others (Gao et al., 2023; Hendrycks et al., 2021b, a) which cover a broad range of multiple choice questions. These types of multiple choice question evaluations, where the answer is chosen based on the choice with the highest likelihood can be brittle as rankings can be sensitive to minute details (Alzahrani et al., 2024). Other evaluation benchmarks like GLUE (Wang et al., 2018) consist of a bundle of different tasks with task specific evaluation metrics. Another approach to evaluating LLM performance is the approach of LLM as a judge. Chatbot arena (Zheng et al., 2024) is an example of this style of evaluation and while it can correlate with human judgment, like other LLM applications it can be subject to hallucinations. In this work, we focus on directly evaluating the tasks of interest with existing scoring practices.

Zhang et al. (2024) find that fine-tuning LLM scaling factors appear to be very task dependent; however, there has been little published work examining fine-tuning focusing on the e-commerce domain. There has been some prior work investigating the use of LLMs on e-commerce tasks. ECOMGPT (Li et al., 2023) looked at framing e-commerce tasks as instruction fine tuning, but doesn’t explore LoRA (Hu et al., 2021) or how different tasks enhance or interfere with each other or the amount of data required for reasonable performance.

The Shopping Queries ESCI dataset (Reddy et al., 2023) contains search queries, released with the aim of fostering research in the area of semantic matching of queries and products. The dataset contains three tasks: Query-Product Ranking, Multi-Class Product Classification, Product Substitute Identification. We use the ESCI Multi-Class Product Classification task. The task is to classify a query and product pair as an exact match (E), a substitute (S), a complementary product (C) or Irrelevant (I). Because Query-Product Ranking and Product Substitute Identification involve more complexity and longer input strings, we do not include them for LLM evaluation in this work.

QueryNER (Palen-Michel et al., 2024) is an e-commerce query segmentation dataset. The task in QueryNER is not to extract aspects, but rather to segment the user’s query into meaningful chunks. QueryNER uses a subset of the Amazon Shopping Queries Dataset (Reddy et al., 2023) as the underlying data. QueryNER consists of an ontology of 17 types. The entity types are: core_product_type, product_name, product_number, modifier, creator, condition, unit of measurement (UoM), department, material, time, content, color, shape, quantity, occasion, origin, and price.

Because QueryNER uses BIO encoding (B for begin, I for inside a span, O for outside a span), we linearized the data in order to have an input prompt and output string. The formatting of the linearized data is a series of (token, label) pairs.

AMASUM (Bražinskas et al., 2021) is a dataset for summarizing product reviews. The product reviews are in English and come from bestreviews.com, cnet.com, pmag.co.uk, runrepeat.com, which mainly consist of electronics and running shoes reviews. The dataset contains a list of product reviews and a summary with the “verdict” on the product and also lists of pros and cons. The original paper focuses on selecting useful reviews in order to summarize.

The goal of our evaluation of LLM performance is not to assess its review selection capability, so we select a small number of reviews to summarize. We select only 4 reviews to be used to generate the verdict summary. Since the dataset has a field with helpful votes where users voted that the review was helpful, we take the top four reviews with the most helpful votes as our selection process.

There appears to be a lack of standard benchmark datasets in English for product description generation despite a decent amount of prior work. Koto et al. (2022) stated they were not able to release the dataset due to copyright issues. Chan et al. (2019) and Zhang et al. (2019) collected data from Taobao¹¹1https://www.taobao.com/. Zhang et al. (2019) also stated that there was no other standard dataset for product description generation. Wang et al. (2017) created their dataset from attribute values and descriptions from Amazon but only in the category ”Computers and Tablets”.

Without a clear prior benchmark for this task, we create a simple product description task from the ESCI Shopping Queries Dataset (Reddy et al., 2023). We assemble an input consisting of a product title, brand, color, and bullet points. The bullet points in the original dataset are aspect-value pair information about a product or short snippets about the product. The expected output is the product description. We filter out items where there is no title, no description, no bullet points, or items where the description is an exact match of the title or bullet points.

To enable instruction fine-tuning, each individual dataset was required to be aligned to a sequence to sequence style task in order to be used with an LLM. Review summarization and product description generation already were easily treated as sequence to sequence tasks. However it was less obvious how to best treat classification and sequence labeling tasks. We treated the classification label as text to be generated. For sequence labeling, the model was expected to output tuples of each token along with its label.

Prompts were designed to provide the model with enough context to accomplish its task. We created task specific templates for each of the tasks. Each prompt asked the model to act as an e-commerce expert to provide context. The prompt then gave a brief description of the task and along with the training example. Examples of prompts are shown in Appendix Appendix.

We chose to use the most common and competitive baselines for each e-commerce task. For classification tasks, ESCI Task 2 and QueryNER, we chose to use BERT (BERT-base) (Devlin et al., 2019) as the baseline model with a learning rate of 3e-5 following Devlin et al. (2019). The training of BERT followed the conventional formulation of the sequence classification problem for the ESCI task and token classification for the QueryNER task.

For generative tasks, review summarization, and product description generation, we chose to use T5 (T5-base) (Raffel et al., 2020) as a baseline model. T5 and BERT are fine-tuned with default parameters released by Hugging face (Wolf et al., 2020). Because in-context learning has been shown to be effective (Dong et al., 2024), we use we include Mixtral 8 x 22b (Jiang et al., 2024) as a theoretical state of the art zero-shot and few-shot baseline.

Mixing tasks (datasets) for fine-tuning large language models (LLMs) can enhance the model’s performance, generalization ability, and adaptability to various tasks, which closely mirrors the industrial application. The trained model was not required to accomplish one task but rather several domain-specific tasks such as query named entity recognition (NER), text summarization, description generation, and classification. Fine-tuning LLMs with mixed and diverse training datasets could help improve performance on each task.

Furthermore, thanks to the nature of the LoRA framework, mixed tasks (datasets) training could also be achieved by merging LoRA weights independently trained on different tasks. Specifically, assume there were $n$ tasks with a specific task number $i\in\{0,\ldots,n-1\}$. LoRA weights trained independently from each task could be defined as $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$. The adaptation equation for mixed LoRA merge was $\mathbf{W}^{\prime}=\mathbf{W}+\frac{1}{n}\sum_{i=0}^{n-1}\mathbf{A}_{i}\mathbf{B}_{i}^{T}$. Mixing LoRA merging provided additional flexibility since additional tasks could be added later instead of retraining whenever a new task was added. Additionally, some types of tasks may benefit each other, while others may lower another tasks performance when merged. Mixing LoRA merge enabled more efficient experimentation with different combinations of tasks compared with directly training with the mixed data set.

The foundation model used in all LoRA fine-tuning experiments was Llama-2-7b as its moderate size and considerable performance on various tasks (Touvron et al., 2023). The supervised fine-tune was conducted with $3$ epochs for each dataset, as we do not see performance gain while we trained it longer. Since all four tasks for the LLM were formulated as a text generation task (see Section 3.5), we followed the most common hyperparameters to finetune the LLM under the text generation setup. Specifically, the model was loaded with $8$ bit, the max length of input was set to $1500$, and the precision of the parameter was set to be bf16. During the supervised fine-tune, we adopted the LoRA (Hu et al., 2021) to conduct efficient model training. The LoRA $\alpha$ was set to be $16$, with dropout rate be $0.05$. The initial learning rate was chosen to be $3e-5$ with a cosine learning rate scheduler. To further improve the training efficiency, we adopted paged_adamw_8bit as optimizer. The LLM model training were conducted on two NVIDIA A100 80GB GPUs.

To optimize computational time, we randomly sampled subsets from each dataset for our experiments. Specifically, for the ESCI dataset, we used samples of 5k, 10k and 50k as the training set and sampled 1k from the original test set to serve as the test set. For the QueryNER dataset, we selected 1k, 5k, and 8k (full 7,841 data samples) samples for training and used the entire original test set as the test set. In the case of the description generation dataset, we randomly chose 1k, 5k, 10k, and 25k samples for training, with 10k samples from the original test set used as the test set. Finally, for the Review Summarization dataset, we sampled 1k, 5k, 10k, and 25k (the entire 25,203 dataset) as the training set and utilized the complete original test set as the test set. All experiments were conducted using the same aforementioned hyperparameters.

Since the datasets we explored contained both classification and generative tasks, we use the task-specific metric to evaluate the performance of the model performance on various datasets. For classification tasks, we use F1 score as evaluation metrics and report both micro-average and macro-average results, while for the generative task, we use the Rouge-1 F1 (Rouge-1) and Rouge-L F1 (Rouge-L) (Lin and Och, 2004) to evaluate the performance.

To ensure accurate mapping of the generated text to the actual class label in the classification task, we implemented a simple but strict evaluation approach. For ESCI classification, the LLM output was required to match the corresponding label exactly. Any deviation from the exact label resulted in the classification being considered incorrect. For QueryNER, labels were extracted using a regular expression expecting a list of tuples of tokens with BIO tags. If the LLM output deviated from the expected output, the labels were considered all Os (no entities identified). Because we noticed the model sometimes being inconsistent with the output format, the regular expression also handles comma separated output without parentheses. In the case of further formatting issues or when the model does not predict labels for all tokens, it is assumed the model failed to generate a valid label sequence and that particular example is assigned all Os. Palen-Michel et al. (2021) highlighted issues with NER scoring. For scoring procedure clarity, once the model output has been extracted from the linearized form, we use seqeval (Nakayama, 2018) for evaluation using the setting which is equivalent with conlleval.

We performed SFT training of Llama2-7b on each dataset with different portions of the data to explore the impact of training data size for fine-tuning an LLM on each task.

We experimented with merging different pairs of LoRA weights for each pair of tasks. For this experiment, we used the LoRA weights from the 5k training set size. To merge the LoRA weights we took the average of the two. The results of merging LoRA weights, shown in Table 4, demonstrated that when weights trained on a task requiring a more rigid structure in the output like ESCI classification or QueryNER, the performance on those tasks suffers. However, the performance of description generation and review summarization remained comparable with the performance from independent training with the same number of examples.

Upon reviewing model output, we found that at least a portion of this degradation in performance on the tasks requiring a more strict output format was attributable to the output formatting requirements. However, some of this apparent degradation may not truly be quite as bad as it appears.

In Section C of the appendix, we show an example of model output for the QueryNER task was shown where the model did in fact output BIO labels some of which are correct labels, but with formatting unrecognized by the scoring script.

We merged the LoRA (Low-Rank Adaptation) weights from all four tasks for inference and performance analysis on each respective test set. The weights were chosen based on training with 5k samples from each dataset to ensure balanced information contribution. To establish a fair comparison between training with mixed data and mixing through LoRA weight merging, we trained the foundation model (Llama-2 7B) on 5k samples from each dataset, resulting in a total of 20k training samples, using the same hyperparameters as in the previous experiments.

The results, detailed in Table 5, indicated that mixed dataset training resulted in lower performance compared to training each task independently. This performance decrease was attributed to the limited capacity of the LoRA adapter and the distinct nature of each task, which deteriorated the model’s ability to consistently produce outputs in the correct format especially in the classification tasks. Furthermore, the LoRA weight merging approach generally showed inferior performance compared to both mixed dataset training and independent task training. In mixed LoRA merging, classification tasks particularly suffered, with output format issues noted (see Section 4). However, for text generation tasks, the performance remained competitive.