LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models

Syntax-based parsers detect templates by identifying repeating patterns as static and others as parameters. Frequency-based parsers like SLCT (Vaarandi, 2003), LFA (Nagappan and Vouk, 2010), LogCluster (Vaarandi and Pihelgas, 2015), and Logram (Dai et al., 2020), build on token recurrence. Similarity-based parsers, including LKE (Fu et al., 2009a), LogSig (Tang et al., 2011), LogMine (Hamooni et al., 2016), SHISO (Mizutani, 2013), and LenMa (Shima, 2016) cluster logs by similarity. Heuristics-based parsers such as AEL (Jiang et al., 2008), IPLoM (Makanju et al., 2009), Drain (He et al., 2017), Spell (Du and Li, 2016), Brain (Yu et al., 2023), and MoLFI (Messaoudi et al., 2018), apply specific strategies including the longest common subsequence-based approach, iterative partitioning, prefix trees, and evolutionary algorithms for template extraction. These methods are fast and cost-efficient but may miss semantic details and require domain-specific tuning.

Semantic parsers have evolved with neural networks like bidirectional LSTM, as seen in Semparser (Huo et al., 2023) and Uniparser (Liu et al., 2022), and pre-trained language models such as LogPPT (Le and Zhang, 2023b). VALB (Li et al., 2023) further enhances the model’s semantic understanding by classifying specific parameter categories. These models require labeled data for training and classify tokens into templates or parameters. They offer semantic understanding and can generalize across log types, but also demand resource-intensive training and periodic updates, presenting significant operational challenges.

Recent studies (Wang et al., 2022, 2023) have incorporated user feedback into log parsers, facilitating human-in-the-loop log parsing. This approach not only enables the parser to swiftly adapt to evolving logs but also enhances the accuracy of template mining.

Large Language Models (LLMs) have emerged as transformative tools in numerous domains, demonstrating their prowess and versatility. Their pre-training on vast datasets, which include diverse content such as code and log data, makes them particularly adept for specialized tasks like log parsing. Studies like  (Xu et al., 2024; Mudgal and Wouhaybi, 2023; Le and Zhang, 2023a; Liu et al., 2023) have begun to tap into this potential, primarily focusing on prompt engineering to improve template extraction efficiency. While these advancements highlight the promise of LLMs in log parsing, they predominantly utilize a line-by-line parsing approach. This method, although innovative, leads to high computational demands due to LLMs’ extensive parameter spaces, making these approaches impractical for real-world applications due to the significant computational overhead.

The benefits of LLMs extend beyond their raw computational ability, offering deep semantic understanding and the capacity to generalize across different log formats, adapting seamlessly to new data types. This adaptability is crucial, as it reduces the need for extensive preprocessing, hyper-parameter tuning, and manual labeling, streamlining the deployment process.

Despite these advantages, the practical deployment of LLMs in log parsing is hindered by their high operational costs. Effective utilization requires careful prompt tuning, a process that can be as resource-intensive as the computational demands of the models themselves. This challenge underscores the need for more efficient approaches that can leverage the strengths of LLMs without incurring prohibitive costs, ensuring their viability for broader, real-world application.

The granularity of log parsing is pivotal for how the parsing result looks like. We primarily characterized the granularity by two dimensions: specificity and applicability.

Existing evaluation metrics, while versatile, emphasize either the accuracy of grouping logs or the fidelity in extracting templates and parameters. Both dimensions are indispensable, especially considering their implications for downstream tasks like log anomaly detection. However, these metrics often overlook the subtle granularity differences inherent in log parsing. Existing benchmark datasets (Zhu et al., 2019; Jiang et al., 2023) are anchored to the annotators’ subjective interpretations, suggesting that multiple valid granular interpretations can exist for a single log. Such diversity challenges the conventional wisdom of treating annotated labels as an unequivocal gold standard. Instead of a myopic focus on exact matches, a more encompassing metric that can quantify and understand this granularity discrepancy is imperative.

Our method hinges on minimal preprocessing, using only a basic regular expression to extract log content. While many approaches demand greater domain knowledge, often employing regular expressions to substitute common variables like IP addresses and block IDs (He et al., 2017), we retain the original message, ensuring the LLM grasps the log’s full context. Unlike other strategies that use distinct separators for log tokenization (Yu et al., 2023; Liu et al., 2022; Fu et al., 2022), we consistently tokenize using spaces. Hence, unless otherwise specified, tokens in following sections are space-separated, capitalizing on the LLM’s native tokenizer. This streamlined preprocessing minimizes the need for specialized expertise, yet upholds strong log parsing efficacy.

Central to our methodology is a base algorithm employing a prefix parse tree, inspired by the efficiency demonstrated in Drain (He et al., 2017). This section elaborates on the data structures integral to the algorithm, detailing their design and their roles in addressing the aforementioned principles. Specifically, we’ll elucidate how incoming logs are matched with existing clusters during tree traversal, how and when LLMs are invoked for template extraction, and the dynamics of updating the tree with new templates obtained from the LLM extractor.

While the base algorithm already paves the way for efficient and precise log cluster matching, there remains room to refine the accuracy of the LLM template extractor. To this end, we introduce variable-aware prompting, amalgamating it with in-context learning. This fusion not only amplifies the LLM’s task comprehension but also augments its overall performance.

Additionally, there exist other straightforward avenues to bolster template extraction capabilities. One could leverage more powerful LLMs available in the ever-evolving landscape of language models. Alternatively, supervised fine-tuning of an LLM using labeled data presents another viable strategy.

Integrating human expertise into the automated log parsing process is key to achieving the right granularity. Human input can be seamlessly incorporated at various stages of the parsing pipeline to enhance accuracy and maintain consistency:

1) Pre-Processing Intervention: Experts annotate a sample of logs before parsing begins. These annotations serve dual purposes: they can be used as seed examples for In-Context Learning (ICL) or to fine-tune LLMs, ensuring the model’s output aligns more closely with specific parsing needs.

2) Real-Time Calibration: During the parsing process, human judgment can be applied to guide decisions on template merging, ensuring the parsing maintains the desired level of granularity throughout.

3) Post-Processing Refinement: After parsing, the system identifies potential merges or splits based on semantic similarity or template variability. Experts review these suggestions, making adjustments to achieve the optimal granularity.

In Table 1, we demonstrate how LogParser-LLM-C incorporates pre-processing intervention, enhancing the base LogParser-LLM’s capabilities. For real-time calibration, human expertise can be used to refine the merging process in line with desired granularity levels, as outlined in line 15 of Algorithm 1. Post-processing refinement can integrate methods like those suggested in (Wang et al., 2023) for effective final adjustments, ensuring parsed logs accurately reflect the intended granularity.

In alignment with prevailing methods outlined in (Liu et al., 2022; Khan et al., 2022; Jiang et al., 2023), we utilize the GA, PA, FGA, PTA, RTA and FTA metrics delineated in Appendix B for evaluation. Furthermore, we use the Grouping Granularity Distance (GGD) proposed in Section 3.2.2 as a more intuitive metric to gauge the granularity discrepancies in parsing outcomes.

Our primary objective in conducting experiments with the smaller-scale Loghub-2k dataset is to assess the efficacy of our method’s key components. Additionally, we employ this dataset as a development set, refining our prompts for LLM template extraction, merging check and verification. The final prompts we adopted are presented in Appendix. B. The most effective configurations determined through Loghub-2k are subsequently applied unchanged to the LogPub dataset for evaluation.

Comparison with Existing LLM-based Parsers Existing LLM-based parsers, which process logs line-by-line, are impractical for evaluation on the expansive LogPub dataset due to the immense number of LLM calls required. We therefore use the Loghub-2k dataset for comparison, but advise caution in interpreting these results because of the dataset’s limited scope and the possibility that a few well-chosen labeled samples might cover the majority of templates. In Table 3’s results, our method either matches or exceeds the performance of existing approaches with an equivalent number of labeled logs, highlighting our method’s effective use of LLMs for log parsing despite the constraints.

Accuracy Evaluation We compare our model to three state-of-the-art methods: two syntax-based methods, Drain (He et al., 2017) and Brain (Yu et al., 2023), and one semantic-based method, LogPPT (Le and Zhang, 2023b). As indicated in Table  4, the previous state-of-the-art methods achieved higher GA and PA values because they were meticulously tuned with hyperparameters on each dataset to optimize these metrics. However, these values alone do not necessarily indicate superior performance. When evaluating with template-level metrics such as FGA and FTA, as well as our proposed GGD, our model outperforms them without the need for any domain-specific configuration.

Ablation The results of our ablation study for different components, including in-context learning (ICL), variable-aware prompt (VA), and automatic template merge (Merge), are presented in Table 4. The numbers clearly demonstrate that each proposed component positively impacts the method’s performance, as evidenced by the reduction in GGD. Notably, the most significant performance boost comes from the transition from GPT-3.5 to GPT-4. Furthermore, the enhancements from other components are even more pronounced with GPT-4, underscoring the potency of more powerful LLMs. Using GPT-4 on its own, even without ICL, yields impressive results, showcasing its capacity to adhere to specific instructions and complete tasks in a zero-shot scenario. However, it is important to note that integrating these components also increases the associated costs when invoking the LLM.

Accuracy and Generalizability Results from the expansive logPub dataset are shown in Table 1. We use LogParser-LLM-C to denote calibrated variants of our method. It is clear that our model, LogParser-LLM , even without granularity calibration, significantly surpasses all baseline methods in GA, FGA, and PTA, marking improvements of 7.8%, 48.3%, and 32.0% compared to the best baseline results. However, PA performance lags, mostly due to granularity nuances complicating the LLM’s ability to generate templates that perfectly match annotated labels. A standout point is the consistent performance of our method across the 14 datasets, achieved without domain-specific tweaks, maintaining uniform settings throughout. Upon introducing domain-specific granularity calibration with ICL in LogParser-LLM-C, there is a noticeable boost, especially in template parsing metrics such as PA and FTA. This highlights the reduced discrepancy in the applicability of log parsing achieved through ICL.

Granularity Discrepancy Evaluation Both Grouping Granularity Distance (GGD) and Parsing Granularity Distance (PGD) are calculated and shown in Table 1 . PGD is computed using spaces as delimiters for tokenization, representing a lower bound since precise tokenization isn’t feasible for such large datasets. This approximation remains valuable for consistent cross-method comparisons.

Unlike message-level GA and PA metrics, which depend on log volume, the proposed metrics avoid template imbalance and provide a clearer performance indicator. For example, in the Linux dataset, Drain’s GA is 68.6 compared to our 53.4. However, Drain’s GGD is 30 versus our 10, indicating significantly more effort needed to align Drain’s results with the ground truth.

Compared to template-level metrics, GGD and PGD show that smaller GD correlates with higher FGA and FTA. However, FGA and FTA can overly penalize repetitive differences. For example, if a ground truth template ”instance: ¡*¿” has many instance IDs not correctly identified as variables, it increases the number of identified templates, skewing precision calculations. GGD and PGD count such differences only once, offering a fairer measurement. For instance, GA and PA for Uniparser on OpenSSH are 0.9 and 0.5, respectively—values that indicate a significant gap compared to other methods and are not informative. Conversely, GGD and PGD for Uniparser on OpenSSH are 15 and 26, respectively, providing an informative and intuitive comparison. This robustness is also observed in HealthApp, OpenStack, and Thunderbird datasets.

Evaluation with Different LLMs By design, our framework is versatile enough to be compatible with any language model that can process individual log messages and accordingly generate log templates. This evaluation’s primary objective is to assess the impact of different LLMs on the efficacy and efficiency of our approach.

Our results, as presented in Table 2, demonstrate that using GPT-4 as the LLM template extractor paired with ICL yields optimal performance. However, this comes at the cost of increased computational time due to GPT-4’s extensive parameter count. Notably, both granularity calibration and ICL enhance performance and concurrently decrease the number of required LLM calls. This is congruent with our framework’s foundational assumption that LLMs can generate nearly perfect log templates. For the fine-tuned LLM, we commenced with the widely-recognized open-source LLM, Llama-2-13b. Despite its performance not being optimal, it remains competitive, closely paralleling the results of prior state-of-the-art semantic-based models like Uniparser and LogPPT. This subpar performance may be attributed to our not having meticulously curated the fine-tuning dataset and its limited size. While a more thoughtfully curated, expansive training dataset and hyperparameter tuning could enhance its performance, this conflicts with our intent: to construct a robust log parser that necessitates minimal human intervention and domain-specific knowledge.

For runtime efficiency, we delineate runtime into two facets: the time required for LLM calls and the time for our log parser’s other operations. This distinction is crucial, given that OpenAI service calls depend on service availability and are subject to rate limitations, which complicates consistent performance evaluation. To estimate cumulative processing time, we multiply the average response time (in optimal scenarios) by the total number of LLM calls. Our base algorithm’s average runtimes stand at 461.67s and 433.22s. If we disregard potential rate caps, the mean response times are 0.52s for GPT-3.5 and 4.18s for GPT-4. With an average of 272.5 LLM calls, this equates to overall runtimes of 612.31s for GPT-3.5 and 1572.27s for GPT-4. For comparison, Drain, one of the fastest existing methods, averages 483.2s to process about 3.6 million logs. In contrast, a conventional line-by-line LLM parsing approach would necessitate 3.6 million LLM calls, leading to an impractical runtime of approximately 22 days for GPT-3.5, not accounting for other operational overheads. This analysis not only emphasizes our method’s competitive efficiency but also its practicality, overcoming the inherent impracticality of existing LLM-based parsers by significantly reducing the number of necessary LLM calls.