FDM-Bench: A Comprehensive Benchmark for Evaluating Large Language Models in Additive Manufacturing Tasks

This task focuses on analyzing G-code to predict anomalies before the printing process begins. FDM printing is prone to defects like spaghetti (SP), under-extrusion (UE), and over-extrusion (OE), which can compromise both the quality and functionality of printed parts. These issues often arise from configuration settings or parameter values specified within the G-code. For example, UE may result from an insufficient flow rate or low nozzle temperature, whereas OE is commonly due to an excessive extrusion multiplier. Detecting and addressing these issues in the G-code before printing can help save time, materials, and costs associated with failed prints.

In this task, LLMs receive G-code from both successful and defective prints, along with information about the printer model and slicer software. The models are then asked to predict which specific anomaly is most likely to occur or if the part will print without issues.

This task assesses the LLMs’ ability to provide accurate responses to FDM-related questions across various user experience levels. Additionally, we evaluate whether the models can identify and adapt to the user’s level of experience. To ensure both a comprehensive evaluation and efficient automatic assessment, we review the models’ answers to two types of questions.

In this section, we describe the process of generating a G-code dataset by printing a sample bridge geometry on a Prusa 3D printer using PLA filament. By adjusting various printing parameters in Prusa Slicer software, we systematically introduce specific anomalies into the printed samples. The key parameters that are modified include bed temperature, nozzle temperature, print speed, layer height, fill angle, and flow rate (controlled through the extrusion multiplier setting). Figure 1 illustrates the four categories of printed parts in our dataset. These categories include:

• 1.

  Non-defective (ND) prints are created using standard, recommended parameter ranges. For example, with PLA filament, a bed temperature of 60°C and a nozzle temperature of 210°C, combined with standard flow rate settings (extrusion multiplier set to 1), yielded stable prints at typical print speeds and layer heights.

• 2.

  OE occurs when excess filament is extruded, leading to overlapping print patterns and dimensional inaccuracies. To create over-extruded samples, we increase the extrusion multiplier above the standard setting, with values ranging from 1.3 to 1.6. While the flow rate is the primary parameter influencing OE, we also slightly adjust the bed temperature, nozzle temperature, and layer height to replicate realistic variations observed in over-extruded prints.

• 3.

  UE occurs when insufficient filament is extruded, leading to visible gaps and weak bonding between layers. Under-extruded samples are produced by setting the extrusion multiplier below the standard value, specifically within the range of 0.6 to 0.9. As with OE, we also adjust bed temperature, nozzle temperature, and layer height.

• 4.

  SP is a printing anomaly where the filament extrudes into the air rather than onto the print bed, creating a chaotic, spaghetti-like pattern. This error typically arises from two scenarios: (1) poor bed adhesion causes the print to detach and shift freely on the bed, or (2) excessive edge deformation leads to collisions between the print and the nozzle, displacing the print. In our dataset, SP is commonly induced by using low bed temperatures (35-40°C), a 90-degree fill angle (placing filament parallel to the bridge’s long axis), and high print speeds (110 mm/s).

In this section, we include questions relevant to a wide range of FDM users. We define three experience levels to better evaluate the LLMs’ performance in answering questions that require varying degrees of technical and theoretical knowledge:

• 1.

  Beginner user ranges from absolute beginners with no prior experience in 3D printing to early learners who are starting to understand fundamental terms and processes. This group typically requires straightforward, non-technical guidance as they become familiar with basic terminology (e.g., nozzle, filament) and settings. Their focus is primarily on understanding the essentials and troubleshooting initial challenges.

• 2.

  Experienced user ranges from regular operators who have been using FDM printers for several months and are familiar with standard terms, materials, and maintenance practices, to advanced operators or technicians who work with FDM printers professionally. While skilled at troubleshooting and optimizing print settings and maintenance practices, users at this level generally do not engage in research aimed at advancing the underlying FDM technology.

• 3.

  Theoretical user includes individuals who engage in research or theoretical work related to FDM technology. This level ranges from students and research assistants studying the FDM process to professors and faculty members conducting or supervising research. Users at this level often explore complex topics such as inter-layer bonding and material properties to advance FDM technology.

We define both free-form questions and MCQs for these three experience levels as follows:

To evaluate the LLMs’ performance, we craft prompts tailored to the specific requirements of each task, emphasizing reasoning and factual accuracy over creative output. Accordingly, we set the temperature parameter to zero across all models. The temperature parameter, which ranges from 0 to 1, controls the randomness in a model’s responses. Lower values produce more consistent outputs, while higher values introduce variability for creative responses.

In addition, we retain each model’s default settings without hyperparameter tuning. This approach ensures a fair comparison, as all models operate under identical conditions, allowing performance differences to reflect each model’s inherent capabilities rather than adjustments in parameter configurations.

To ensure clarity and consistency across tasks, prompts are structured as follows:

• 1.

  Role: The LLM is assigned a role relevant to the task.

• 2.

  Context: A brief context is provided to ground the task requirements.

• 3.

  Task: The LLM receives a clear, action-oriented task.

• 4.

  Output: The expected output format is specified, indicating whether the response should be a single-choice answer, a free-form answer, or identify specific anomalies in the G-code.

We use two evaluation approaches to assess model performance in detecting anomalies from G-code.

In the first approach, models are instructed to select a deterministic label that best represents the type of anomaly present in the G-code. The labels include specific anomaly types, such as SP, OE, UE, or an ND label when no anomalies are detected. For this evaluation, we measure model performance using accuracy, reflecting how often the model correctly identifies the ground truth label for each G-code instance.

In the second approach, LLMs analyze the G-code and assign a probability to each possible label, expressed as a percentage. Here, we focus on the probability assigned to the actual ground truth label as the primary performance measure. This approach offers insights into the model’s confidence in its predictions and its capability to differentiate accurately among various potential anomalies.

In this study, we evaluate the performance of four state-of-the-art LLMs on FDM-specific tasks. The selected models include both closed-source and open-source options, allowing for comparative analysis across architectures, model sizes, and accessibility. Each model undergoes identical tasks to assess its capabilities in FDM anomaly detection and responding to user queries.

The closed-source LLMs include:

• 1.

  GPT-4o: Developed by OpenAI, GPT-4o is an advanced iteration of the Generative Pre-trained Transformer series, released in May 2024. While the exact parameter count remains undisclosed, GPT-4o supports a maximum input context window of 128,000 tokens and can generate up to 16,384 tokens in a single response.

• 2.

  Claude 3.5 Sonnet: Anthropic’s Claude 3.5 Sonnet, released in June 2024, is designed to excel in natural language understanding and generation. While the exact parameter count is undisclosed, it supports a maximum input context window of 200,000 tokens and can produce up to 4,096 tokens in length. Hereafter, any reference to Claude refers specifically to this model.

The open-source LLMs include:

• 1.

  Llama-3.1-70B: Released by Meta in July 2024, this model comprises 70 billion parameters and supports an input context window of 128,000 tokens, allowing for the effective processing of extensive textual data.

• 2.

  Llama-3.1-405B: Introduced by Meta in July 2024, this larger model includes 405 billion parameters. Designed for handling complex tasks with high accuracy, it also supports an input context window of 128,000 tokens.

This section compares the anomaly detection performance of four LLM models through both deterministic labeling and the probabilistic scoring method.

This section investigates the performance of LLMs in answering free-form user queries across beginner, experienced, and theoretical FDM users. Fourteen evaluators score these responses across three key metrics: accuracy, precision, and relevance to user experience level, on a scale from 1 to 5. Figure 4 presents each model’s average performance on these metrics, with standard deviation bars indicating the variability across different expertise levels.

As shown in Figure 4, GPT-4o achieves the highest accuracy among the models, with minimal variation, suggesting consistent performance in accurately addressing questions across all user levels. Following GPT-4o, Llama 405B ranks second in accuracy, with a small margin difference, indicating strong performance. In terms of relevance and precision, Llama 405B outperforms the other models, highlighting its ability to provide responses that are concise and appropriately tailored to each user’s expertise level. In contrast, Llama 70B demonstrates the lowest performance across all three metrics. Notably, Claude exhibits the largest variability in its scores, indicating less consistent responses across different user levels.

These findings suggest that while GPT-4o excels in delivering accurate answers, Llama 405B is more effective in ensuring relevance and precision, albeit with a slight trade-off in accuracy.

This section compares the performance of the LLMs in answering MCQs across three user expertise levels: beginner, experienced, and theoretical. Figure 5 shows the accuracy of each model across these categories.

Overall accuracy, calculated as the average across all question levels, shows that the models perform similarly, with Claude and Llama 405B achieving slightly higher accuracy than the others. However, examining accuracy across user levels reveals clearer differences among the models. Notably, the top-performing model varies by user level. For instance, Llama 70B achieves the highest accuracy for beginner-level questions, while Claude performs best at the experienced level. However, neither ranks highest for theoretical questions.

Additionally, individual model performance shifts significantly across expertise levels. For example, while Llama 70B performs best on beginner questions, it shows the lowest accuracy at the experienced level, highlighting differences in model adaptability across expertise levels. As expected, most models perform best on beginner questions, likely due to their simpler content. Interestingly, overall accuracy is higher for theoretical questions than for experienced questions, possibly reflecting the nature of the training data used for these models.

These findings suggest that LLM effectiveness varies significantly with question complexity and technical demands. This indicates that some models may be better suited to specific user expertise levels.