GenAI for Simulation Model in Model-Based Systems Engineering

DEVS represents a formal modeling specification used to describe discrete event systems. DEVS models typically comprise the following core components:

Atomic Model defines the fundamental constituent units of a system, capturing its behavior and state. The atomic model in DEVS is defined by a seven-tuple constructor:

The elements of (1) are defined as follows: 1) State set ($S$): Describe all possible states of the model. 2) Input Event Set ($X$): Define the types of external events the model can receive. 3) Output Event Set ($Y$): Define the types of external events the model can generate. 4) Internal Transition Function ($\delta_{int}$): Define the rules governing the transition of a model from one state to another, independent of external events. 5) External Transition Function ($\delta_{ext}$): Defines the procedure for updating the model’s state upon receiving an external event. 6) Output Function ($\lambda$): Defines the output behavior triggered by the model in a particular state. 7) Time Advance Function ($ta$): Defines the duration for which the model stays in its current state before a state transition occurs.

Coupled Model integrates multiple atomic models or other coupled models to create more complex system structures. The coupled model in DEVS is defined by a four-tuple constructor:

The elements of (2) are defined as follows: 1) Set of Components ($D$): Include multiple atomic models or other coupled models. 2) External Input Coupling ($EIC$): Defines how external input events are passed to the submodel. 3) External Output Coupling ($EOC$): Define how the output events of the submodel are passed to the outside of the system. 4) Internal Coupling ($IC$): Define event transfer relationships between sub-models or between sub-models and external components.

The operational mechanism of the DEVS model is based on the event-driven principle and progresses in discrete time steps. Although the core design of the DEVS architecture primarily focuses on modeling discrete events, extensions of DEVS and some of its derivative frameworks support continuous-discrete hybrid systems by introducing mechanisms for “continuous state change” or by combining continuous state variables with discrete event processing [24]. Hybrid DEVS and similar extensions provide modeling capabilities for continuous-discrete hybrid systems, but current DEVS tools (e.g., CD++ and DEVSJAVA) offer limited support for hybrid modeling. Most of these tools remain focused on traditional discrete-event systems, and the hybrid modeling functionality is neither fully optimized nor standardized [25]. Some tools may necessitate users to develop additional modules for hybrid modeling or perform extensive custom configurations, thereby increasing development costs.

The structure of the X language is illustrated in Fig. 1. X language is a modeling and simulation language developed based on DEVS. It complements the description of continuous port connectivity and integrates both continuous and discrete event port connectivity to model and simulate the interaction between continuous and discrete behavior-dominated hybrid models. X language supports the integrated modeling of system architecture and physical characteristics, thereby enabling comprehensive modeling and simulation analysis in system designs [26, 27]. Additionally, X language supports dual-mode modeling (graphical and textual), with graphical representation enhancing interactivity, whereas textualization enables the integration of GenAI within X language.

The core classes of X language are presented in Table. II. This section provides a detailed description of the functions of these classes.

Couple class: Couple class is a crucial component in X language, facilitating the integration of simulation analysis modeling and system architecture modeling. It corresponds to the coupled model in the DEVS architecture. The keyword Part in the attribute component and the keyword Connection in the connection component are specific to the couple class and used to describe the model’s composition and the connection relationship, respectively, and together constitute the architecture of the couple class model.

Discrete class: Discrete classes facilitate the modeling of discrete-event systems. The discrete class is indivisible, serving as the fundamental simulation unit within a discrete model. The keywords State and Transform in the state component are specific to the discrete class and used to represent transfer scenarios and conditions between different states in discrete events.

Continuous class: Continuous class is established to accomplish the modeling of continuous systems. The continuous class facilitates the construction of non-causal models, primarily based on equation solving. The keyword Equation in the equation component is specific to the continuous class and used to describe the behavior of the continuous class.

The continuous and discrete classes are atomic models in X language, corresponding to the atomic models in DEVS. The mapping between X language couple class, discrete class, and continuous class models and the corresponding elements in DEVS is illustrated in Fig. 2. X language couple class and atomic class models encompass all elements of the coupled and atomic models in DEVS, with some extensions beyond these elements.

Complex product simulation models are characterized by high complexity and multi-scale. For such model code generation scenarios, generating accurate and consistent simulation code directly through GenAI is challenging. In this paper, the simulation model is constructed using scalable templates based on X language to address the issues above. X language scalable template deconstructs the syntax while preserving key elements and the overall model structure, thereby allowing model length and content to be dynamically adjusted based on the number and attributes of the included elements.

The core idea behind employing scalable templates for constructing simulation models is to modify the approach to code generation tasks. This study generates the simulation model through a modular code completion task. Compared to generating a complete simulation model, modular code completion can effectively address issues such as performance degradation, context loss, and output truncation caused by excessively long input contexts [29]. However, modular code completion can reduce consistency from module to module and system to subsystem. Therefore, when generating simulation models using scalable templates, additional algorithms are required to enhance the consistency of the models, which will be explained in detail in Section III.

There are several rules to follow when building a scalable template: 1) The top-down design principle should be followed when constructing the template. Specifically, model templates at the system level should be constructed prior to model templates at the component level. 2) When constructing the template, the most effective description of the system-level structure of the simulation model and its contained elements should be extracted to minimize the amount of text generated by the LLM and improve accuracy. 3) The coupling relationships between modules must be clarified when constructing the template. The coupling relationship of modules should be considered when using GenAI to generate simulation models.

Based on the characteristics of X language and the modeling specifications of the scalable templates discussed above, this paper develops scalable templates for X language couple class model, discrete class model, and continuous class model, as follows. A detailed description of the usage of these templates will be provided in Section III.

This section elaborates on the three steps of the Simulation Model Generation Method Based on Scalable Templates and Transformer-based Models.

Step 1: Identify the relevant corpus for target simulation models. Product design document refers to the natural language documentation that engineers use as a reference when designing and modeling a system. However, such documents may contain irrelevant information, such as background knowledge or excessive details, which are unnecessary for constructing a simulation model. Therefore, extracting the relevant information from these documents for model construction is essential. We use BERT fine-tuned for Named Entity Recognition (NER) and single-sentence classification tasks to process the documents. If a sentence is not labeled in either task, it is considered redundant and excluded from the corpus. The remaining sentences are then added to the simulation model corpus. Subsequently, the product design documents are manually reviewed, and the corpus is adjusted as necessary.

Step2: Couple class model generation. The couple class model must be constructed before the atomic class model in accordance with the principle of forward design through top-down model generation. Fine-tuned BERT for NER tags tokens, allowing us to identify system relationships within the simulation model corpus. This process ultimately enables the construction of the complete system composition by aggregating system relationships. The fine-tuned BERT model for single-sentence classification is capable of categorizing individual sentences. In Step 2, we employ it to determine whether the sentences in the model corpus describe the system’s connectivity relationships and add them to the connection corpus. Then, Tramsformer-based model inference is applied based on the connection corpus and the system composition to derive the connection relationships between subsystems. After defining the system composition and system connectivity relationships, the couple class model is constructed by populating the system composition into the keyword import of the couple class template and the system connection relations into the keyword connection.

Step3: Atomic class model generation. The name of the atomic class model corresponds to the name of the subsystem, and the keyword Port of the atomic class model is derived by analyzing the keyword Connection of the couple class model. Additionally, the valuetype of the port is reasoned through the port names and the model description document. Due to the complexity and variable format of the keyword State of the discrete class model, the keywords Value and State are coupled and generated by the fine-tuned Transformer-based model. The fine-tuning method for this model is described in detail in Section III-C. The generation of the keyword State of the discrete class model should employ the Few-shot [30] method to construct the prompts, as illustrated in Table. III. Similar to the discrete class, the value and equation parts of the continuous class, which are coupled, are generated uniformly by Transformer-based models. After completing the construction of the state and equation parts, it is necessary to verify these components for any undefined functions. If there is an undefined function, the function class model of X language needs to be generated according to the function name and the code of the atomic class model. The function class supports the procedural programming paradigm so that the function class simulation code can be generated using GenAI for code scenarios.

This section details the training method for the NER task fine-tuned BERT model used in Step 2 of Section III-A. The application of BERT [31] is a pre-trained deep learning model that uses a bidirectional Transformer encoder to understand words in context and generate word vector representations with rich semantic information. One of the notable advantages of the BERT model lies in its high flexibility, allowing it to effectively adapt to various downstream tasks through a range of fine-tuning strategies [32]. NER-BERT presented in this section is employed to analyze the simulation model corpus in the early stages of model construction, identifying module containment relationships, extracting relevant information, and omitting irrelevant content. The extracted containment relationship data informs the construction of both the system model and its corresponding subsystems.

The training method of NER-BERT is illustrated in Fig. 4. When providing BERT with an input dataset, the constructed data source must meet the following conditions: 1) It is the whole or part of the actual industrial model construction documentation; 2) It contains a sufficient number of parent and subsystem containment relationships; 3) It has substantial data without inclusion relationships as the data in question.

Upon completing the dataset preparation, fine-tuning training of the model can commence, enabling the BERT to perform the NER tasks outlined above. The input to BERT consists of the original word vectors for each word in the text, whereas the output comprises the vectors for each word or phrase after integrating the full semantic information of the text. As shown in the NER-BERT training section of Fig. 4, the fine-tuning process for the NER task connects a fully connected layer and a softmax layer to the original BERT output section. This addition enables the model to predict the tokens’ tags based on these vectors.

After training BERT for single-sentence classification scenarios in a similar manner, the system’s composition and connection relations can be extracted using both NER-BERT and the fine-tuned BERT for single-sentence classification. These relations can then be used to construct X language couple class model.

After constructing the couple class model, the next step is to generate the atomic class model, utilizing both the couple class model and the component-level model corpus. This section describes the training method for Transformer-based models, which is employed in Step 3 of Section III-A for the code generation of the behavioral component of the atomic class model. The models trained in this section are primarily employed to generate the behavioral simulation codes within the atomic class models of X language, specifically the keyword State in discrete class models and the keyword Equation in continuous class models.

Mainstream code evaluation methods typically evaluate code based on metrics such as syntactic and semantic consistency (e.g., CodeBLEU [36]) and functional correctness (e.g., Pass@k [37]). However, simulation modeling of complex products introduces unique requirements and challenges for evaluation. In comparison to other code types, simulation model codes are characterized by their modular, hierarchical, and parametric structure. These characteristics cause the evaluation results of the above methods for evaluating simulation model codes to deviate from their actual quality. In contrast, the manual evaluation of simulation models, although capable of assigning reasonable scores based on the characteristics of the simulation model, ultimately depends on the scorer’s judgment in the absence of precise evaluation criteria. Therefore, this section introduces code evaluation metrics specific to X language simulation models tailored to their unique characteristics. These additions enhance the relevance of X language simulation model evaluations and provide specific metrics that guide manual evaluation. The evaluation metrics in this paper primarily evaluate X language simulation model based on Degree of Error and Model Consistency [38]. These two metrics are described in detail below.

Degree of error: In product design scenarios, Simulation models with a low degree of error tend to require less effort during subsequent modifications. Thus, in addition to evaluating the accuracy of the generated model, the Degree of Error index of any erroneous model also holds a reference value. The Degree of Error differs from the n-gram [39] approach. It evaluates the actual impact of errors on the model and assesses the ease of implementing modifications, and it often requires manual evaluation combined with a code checker and compiler.

Model consistency: Model consistency refers to the requirement that the representation of the same element in both the upper and lower models must remain consistent. The primary metrics of model consistency examined in this paper include: 1) Name consistency between the attribute part of the system model and the header part of the subsystem 2) Port consistency between the connection part of the system model and the definition part of the subsystem 3) Consistency between the functions invoked in the atomic class model and the definitions provided in the function class model

Based on the above two metrics, the simulation model evaluation method proposed in this paper is outlined as follows:

A complete set of simulation models includes a top-level model and its subsystems, with each subsystem potentially containing additional subsystems. The parent model is always a couple class model, whereas sub-models may be an atomic class model. The score of the parent model is derived from the simulation correctness of itself and its sub-models, specifically:

In (6), $A_{parent}$ signifies the simulation correctness of the parent model (couple class), $C_{subsystem,i}$ denotes the weight of the i-th subsystem model (couple class or atomic class), and $A_{subsystem,i}$ indicates the simulation correctness of the i-th subsystem model. For a set of simulation models undergoing evaluation, the final score is denoted as $Score_{top}$, representing the score of the top-level model.

In this paper, the simulation correctness $A_{i}$ is calculated as (9). X language simulation model is modular, allowing us to connect the ports of the model being tested to the correctly defined subsystem simulation models for test simulation. If the outputs of the model being tested are correct during the simulation, the model is considered fully correct.

In (9), $\varepsilon$ represents penalty coefficient, a constant less than 1. $P_{i}$ denotes the correctness similarity of the model. $P_{i}$ is less than 1, with values closer to 1 indicating greater proximity to the correct model. The correctness similarity $P_{ci}$ of the couple class model is calculated as (10).

$k_{h},k_{a}$ and $k_{c}$ denote the weights assigned to the correctness similarity for the attribute part and the connection part, respectively.$P_{header,i},P_{attribute,i}$ and $P_{connection,i}$ denote the correctness similarity for the header part, the attribute part and connection part, respectively. $P_{attribute,i}$ and $P_{connection,i}$ are calculated by comparing the product design document with the attribute part and the connection part of the couple class model:

$P_{port,i}$ denotes the correctness similarity for the keyword Port in the attribute part. $F1$ indicates F1 score in machine learning of keywords Part and Connection. The correctness similarity $P_{i}$ of the atomic class model is calculated as (13).

$k_{h}$, $k_{d}$, $k_{s}$, and $k_{e}$ denote the weights assigned to the correctness similarity for the header, definition, state (for discrete class) and equation parts (for continuous class), respectively. $P_{header,i}$, $P_{definition,i}$, $P_{state,i}$, and $P_{equation,i})$ denote the correctness similarity for the header part, definition part, state part (for discrete class), and equation part (for continuous class), respectively. $I(d)$ and $I(c)$ represent indicator functions for the conditions “model is a discrete class model” and “model is a continuous class model” respectively, where the value is 1 when the condition is satisfied and 0 when it is not. $P_{header,i}$, $P_{port,i}$, $P_{definition,i}$, $P_{state,i}$, and $P_{equation,i}$ are calculated as (14) and (15).

$\varepsilon_{h}$ and $\varepsilon_{d}$ represent penalty coefficient. $CE$ and $IE$ denote elements that are consistent and inconsistent with other models, respectively. ${\alpha_{i}}$ and ${\beta_{i}}$ denote the attenuation of “correctness similarity” in the i-th model due to syntax errors and simulation logic errors, respectively, a constant less than 1. $m$ and $n$ denote the numbers of syntax errors and simulation logic errors in the model, respectively. Syntax detectors and compilers can typically detect syntax errors, whereas simulation logic errors must be identified through manual inspection. Consequently, from the perspective of modification effort, simulation logic errors are often more severe, necessitating that ${\beta_{i}}$ be smaller than ${\alpha_{i}}$. Considering that the Degree of Error in the state section is related to its length, the formulas for ${\alpha_{i}}$ and ${\beta_{i}}$ are provided to ensure that $P_{state,i}$ is not influenced by the length of the keyword State:

${\operatorname{len}_{\mathrm{c}}}$ and ${\operatorname{len}_{\mathrm{i}}}$ denote the standard keyword State length and the keyword State length of the i-th model, respectively. $\alpha_{c}$ and $\beta_{c}$ represent the standard attenuation coefficients. For the above evaluation metrics, $P_{attribute,i}$, $P_{header,i}$, and $P_{definition,i}$ focus on the consistency of the model, whereas $P_{state,i}$ and $P_{equation,i}$ emphasize the Degree of Error in keyword State (for discrete class) and equation part (for continuous class) of the model.

The penalty coefficients in the above equation are determined based on the actual importance of each component of the model. The weights assigned to each model and to each component within the model are calculated using EWM. EWM is a multi-criteria decision-making approach that leverages the concept of information entropy to determine the weight of each criterion in an evaluation. This method assigns weights based on the variability of the data associated with each criterion; criteria with higher variability are assigned greater weights as they contribute more informative value to the evaluation [40].

This set of evaluation metrics considers the modularity of the simulation model and the characteristics of the design and development process. It will be employed in Section IV to evaluate the quality of the generated simulation model.

The preparation for the experiment will be presented in terms of case introduction, data preparation, and experimental equipment.

Case introduction: The aircraft electrical system examines the variations in electrical current, voltage, and power utilized during the aircraft’s actual flight. The aircraft electrical system comprises six components: power supply, flight scenario control module, control bus, radar, rudder, and thrust module. The relationship between the subsystems is illustrated in Fig. 6. The functions of these six subsystems are presented in Table. V.

Data preparation: In this paper, we select relevant papers on the aircraft electrical system [41, 42] and the model-building instructions document compiled by our team for constructing the aircraft’s electrical system as the data preparation for this experiment.

Experimental equipment: For the hardware configuration, the experimental computing platform includes an A100 80GB GPU, an i9-13900K processor, and 256GB of RAM. The software environment comprises Ubuntu 20.04 LTS as the operating system, PyTorch 2.4 as the deep learning framework, and CUDA 12.4 with cuDNN 9.1 for GPU acceleration.

This section utilizes the NER-BERT model fine-tuned in Section III-B to extract the system composition of the aircraft electrical system. The process is illustrated in Fig. 7.

The tagging results of NER-BERT are compared with manual annotations, and if the tags for each token in a sentence are correct, NER-BERT is deemed to have correctly tagged the entire sentence. Ultimately, the accuracy of NER-BERT is 81.3%. Although this accuracy rate is relatively modest, we observe that the causes of incorrect token tags in NER-BERT can generally be categorized into two main groups: 1) Errors in individual token tags within longer word sequences, i.e., incorrect boundaries of the tags. 2) Misclassification of irrelevant tokens, which do not belong to any entity, as entity tags. Specifically, the precision of entity tags for a given token is 95.06%, whereas recall achieves 100%. In addition to the aircraft electrical system, we also performed experiments on the electric vehicle system, railroad crossing system, and aircraft take-off system. The results of NER-BERT’s entity tagging for these product design documents are presented in Table. VI. In the experiments, the recall rates for entity tags reached 100%. This indicates that no system composition information is omitted, allowing engineers to adjust only the NER-BERT tagged content to derive the system composition, thus eliminating the need to read the entire product design document.

The atomic model generation method is presented in Step 3 of the Simulation Model Generation Method Based on Scalable Templates and Transformer-based Models in Section III-A. Take the subsystem AutoPilot as an example. The AutoPilot subsystem performs the functions of the rudder module. Based on the connection part of the couple class model, AutoPilot can retrieve the relevant port connection relationships and populate the keyword Port of AutoPilot with this information. The prompts are formulated based on the structure outlined in Table. III and serve as inputs to the Transformer-based models trained in Section III-C to generate the behavioral code of the atomic class (state for discrete class, equation for continuous class) and its corresponding keyword Value. The comparison of generated model and manual written model is presented in Fig. 8. Although there are differences in expression between the generated and manually written models, the generated model maintains the same semantics and adheres to X language syntax specifications.

After completing the generation of all atomic models, both the generated couple class models and atomic models are imported into X language development tool, XLab. XLab can transform the generated text code into graphical models. In the aircraft electrical system model, each subsystem’s graphical or textual composition, definitions, connectivity relationships, and internal logic are shown in Fig. 9. The mapping relationship between the final model and the various modules of the aircraft electrical system is outlined as follows: power supply - Battery, flight scenario control module - Control, control bus - BallisticSceneControl, radar - Radar, rudder - AutoPilot, thrust - Thrust. So far, we have successfully generated a complete simulation model of the aircraft electrical system using the Simulation Model Generation Method Based on Scalable Templates and Transformer-based Models proposed in this paper.

This section evaluates the impact of the aforementioned model generation methods on generating simulation models using various Transformer-based models based on the evaluation metrics outlined in Section III-D.

We selected three open-source models for the code generation scenario (CodeQwen1.5-7b, CodeGemma-7b [43], and DeepSeek-Coder-6.7b [44]) and evaluated the quality of the aircraft electrical system models generated with and without the use of the simulation model generation method proposed in this paper. Additionally, we selected two high-performance proprietary models (Claude-3.5-Sonnet, GPT-4o) and evaluated the aircraft electrical system models they generated directly. For the aircraft electrical system, set the values of $\varepsilon$, $\varepsilon_{header}$, $\varepsilon_{port}$, and $\varepsilon_{definition}$ in the evaluation metrics to 0.8, 0.6, 0.6, and 0.6, respectively. Set the values of $\alpha_{c},\beta_{c}$, and $len_{c}$ to 0.2, 0.1, and 1 state, respectively. We generated 20 sets of aircraft electrical system models using each model, both with and without the proposed method, and then manually evaluated them based on the evaluation metrics described above. We obtained several atomic class models and one couple class model using each method. We calculated the scores of atomic models as 10 and calculated their final scores as (6). We employed EWM to calculate the weights of the models and components, accentuating the gap in the final scores. The average scores of the 20 sets from different models and methods were compared, and the scores of atomic models and the final scores are presented in Table. VII. The meanings of Meth.a and Meth.b in Table. VII are as follows: Meth.a) The simulation model generation method proposed in this paper. Meth.b) Generate the simulation model directly using Transformer-based models. The prompts include X language BNF, X language model examples, a description of the model to be generated, and a description of the parent system model.

As shown in Table. VII, the simulation model generation method proposed in this paper improves the quality of the simulation models generated by mainstream Transformer-based models, such as CodeGemma, CodeQwen, and DeepSeek Coder. The quality of the generated simulation model surpasses that of directly using Claude 3.5, Sonnic, and GPT-4. This suggests a practical direction for generating simulation models focused on the physical properties of systems.

Additionally, we observed a phenomenon during our experiments: Transformer-based models perform poorly on the simple model Radar, which we believe is due to its a priori knowledge of radar. Instead of assisting in generating the simulation model, this prior knowledge hinders the process, which contradicts our expectations. This is most likely since the a priori knowledge related to radar in the Transformer-based models differs from the description of radar in the model design document. The Transformer-based models utilize their a priori knowledge when generating the simulation model, leading to a deviation between the final simulation model and the model design document. After renaming the Radar module, the quality of the simulation code for this module improved. This suggests that Transformer-based models’ generalization ability may hinder the generation of high-quality simulation models when the simulation model is well documented. Determining how to help Transformer-based models better balance the application of prior knowledge with task requirements when generating content based on existing knowledge will be one of the future research directions for the application of GenAI in MBSE.