A generalised novel loss function for computational fluid dynamics

Computational fluid dynamics (CFD) data developed from automotive, aerospace, maritime and medical applications diverges markedly from traditional images. This is due to the fact that CFD studies the movements of fluids around or through objects of interest. CFD techniques then result in the generation of gradient containing fields that offer insights into the physical changes to the velocity, density and pressure of the fluid. These produced fields, both in 2D and 3D forms, do not conform to traditional visual patterns seen in images [1], such as faces [2], text and numbers [3] or more general images [4], upon which the focus of machine learning often sits. The gradients in CFD flow fields are frequently irregular and influenced by various fluid specific effects [5]. This results in CFD data containing critical components that manifest at a small scale within a much larger flow field, where the majority of the field carries minimal information. Effectively adapting and translating generative machine learning architectures for CFD requires a more nuanced approach [6] to isolate and evaluate predictions in these intricate regions. Historically, this has seen the development of highly specific, tailored loss functions [6, 7, 8, 9]. Where, in machine learning, a loss function serves to map a change in performance to a numerical value, often representing a performance metric. It is employed to assess the learned behaviour of a network, architecture, algorithm, or system of algorithms. The resulting value from the loss function is then used for optimization, where the objective is to decrease or increase the value of the function, based on the specific task.

There are two common approaches to loss function development for a deep learning architecture; generalised or specific. A generalised loss function is not tailored to a specific problem in isolation, but assesses overall qualities of a generated or predicted instance, such as the difference between pixels in a generated image and those in a ground truth [10, 11]. Contrary to this is the development of a specific or custom loss function. This is an involved process requiring knowledge of the important aspects of the optimisation being undertaken and the features of the data to focus on [12]. Tailored loss functions are often problem-dependent, relying on a deep understanding of the domain and specific task at hand. This is challenging when applied to the generation of fluid flow fields, as these specific loss functions [9], make comparisons between related works difficult, as the learnt performance of two identical architectures may be different. Additionally, any loss functions utilized for deep learning must be robust enough to measure deviations or convergences concerning a goal, whilst also exhibiting stability in evaluating performance. Loss function instability can significantly impact the effective training of machine learning architectures by not allowing a network to quickly or effectively capture, learn and subsequently represent the underlying data distribution [13]. Selecting a suitable loss function is a non-trivial decision, often requiring tailoring it to the specific nuances of the problem at hand.

This paper proposes, develops and tests a dynamic loss function that identifies, extracts and uses the gradient containing regions of an image to produce higher quality flow field predictions at faster rates throughout the training process by automatically and dynamically assigning different weights to different regions of the field, based on estimating the local velocity gradient. Additionally, a further variation of the loss function is tested where a priori knowledge is used to further accelerate training speed, loss convergence and image quality by varying known optimal loss function parameters during training. The loss function was implemented in a way that allows for successful operation regardless of the dataset in question, where the weighting is determined on a case by case basis without adaptation or modification for any specific features of a given dataset. The approach is based on the commonly used Mean Square Error (MSE) loss [10, 11], and is modified to automatically identify and dynamically produce a weighting relating to the regions of greatest pixel disparity according to their gradient intensities. The motivation for the development is discussed (Section 2), with the method detailed and equations provided in full (Section 3). The experimental design and algorithm (Section 4) are provided, with the results (Section 4) explained, after the proposed method was comprehensively evaluated. Conclusions are drawn in Section 7.

Deep learning architectures, particularly in generative tasks, rely heavily on loss functions to guide the learning process. These functions quantify the discrepancy between generated data and ground truth, enabling the model to adjust its parameters and improve its output. While generic loss functions like Mean Squared Error (MSE), Cross-Entropy (CE), and Binary Cross-Entropy (BCE) are widely used, their inherent limitations necessitate exploring tailored alternatives, especially for complex domains like Computational Fluid Dynamics (CFD). MSE loss, a common choice for image generation [14], operates by calculating the average squared difference between corresponding pixels in the generated image and the ground truth [15]. Its effectiveness stems from its straightforward pixel-by-pixel comparison, assuming equal importance of all pixels [16]. However, this assumption can be problematic in applications where specific regions or features hold greater significance.

CE and BCE loss, on the other hand, are frequently employed in classification tasks involving probability distributions [17]. CE loss measures the difference between the predicted probability distribution and the actual distribution, making it suitable for multi-class scenarios. BCE loss, a special case of CE, is specifically designed for binary classification, evaluating the difference between the predicted probability of the positive class and the true binary label [17]. Both CE and BCE are extensively utilized in training deep learning architectures such as neural networks (NNs), convolutional neural networks (CNNs) and generative adversarial networks (GANs).

GANs have emerged as a powerful tool in fluid dynamics, particularly because of their ability to learn from limited datasets [18] – a crucial advantage given the cost and difficulty of generating large CFD datasets [19]. These networks consist of a generator, which synthesizes new data instances, and a discriminator, which attempts to differentiate between generated and real samples. Loss functions play a critical role in training both networks, providing feedback on the generator’s performance and the discriminator’s ability to distinguish real from fake data [20]. GANs, in their original and conditional (cGAN) forms, have been successfully applied to various fluid flow problems, including laminar vortex shedding prediction [21], airfoil flow field reconstruction [22], and supercritical airfoil analysis [23]. These networks have been trained with a range of loss functions, from basic implementations [24] to highly specialized variants [25]. Despite their successes, generic loss functions like MSE, CE, and BCE loss suffer from a key limitation: they treat all errors equally, irrespective of their relative importance within the specific application [26]. This equal weighting scheme fails to capture the intricacies of complex data distributions and relationships [16], potentially leading to generated instances that exhibit superficial similarity to the ground truth while missing crucial underlying features [27]. While modifications to these generic functions have been proposed to address this issue, they often result in highly specialized solutions tailored to a single problem or dataset [12] [28] [29]. These specialized functions lack generalizability and require substantial re-engineering for adaptation to different datasets, making cross-study comparisons challenging.

Consequently, a pressing need exists for a loss function that is general in nature, but can identify, weight and score the regions of importance in fluid field generation. Such a loss function needs to be implemented such that it does not require complex modification or manual intervention by researchers [30] when applied to different contexts or datasets. To achieve this, a novel adaptation to the MSE loss function is proposed. This modification seeks to automatically and dynamically identify the regions where the field changes quickly. This gradient region is then used to weight the loss between the generated field and a ground truth CFD dataset instance on a field by field basis. The determination of gradient weighting is designed such that it will work on any given velocity, pressure and density field, to ensure broad applicability. The novel loss function is further explained in Section 3.

The most widely applicable loss function for general image reconstruction, the Mean Squared Error (MSE) loss [10, 11], is used to determine a scalar (0D) mean error value between a ground truth instance ($I_{R}$) and a secondary, or generated instance ($I_{G}$) by subtraction and then summation of the error in each direction ($h,w$) over $n$ instances. The MSE loss is denoted by Eq. (1). The proposed novel method differs from Eq. (1) by seeking to dynamically identify the regions of importance in the flow fields where strong gradients of interest are present and then weight these regions accordingly in the loss calculation. Computational fluid dynamics data is well suited to this, as strong gradients hold importance in the evaluation of the underlying data. To achieve this custom loss, the MSE loss, Eq. (1), has been modified to include a 2D dynamic weighting ($W_{i}$) for each generated image ($I_{G}$), where $W_{i}$ is produced from the disparity of the known CFD ground truth instance ($I_{R}$) on an field-by-field basis. The detailed process to determine this weighting is provided in Section 3.3. The modified form of the MSE loss equation is given below, Eq. (2), and it is called Gradient Mean Squared Error loss (GMSE).

The following sections detail the conceptualisation, development and testing of the dynamic weighting function. The assessment of it, the test architecture, implementation and algorithm structure are provided, as are the experimental specifics.

This section provides a detailed overview of the dataset utilised for the experiments and the testing methodology. The evaluation metrics employed to assess the impact of parameter variation on the proposed loss metric are outlined, along with the settings and results derived from the conducted experiments.

Table 1 presents the structural similarity index measure (SSIM) at epochs 1, 5, 20, 100 and 300. The maximum normalised rate of loss is also indicated. The standard deviation ($\sigma_{s.d.}$) is also provided to give an indication of the performance variability with respect to modifying the gamma ($\gamma$), sigma ($\sigma$) or offset ($C_{o}$) parameters of the baseline GMSE loss function. The MSE loss optimised network performance and that of the dynamic gradient mean squared error loss function (DGMSE) are also provided. Indicated in bold is the GMSE baseline for the gamma ($\gamma$), sigma ($\sigma$) and offset ($C_{o}$) variations.

The GMSE and DGMSE loss functions were seen to outperform the MSE loss function in both SSIM and maximum loss rate across all epochs of training. At early epochs, when the SSIM is low, the GMSE (0.068) and DGMSE (0.075) still outperformed MSE optimisation (0.032). The disparity between the approaches widened further through training, where the GMSE and DGMSE loss functions proved beneficial, reporting final SSIM’s of 0.988 and 0.989 respectively, compared to 0.933 for the MSE loss function. Marking a reduction in structural dissimilarity of 82.1% and 83.6% in comparison to the MSE loss. An increased rate of maximum loss was also observed for the GMSE and DGMSE variants of -0.143 (33.6% higher) and -0.189 (76.6% higher) respectively, compared to -0.107 for the MSE loss function. This stronger rate of normalised loss convergence indicates that the network can better optimise in response to the initial loss between the generated instances and the real dataset instances.

Variations to the gamma ($\gamma$) variable were seen to influence both the SSIM and the rate of loss. The standard deviation of gamma variation was observed to be larger (0.082, 0.108 and 0.146) during early training (1, 5 and 20 epochs) before reducing to smaller values (0.033, 0.018) towards the completion of training (100 and 300 epochs). This correlates to the close SSIM observed at later training stages, with most structural similarities being clustered around 0.983 to 0.989, with a maximum recorded SSIM of 0.991. The maximum SSIM outperformed the MSE baseline by 86.6%. There is more variance reported in the rate of loss recorded for the gamma variations, with a standard deviation of 0.017, a maximum of -0.177 ($\gamma=0.60$) and a minimum of -0.115 ($\gamma=2.00$). $\gamma>1.20$ are seen to correspond to reduced rates of loss, and variations where $\gamma<1.20$ correspond to an increased rate of loss.

Variations to sigma ($\sigma$) were also observed to effect the SSIM and the rate of loss, with different performance depending on the strength of the blur. Higher blur strengths ($20<\sigma<40$) were seen to correspond the highest SSIM at epochs 1 (0.285), 5 (0.654), 20 (0.926) and 100 (0.982). Blur strengths between 10 and 30 also corresponded to the highest rates of loss (-0.143, -0.141 and -0.133), outperforming the MSE baseline by 33.6%, 31.7% and 24.3% respectively. Again, the variation of SSIM was seen to decrease as training progressed, with a standard deviation of 0.022 reported at the completion of training.

Variations to the offset ($C_{o}$) had the largest SSIM deviation at the completion of training (0.028) indicating that the offset weighting can still affect the image quality further into the training cycle. This is reflected by the higher SSIM values recorded for offsets of 0.1, 0.2 and 0.3, which highly weight the gradient regions. These offset values corresponded to the highest SSIM values recorded for the offsets (0.986, 0.988, 0.976 respectively). The opposite is true of $C_{o}>0.3$ which were observed to record SSIMs between 0.923 and 0.926, thus under performing the MSE loss by 10.5% to 14.9%. It should be noted that all variations to the baseline and offset (($C_{o}$) still reported higher rates of loss (-0.110 to -0.158) than the MSE baseline by a minimum of 2.8% to a maximum of 47.7%.

Three plots detail the generator loss during training. The Adam optimiser has been used to reduce this loss during training for each individual test. The maximum loss values have been normalised to 1 for all data shown. Fig. 5 provides the gamma ($\gamma$) generator loss, Fig. 6 provides the sigma ($\sigma$) generator loss and Fig. 7 provides the offset loss ($C_{o}$) generator loss. The x axis details the epochs of training, and the y axis provides the normalised loss of each test. Indicated in red is the MSE baseline, black details the GMSE baseline and blue the DGMSE loss. Three additional figures are provided, covering the discriminator loss during training. A separate CNN network was used to assess both the real and generated images during training. The goal of the generator was to produce images that would be awarded a value of 1, indicating the generated fake image belonged to the real dataset when scored by the discriminator. Fig. 8 provides the gamma variant discriminator performance, Fig. 9 provides the sigma variant and Fig. 10 the offset variant performance. The y axis shows the score from the discriminator where 1 is the target and the x axis denotes the epoch of training. Again, indicated in red is the MSE baseline, black is the GMSE baseline and blue the DGMSE loss.

The generator loss of the gamma variations provided in Fig. 5 were seen to converge faster than the MSE loss optimised network. Both the GMSE and DGMSE loss functions provided beneficial loss characteristics in comparison to the gamma variants themselves, which were seen to converge at different rates, but to a slower extent than the GMSE baseline (black). Initial rates of loss in the gamma 0.20, 0.40 and 0.60 were seen to outperform the GMSE baseline to a limited extent, but with ultimately lower maximum rates, as reflected by Table 1. Gamma variations of 0.20 to 0.80 were seen to converge to a higher final loss than $\gamma>=1.00$ (Fig. 5), indicating that the loss function may benefit from dynamic changes to $\gamma$ during training. The dynamic variant, DGMSE, was seen to outperform all gamma variants and both the MSE and GMSE baselines and resulted in the lowest final loss. The ability of the gamma variants to fool a discriminator network, Fig. 8, is seen to have limited variation and relatively robust performance compared to the sigma discriminator loss (Fig. 9) and the offset discriminator loss (Fig. 10). All gamma variants are observed to fool the discriminator network, at earlier epochs, compared to the MSE baseline. $\gamma>1.00$ variations are seen to have better performance at early epochs ($<50$), but worse performance later in training ($>50$). The GMSE baseline, with no $\gamma$ shift was observed to perform robustly, with moderate performance in early training but a better ability to overcome the discriminator at later stages. The DGMSE loss was observed to perform well early in training and until completion, where it recorded the highest final loss. The second highest final loss was the GMSE loss function, followed by the $0.20<\gamma=<1.00$ variants, then the MSE loss and finally the $1.20<\gamma=<2.00$ variants. Thus, further supporting the adoption of a dynamic approach (DGMSE) or $\gamma=<1.00$.

Sigma was seen to also effect the generator loss during training (Fig. 6), but to differing extents depending on the strength of the blur. In the initial stages of training ($<50$ epochs), strong blurs ($\sigma>10$) corresponded to better loss rates. As training progressed however ($>50$ epochs) smaller blurs became more performant and resulted in lower final loss values at the completion of training. The MSE baseline was found to outperform the sigma 2 and 5 variants, only at the completion of training. Both the GMSE and DGMSE outperformed the baselines and all sigma variants, with the DGMSE displaying beneficial initial loss, where it outperformed all variants with the highest recorded normalised loss rate (-0.189). The ability of the sigma variants to fool the discriminator (Fig. 9) is also dependent on blur strength. Low strength blurs ($\sigma=2$) are observed to perform similarly to the MSE baseline, whilst stronger blurs offer a discriminator loss advantage with markedly better early perform loss (epochs $<100$), after which the strength of the blur is less impactful and closer loss performance is observed. Again, the DGMSE is the most performant, followed by the sigma variants in descending order of strength (40 to 20), then the GMSE variant, sigma 5 variant, the MSE baseline and finally the sigma 2 variant.

The offset weighting of the gradient free region was also seen to have an effect on generator loss convergence, as provided in Fig. 7. Lower values indicate where the gradient region is highly important and higher values prioritise the gradient free region. Initial convergence of all offset variants outperformed the MSE baseline, with this continuing for the entirety of training, with $C_{o}=0.1$ a notable exception to this, as it resulted in higher loss from 130 epochs onwards. The $C_{o}=0.2$ variant was also found to result in higher loss than the MSE baseline at the completion of training. These findings indicate that both the gradient and gradient free regions retain importance during training and the weighting should account for both. The higher performance of the GMSE and DGMSE baselines indicate the benefits of either a balanced or dynamic approach. For the offset discriminator loss performance (Fig. 10), offsets equal to or above 0.5 are seen to perform close to the MSE baseline in early training between 0 and 100 epochs, before converging to lower loss values (indicating decreased performance). Conversely, offsets less than 0.5, where the gradient regions are weighted heavily, correspond to better initial training (0 to 100 epochs) as well as better final convergence. The most beneficial final loss is achieved by the DGMSE variant, followed by $C_{o}=0.1$, then the GMSE baseline (with $C_{o}=0.1$), then $0.3<C_{o}<0.5$, the MSE baseline and then $0.6<C_{o}<0.9$.

Qualitative field assessment also offers insight into the behaviour of the baseline MSE and GMSE loss optimised networks. Fig. 11 shows the generation abilities of the two baseline networks as training progressed. Both networks have similar performance after a single epoch, before deviation at the displayed 5, 10, 20, 50 and 300 epochs. It was seen that a circular spatial reconstruction error was present where the payload bay of the submarine was located in the fluid flows of [39]. This error does not appear in the GMSE loss optimised network, where these region had a higher weighting for accurate generation. The strong gradient regions around the submarine developed earlier in the training process for the GMSE loss optimised network and were refined at later stages of training.

The results of these tests demonstrate the potential of gradient capture driven GMSE and DGMSE loss functions for accelerating training speeds of generative architectures and ensuring higher quality generations for fluidic applications. The dynamic extraction and weighting of regions of importance was demonstrated to be a robust means of accelerating loss convergence at both the generator and discriminator network of a cGAN architecture, resulting in several advantages when compared to MSE loss. These advantages are evident in several aspects. The first is the structural similarity index measure (SSIM) performance which was observed to consistently surpass that of the MSE baseline across all variants of the GMSE loss function. The higher recorded SSIM indicates a more accurate reconstruction of the complex gradient structures of the flow field, as evidenced by both Fig. 11 and the data of Table 1. The 82.1% and 83.6% reduction in relative SSIM error achieved by the GMSE and DGMSE loss function highly the potential of leveraging gradient information to improve generation fidelity.

Notably, the DGMSE and GMSE variants resulted in a faster rate of convergence during training, evidenced by the higher maximum loss rates in comparison to the MSE baseline. This accelerated learning can be attributed to the GMSE function’s ability to provide more informative gradients to the generator, allowing for faster and more effective optimisation. This finding carries significant practical implications, as faster training directly translates to reduced computational cost and faster development cycles for deep learning. Furthermore, the DGMSE and GMSE loss functions success in consistently and rapidly fooling the discriminator network highlights its capacity to guide the generator towards producing more realistic and challenging flow fields for the discriminator to score. This superior performance, particularly at early epochs (Fig. 8, Fig. 9, Fig. 10) suggests that the GMSE loss facilities a more efficient exploration of the data and results in a more effective approximation of the underlying data distribution of the RANS dataset.

Exploration of the GMSE function’s parameters reveals their influence on its efficacy. The blur strength, controlled by $\sigma$, highlights a crucial trade-off. Weaker blurs ($\sigma<10$) retain precise gradient localization but risk overfitting to small-scale features, while stronger blurs ($\sigma>20$) provide a more generalized weighting, potentially diluting the focus on crucial regions, but are still seen to outperform MSE loss. The optimal blur strength is likely dataset-dependent, but DGMSE substantiates the benefits of starting with a strong blur and decreasing the strength during training. This technique is seen to result in beneficial early and late stage SSIM, as well as a higher maximum loss rate (Fig. 6).

The gamma parameter, $\gamma$, governs the contrast enhancement of the weighting mask. While its overall impact on SSIM appears limited within the tested range, it significantly affects the convergence behaviour (Table 1). Higher gamma values ($\gamma>1$), emphasizing high-gradient regions, lead to faster initial convergence but potentially at the cost of a higher final loss. This suggests a dynamic gamma adjustment strategy could further optimize training by initially prioritizing rapid error reduction and later shifting focus to fine-grained details.

The offset parameter, $C_{o}$, dictates the relative weighting of gradient-rich versus gradient-free regions. Assigning high weights to gradient-free regions ($C_{o}>=0.4$) consistently degraded performance, highlighting the importance of prioritizing areas containing crucial flow structures. Conversely, excessively low offset values ($C_{o}=0.1$) might overshadow the contribution of the surrounding context. The optimal offset likely depends on the specific dataset and the relative significance of high-gradient features versus their surroundings.

The DGMSE and GMSE loss functions are also seen to achieve these advantages without requiring domain specific knowledge or complex modifications to the training data, to account for highly customised loss functions [44]. While these customised loss functions for CFD data do exist, they introduce significant complexity [29] and can lead to unstable or incomparable training [29] – limiting their generalizability. The DGMSE and GMSE loss function, in contrast, relies on the simpler method of prioritizing the regions of high gradient disparity – making it easily adaptable to other datasets exhibiting similar characteristics – such as temperature fields [45], hydrodynamic [46] and aerodynamic data [47] and fields with strong edge structures [1].

Generational architectures are susceptible to a range of different failure modes or issues that render training challenging. Most often discussed are mode collapse and convergence failure [48]. Mode collapse occurs where the generator struggles to produce a variety of data and produces one output for a larger range of inputs [49]. This is prevalent when working with small datasets and model sizes, where either the model or dataset is not sufficiently large, and struggles to approximate the underlying data distribution. This often leads to generative errors, in the case of GAN architectures. In a limited capacity, this can be seen in Fig. 11 where the centre of the submarine in the MSE images (top) can be seen to feature a consistently low field generation, despite this not being the desired output. This can be contrast against the more correct GMSE generation (bottom). The effective prevention of mode collapse is challenging but may be alleviated by the function of the GMSE loss function. Giving an increased weighting to the images regions of importance in the GMSE loss function results in the generator being proportionally punished for generating poor outputs, in areas such as the submarine itself, leading to a more robust generative ability. Using the natural weighting behaviour of the loss function to combat mode collapse offers an easier pathway to prevention than the implementation of other, more complex methods of prevention, like those explored in literature [50]. This is because the generator is incentivised to produce higher quality images by way of the weighted loss term.

This research introduces the GMSE loss function as a powerful and versatile tool for enhancing the training of generative deep learning architectures, with a focus on fluid fields and gradient-drive data. The ability of both the DGMSE and GMSE loss function to dynamically prioritise gradient containing regions, whilst remaining conceptually simple and adaptable underscores its potential for broader applicability. Future research should investigate the performance of the GMSE loss function with other gradient-containing datasets, explore its application to higher dimension data (such as 3D volumes) and delve into its potential for mitigating mode collapse in generative models. This work represents a significant step towards efficient, effective and generally applicable training techniques for generative architectures across a range of fields.

Zachary Cooper-Baldock: Investigation, Conceptualization, Methodology, Formal analysis, Writing - original draft, Writing - review and editing. Paulo E. Santos, Russell S.A. Brinkworth, Karl Sammut: Supervision, Conceptualization, Methodology, Writing - review and editing.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data will be made available upon reasonable request.