Generating Synthetic Net Load Data with Physics-informed Diffusion Model

In this work, we build on top of the denoising diffusion probabilistic model (DDPM), which achieves state of the art results in fields such as image generation [35] and speech synthesis [36]. Aimed at producing higher quality time series data, our implementation of the DDPM primarily draws inspiration from the speech synthesis model described in [36]. The key modification is the utilization of the $L_{2}$ norm as loss function, which is more suitable for net load profile generation based on numerical study results.

Diffusion in the context of statistics refers to transforming a complex data distribution $p_{\text{data}}$ to a simple prior distribution $p_{\text{prior}}$ on the same domain. In the DDPM, this process is called the forward process. Conversely, the reverse process transforms $p_{\text{prior}}$ to $p_{\text{data}}$. In practice, we let $p_{\text{prior}}$ be a Gaussian distribution. Both the forward and the reverse processes can be modeled as Markov Chains, and the reverse data transformation can be learned by a deep denoising neural network. The overall framework of the diffusion process for net load profile generation is shown in Fig. 2.

This subsection presents how we create solar PV generation basis profiles, which will be later used in the proposed PDM. The solar PV generation depends on the weather and solar PV system specifications. By leveraging the technical parameters a solar PV system along with pertinent weather data, the output of a solar PV system can be accurately estimated using the PVSPM. Our research employs models from two key sources: the PV performance model of Sandia National Laboratory [41] and PVWatts from the National Renewable Energy Laboratory [42]. The technical parameters of a solar PV system include the system’s DC rating ($P_{dc0}$) in kW, the tilt angle ($\theta_{t}$) and azimuth angle ($\theta_{az}$) of the solar PV array, the nominal efficiency of the inverter ($\eta_{nom}$), and the system’s overall loss ($l$). Weather data encompasses temperature, wind speed, direct normal irradiance (DNI), diffuse horizontal irradiance (DHI), and global horizontal irradiance (GHI), collectively represented by $\mathbf{W}\in\mathbb{R}^{5\times T}$.

The following procedure is followed to create the solar PV generation basis profiles. We pick a few representative azimuth angles ($\theta_{az}$) and form a set $A=\{90^{\circ},120^{\circ},150^{\circ},180^{\circ},210^{\circ},240^{\circ},270^{\circ}\}$. The tilt angle ($\theta_{t}$) is set to equal to the latitude of the customer location. The representative parameters of the solar PV system including the nominal inverter efficiency and panel loss are obtained from the Sandia Module database [43]. In the dataset, all customers are located in the same city with the same longitude and latitude for solar PV systems. Thus, for a given day, weather data $\mathbf{W}$ can be accurately determined by using the conditional information: $\mathbf{W}=\mathbf{W}(\mathbf{y}_{D})$. Consequently, variations in the solar PV generation basis profiles are attributed solely to the system size and azimuth angle. The solar PV output, given the specific date information $\mathbf{y}_{D}$ and azimuth angle $\theta_{az}$, can be calculated as follows:

where $f_{\mathrm{PVSPM}}(\cdot)$ represents the PVSPM model, which uses the weather data and the azimuth angle as inputs. The output, $\mathbf{P}_{basis}(\theta_{az},\mathbf{y}_{D})\in\mathbb{R}^{T}$ serves as one of the solar PV generation basis profile associated with the azimuth angle $\theta_{az}$ and date $\mathbf{y}_{D}$.

The entire solar PV generation basis profiles for all $\theta_{az}\in A$ can be calculated:

where $\mathbf{P}_{basis}(\mathbf{y}_{D})\in\mathbb{R}^{L\times T}$ is the entire solar PV generation basis under the date condition $\mathbf{y}_{D}$. $L=7$ denotes the cardinality of set $A$, indicating the number of azimuth angles in the solar PV generation basis profiles.

Although diffusion models are successful in generating images and audio, they struggle to produce high quality net-load time series data. The net-load data can be partitioned into two distinct components: the electric load and solar PV generation. The second component can be calculated by a physical equation using the PVSPM. To effectively utilize the physical model, we propose a physics-informed diffusion framework. This innovative framework integrates physical models into the diffusion process, enhancing its generalizability and accuracy.

Let us assume that a signal $\mathbf{S}$ to be generated can be decomposed into two main components:

where $\mathbf{S}_{d}$ is the component that follows an unknown distribution, which will be learned by diffusion models. Thus, $\mathbf{S}_{d}=\mathbf{x}_{0}$ is calculated through a Markov chain $\{\mathbf{x}_{N},\cdots,\mathbf{x}_{0}\}$, given $\mathbf{x}_{N}\sim\mathcal{N}(0,I)$ and transition kernel (7). $\mathbf{S}_{p}$, on the other hand, can be calculated using a parameterized function with the basis profiles of a physical model and contextual variables.

Suppose the physical component can be represented as:

where $\mathbf{y}$ contains the contextual features and $\phi$ denotes the unknown parameters. $\mathbf{P}_{basis}$ is the PV generation basis profiles, which can be calculated through the PVSPM model. The details about $\mathbf{P}_{basis}$ will be introduced in the next subsection.

The first component of the time series, $\mathbf{S}_{d}$, is the end point of a Markov chain, whereas the second component $\mathbf{S}_{p}$ can be calculated through a parameterized physics-informed function. After both the Markov chain and the physics-informed function are learned, computing $\mathbf{S}$ becomes straightforward. However, since both the transition kernel and the physics-informed function contains unknown parameters, i.e., $\theta$ and $\phi$, it is difficult to learn them together.

The intuitive strategy might involve alternating updates for the two sets of parameters. However, this approach requires long training time and encounters instability and convergence issues. To address these challenges, we propose to learn $\theta$ and $\phi$ simultaneously by embedding the solar PV generation basis profiles $\mathbf{P}_{basis}(\mathbf{y}_{D})$ into the denoising network. The details of this approach will be elaborated in the following subsection.

The most popular denoising network architecture in the image generation field is the U-net, which achieves state-of-the-art performance due to its ability to capture the spatial correlations of pixels in images. However, when it comes to generating net load profiles, the denoising network should be tailored to learn the temporal correlations and unique daily patterns in the energy usage. For instance, residential energy consumption profile often peaks during morning and early evening hours, which coincide with the daily routines of residential customers. Similarly, solar generation profiles are closely linked to the intensity of solar radiation and has strong temporal correlations. In this subsection, we design a denoising network that is tailored for synthesizing net load time series data.

The overall architectures of both the baseline and the physics-informed denoising networks are illustrated in Fig. 3. The architecture of both the baseline and the physics-informed denoising networks, as outlined, integrates advanced neural network components to effectively handle the complex temporal correlations in the net load time series data. The proposed PDM’s network design encapsulates a blend of LSTM networks, multi-head self-attention mechanisms, and a solar PV embedding module to enhance the model’s accuracy and efficiency. The structure and operational mechanism of these components are presented below.

LSTM Embedding: Central to capturing temporal correlations in net load profiles, the LSTM network offers a robust framework for modeling time-dependent data. By utilizing a 1-layer LSTM network to process the Gaussian noise input $\mathbf{x}_{n}\in\mathbb{R}^{T}$, the model generates latent states that encapsulate temporal patterns and dependencies. The dimension of the LSTM’s hidden layer, denoted by $H$, is a crucial hyperparameter, establishing the foundation for the level of temporal complexity attainable by the network.

Positional Embedding: Regulating the diffusion/denoising process’s noise level, $\sqrt{\bar{\alpha}_{n}}$, the Positional Embedding module employs a Transformer-style sinusoidal embedding function. This approach effectively encodes the noise level within the model, ensuring that the temporal dynamics influenced by the diffusion process are accurately represented. The output of this module is adjusted to match the LSTM’s hidden layer dimension $H$.

Multi-head Self Attention: The incorporation of a multi-head self-attention module, functioning as a Transformer encoder, introduces a sophisticated mechanism for analyzing and integrating information across different segments of the input data. Maintaining both input and output dimensions at $H$, this module enhances the model’s ability to discern complex interdependencies within the net-load data.

MLP: Following the multi-head self-attention module, a Multi-Layer Perceptron (MLP) module with Leaky ReLU activation function in introduced. This module retains the dimensionality at $H$, which helps maintain the processed data’s depth and complexity.

Conditional Embedding: This module is another MLP, which encodes the conditional information with dimension $C$ and produces an output with dimension $H$ that aligns with the output of the primary MLP module.

Linear Layer: Concluding the network architecture, a Linear Layer module performs a crucial transformation, which changes the data dimension from $H$ to the final output dimension $T$. This step is pivotal in aligning the processed data with the original signal space, ensuring that the output closely matches the targeted net-load profile.

The details of the PV embedding module are shown in Fig. 4. This module contains 5 submodules. The operational flow and functionality of each submodule are outlined below:

Weather Query: This initial step involves taking specific date information, denoted as $\mathbf{y}_{D}$, and querying to retrieve corresponding weather data. The outcome is a comprehensive set of weather variables for the given date, symbolized as $\mathbf{W}$, which includes temperature, wind speed, DNI, DHI and GHI.

PVSPM: Leveraging the weather data acquired from the previous step, along with representative azimuth angles $\theta_{az}$, this submodule employs the PVSPM to calculate the solar PV generation basis profiles. The output, $\mathbf{P}_{basis}$, is a matrix in $\mathbb{R}^{L\times T}$, where $L$ represents the number of representative azimuth angles, and $T$ signifies the time dimension.

Flattened Layer: Upon receiving $\mathbf{P}_{basis}$, this layer acts to reshape the matrix into a flattened vector. The transformation process effectively converts the $L\times T$ matrix into a one-dimensional vector, facilitating its subsequent processing. This step is crucial for aligning the data structure with the needs of subsequent neural network modules.

PV-basis Embedding: Following the flattening process, this submodule employs a MLP architecture with three layers. It utilizes the Tanh function as its activation mechanism, processing the flattened vector to generate a feature representation that captures the intricacies of solar generation potential under varying weather conditions.

Cond-PV Embedding: Parallel to the PV-basis Embedding, this submodule is another MLP, albeit with two layers. It also adopts the Tanh function for activation. The primary role of this submodule is to further process and refine the information, preparing it for final integration into the diffusion model.

The culmination of these steps is the element-wise multiplication ($\otimes$) of the outputs from the PV-basis Embedding and Cond-PV Embedding submodules. This operation fuses the learned representations of solar generation and other conditional information, resulting in a highly nuanced feature vector ready to be utilized within the diffusion model’s framework. This integrated approach allows the model to leverage detailed solar PV generation information, thereby enhancing its predictive accuracy and relevance for applications involving physical signals. Finally, the input size and output size for each module are summarized in Table I.

Net load data. The net load data for residential customers in Austin, Texas, collected by Pecan Street Inc. are used in our experiments [40]. This dataset includes the daily energy consumption, solar PV production, and net-load time series of individual customers. Specifically, the net load data of 25 residential households from January 1, 2018, to December 30, 2018 are provided. Measurements were recorded at 15-minute intervals, achieving a 99% completeness rate across all intervals for the 25 homes. The missing values are imputed using the average net load of the adjacent hours. Given the 15-minute sampling rate, the dimensionality of the daily net load profile, $T$, equals $96$. Furthermore, the net load profiles for each customer are Min-max normalized to fall within the range from $-1$ to $1$.

Conditional information. The conditional information $\mathbf{y}$ contains user ID, solar PV system information, and variables associated with the specific dates. For user identification, We employ one-hot encoding for the 25 users, denoted as $\mathbf{y}_{ID}\in\mathbb{R}^{25}$. The dataset provides detailed solar PV system information, including the total and orientation-specific (west, south, east) capacities of solar PV systems installed at each household, represented as $\mathbf{y}_{PV}\in\mathbb{R}^{4}$. To account for the variations in electricity usage behavior across time, we represent specific dates using the following variables: one-hot encoding for month (12 dimensions), day of the week (7 dimensions), and day of the month (31 dimensions), resulting in $\mathbf{y}_{date}\in\mathbb{R}^{50}$. Consequently, the total dimension of the conditional information, $\mathbf{y}=[\mathbf{y}_{ID},\mathbf{y}_{PV},\mathbf{y}_{date}]^{T}$ is $79$.

Weather data. Hourly weather variables such as DHI, GHI, DNI, temperature and wind speed, are collected from the National Solar Radiation Database [44]. The meteological variables are then converted to 15-minute intervals through linear interpolation to align with the net load data. The approximate longitude and latitude of Austin, Texas $(30.27^{\circ}\text{N},-97,74^{\circ}\text{E})$ is used as a common proxy location for all customers as their exact locations are not available. The tilt angle $\theta_{t}$ used to calculate the solar PV generation basis profiles is set as the latitude, i.e., $\theta_{t}=30.2672$. The azimuth angle is set from $90^{\circ}$ to $270^{\circ}$ with interval of $30^{\circ}$. Then we can generate AC output for panels facing different directions ranging from east to west.

In the experiments, we randomly select net load profiles for $5$ customers with $365$ days, thus the total number of dataset contains $1825$ trajectories. For each customer, we randomly select $60\%$ of the dataset for training and the remaining dataset are used for testing.

Our benchmark models include WGAN, VAE and NF developed in [25]. The idea behind GANs are adversarial training of two neural networks: the generator and the discriminator. The state-of-the-art Wasserstein GAN with gradient penalty (WGAN-GP) limits the gradient norm of the discriminator’s output with respect to its input to enforce the 1-Lipschitz conditions [45], which can reduce mode collapse and improve the stability of training. VAE contains an encoder and a decoder network, which are jointly trained to maximize a lower bound on the likelihood. NF learns a sequence of transformations, a flow, from a density known analytically, e.g., a normal distribution to a complex target distribution [25].

Our study also uses a baseline diffusion model for benchmarking purposes. To ensure a fair evaluation, both the baseline diffusion model and the physics-informed model were configured with identical hyperparameters. To balance sample generation quality and speed, we design the noise schedule, $\beta_{t}$, as a linear function for $t=1,\cdots,500$. To accelerate training, we employ a learning rate scheduler that applies a decay factor of 0.9 to the learning rate every 1000 steps. During the diffusion process, the result of each iteration is clipped to be within the range from $-1$ to $1$. Additionally, we utilize an exponential moving average (EMA) for parameter updates in our model. The EMA model is refined at each step by combining $\mu$ times the current EMA model with $1-\mu$ times the newly updated weights from the most recent forward and backward pass. Here $\mu$ is called the “smoothing factor”.

All the hyperparameters of diffusion models are shown in the Table II. Training is completed on a Linux machine with Nvidia 2080 Ti GPU. It’s worth noting that the training time for diffusion models exceeds 2 hours, and the sampling time for 1 trajectory is about 15 s. Note that we plan to reduce the training and sampling time in the future work. In this paper we mainly focus on improving the quality of the synthetic data.

For evaluating synthetic images, the key metrics are the Fréchet Inception Distance (FID) and Inception Score (IS). These metrics have been widely applied to assess the quality and diversity of the images produced by generative models, providing insights into how well a model can generate new, realistic images that resemble a given dataset. However, the evaluation metrics developed for synthetic image can not be directly applied for net-load profiles creation. This is because the number samples for net-load profiles is much smaller than that of the image datasets. Thus, we introduce an evaluation system with a set of complementary metrics to thoroughly assess the quality and diversity of synthetic net load data. Our evaluation framework is structured into four parts:

Synthetic Conditional Net Load Data. This part of the evaluation compares synthetic net load profiles generated under given conditions with actual data. The objective is to evaluate the generative models’ ability to capture complex net load patterns accurately.

t-SNE Visualizations for Each Customer’s Net Load Profile. Utilizing t-SNE visualizations for each customer, this part of the evaluation aims to uncover the generative models’ ability to reproduce the nuanced conditional distributions that are unique characteristic of individual customers. It provides insight into whether the generative models can learn the diverse and specific net-load patterns of different users.

Probabilistic Forecasting of Marginal Distribution. In this part, continuous ranked probability score (CRPS) and quantile score (QS) are employed to assess the accuracy of probabilistic forecasts. CRPS evaluates the models’ skill in forecasting marginal distributions for specified times of the day. QS is used to compare the accuracy of forecasts at specific quantiles, highlighting the models’ precision in predicting extreme conditions.

Other Quantitative Metrics. In this part of the evaluation, we select 6 quantitative metrics to measure the overall similarity between the distribution of the generated and actual data.