A New Time Series Similarity Measure and Its Smart Grid Applications

Modern power systems face serious challenges imposed by the integration of renewable power plants, the emergence of distributed generation resources, and the requirements for decarbonisation. To address these challenges, traditional power systems must evolve into smarter, more efficient, and more sustainable energy networks, that is, smart grids, with the help of grid digitalisation. Digitalisation refers to the use of digital and other advanced technologies to monitor and coordinate the needs and capabilities of generators, grid operators, end users and other stakeholders in the electricity market [1, 2]. With the increased usage of digital devices and the improvement of computational capacity on smart sensors, data mining has become an essential tool to realise the true benefits of grid digitalisation in extracting useful information from the vast amounts of data collected by data acquisition terminals in smart energy systems. This will ultimately lead to higher customer satisfaction and energy efficiency [3].

Data mining in this context refers to any mathematical techniques that help discover energy usage patterns, among other insights, to facilitate the implementation of smart grid applications, e.g., studying occupancy and occupant behaviour. In this way, energy providers can offer more tailored services to consumers, detect faults remotely, operate demand response programs, optimise energy use, and reduce green energy spillage by load shifting. Energy usage patterns contain trends and features that are indicative of long time series and can be used to improve decision making through knowledge discovery in smart grids [4, 5]. Discovering patterns requires approaches to measure the similarity between two or more time series, allowing the identification of patterns, trends, anomalies, and outliers in time series data. This problem is called range queries in the literature when identified patterns are extracted based on a given distance threshold [6]. Alternatively, it is referred to as nearest-neighbour queries when the extracted patterns are the ones with the smallest distance [6]. Consequently, distance measurements are critical in identifying featured patterns for smart grid applications.

In general, time series distance measurements fall into two categories: shape-based similarity and structure-based similarity [7]. The former compares the similarity of two time series pointwise, whereas the latter converts the data into higher-level structures for comparison [8, 5]. Shape-based similarity approaches are generally preferred in electricity data analysis because user behaviours affect the use of appliances, making it more accurate and interpretable [9, 7]. Within the shape-based similarity measures, two approaches are extensively used in the literature, namely Euclidean distance (ED) and dynamic time warping (DTW) [8, 7, 5]. In addition to ED and DTW, different studies apply other basic spatial measurements such as Manhattan or Chebyshev distances, but the performances are reported to be no different from ED [10].

Depending on consumers’ preferences and external factors, such as weather, day-to-day residential electricity consumption can deviate from expected values in time (by shifting loads) and amplitude (by changing consumption level). Therefore, comparing two consumption profiles makes more sense when both aspects are considered. However, ED only accounts for changes in electricity loads at the same time without temporal considerations. To address this issue, DTW was introduced for load signature detection [11, 5], which considers the temporal dynamics of a time series by disregarding the indices (timestamps). However, DTW cannot take into account the time shifting or the exchange of the temporal sequence of the electrical load, both of which significantly affect energy consumption. This issue is explained in depth in Section II with a discussion of measuring electric vehicle (EV) charging series. To address this challenge, our paper presents a new distance measure that considers both the temporal and the amplitude changes in the electricity time series data. It takes into account load increments, decrements, shifts, and temporal sequence exchange. Therefore, it can quantify both temporal and amplitude differences in time series. We specifically developed this measure for the analysis of electricity data, but it can also be used in range queries and nearest neighbour queries for other data mining and time series analysis. The performance of the proposed measure is investigated using real-world data employed in three different applications, namely anomaly detection, load shifting, and behind-the-meter (BTM) equipment identification.

This paper is organised as follows: Section I provides background on applying data mining in power systems and identifies issues in distance measurement. Section II discusses these problems in more detail by outlining the issues with existing similarity measures. Section III presents the proposed methodology based on the requirements of an ideal similarity metric and compares the proposed method with ED and DTW. Simulation studies are given in Section IV, where the results are analysed. In Section V, limitations and future work are discussed. The article is concluded in Section VI.

More specifically, ED only measures the amount of energy change at each interval, whereas DTW considers two time series similar if they have similar patterns, even if these occur at different times. This difference is illustrated in an example shown in Fig. 1, where we compare two data sets of EV charging between 10 am and 7 pm. ED compares electricity consumption at the same interval by comparing charging levels at the same points in time, disregarding any temporal delay or shift. However, DTW considers the temporal delay by comparing the point value of series $\mathbf{A}$ with the previous, current, and next counterparts in series $\mathbf{B}$. For example, it compares the charging consumption at 12 pm in series $\mathbf{A}$ with the value at 11 am in series $\mathbf{B}$, resulting in a zero difference. However, from a power system engineering point of view, they should not be considered identical, as it requires a certain amount of effort, incentive, or compromise in comfort to shift the load to an hour later. The ability to quantify temporal changes is referred to as temporal sensitivity hereafter. Furthermore, DTW finds the EV charging at 1 pm in time series $\mathbf{A}$ similar to the values from noon to 2 pm in time series $\mathbf{B}$ with zero difference, while these two patterns are not similar. In time series $\mathbf{A}$, 12 kW is consumed in one hour, whereas in time series $\mathbf{B}$, we see a shift in EV charging by one hour and an extension of charging for the next two hours. As a result, these two patterns show different levels of energy consumption, starting points, and duration, making them dissimilar. This characteristic that DTW cannot handle is later referred to as temporal uniqueness.

Thus, given two time series $\mathbf{X}$ and $\mathbf{Y}$, a distance matrix with elements $d_{i,j}=(X_{i}-Y_{j})^{2}$ contains the point value distance between $X_{i}$ and $Y_{j}$. A route selection matrix with elements $r_{i,j}\in\{0,1\}$, specifies the path for computing the similarity of $\mathbf{X}$ and $\mathbf{Y}$ by accumulating selected point value distances in $d_{i,j}$. Hence, an ideal similarity measure, $M(\mathbf{X},\mathbf{Y})=\sum_{i=1}^{m}\sum_{j=1}^{m}r_{i,j}\cdot d_{i,j}$ should meet the following seven requirements:

1. 1.

   Non-negativity: $M(\mathbf{X},\mathbf{Y})\geq 0$

2. 2.

   Identity: $M(\mathbf{X},\mathbf{Y})=0$, if and only if $\mathbf{X}=\mathbf{Y}$

3. 3.

   Symmetry: $M(\mathbf{X},\mathbf{Y})=M(\mathbf{Y},\mathbf{X})$

4. 4.

   Triangle inequality: $M(\mathbf{X},\mathbf{Y})\leq M(\mathbf{X},\mathbf{Z})+M(\mathbf{Z},\mathbf{Y})$ for any time series $\mathbf{X},\mathbf{Y},\mathbf{Z}$

5. 5.

   Temporal uniqueness: $\sum_{i=1}^{m}r_{ij}=1\;\forall\;j\in\{1,\dots,m\}$

6. 6.

   Temporal sensitivity: $d_{i,j}\neq 0$, if $i\neq j$

7. 7.

   Optimal match:
   $M(\mathbf{X},\mathbf{Y})=\displaystyle\min_{r\in\mathcal{R}}\{\sum_{i=1}^{m}\sum_{j=1}^{m}r_{i,j}d_{i,j}$}

Upon comparing ED and DTW with the ideal measurement requirements outlined above, it is apparent that ED fails to satisfy requirements 5 to 7. DTW fails to meet requirements 2, 5 and 6, while requirement 3 is partially fulfilled depending on whether a symmetric step pattern is applied [12]. A detailed comparison between the two methods is illustrated in Fig. 2 for two sample time series. Time series $\mathbf{A}$ and $\mathbf{B}$ in the time domain are displayed on top, while the distance matrix built from the pointwise distance $d_{i,j}=(A_{i}-B_{j})^{2}$ is shown on the bottom. The selected route where $r_{i,j}=1$ is marked in purple for ED and green for DTW. To conclude the measures of ED and DTW as defined in Eqs. (1) and (2), the distance of the series $\mathbf{A}$ and $\mathbf{B}$ can be calculated as the square root of the accumulation of all pointwise distances on the path following the two routes. As shown by the purple line, ED only accounts for pairwise value changes of the two series at the same interval, i.e., along the diagonal of the distance matrix. Whereas DTW considers the patterns’ dissimilarity by calculating the warping path, which minimises the total distances between corresponding points (the green line in the distance matrix in Fig. 2). It allows “stretched” and “shifted” patterns to be identified as identical by choosing multiple points in a series to compare with another point in the other time series from different time steps. Furthermore, DTW inherently ignores the fact that we cannot compare one interval from series $\mathbf{A}$ with three intervals from series $\mathbf{B}$. Consequently, DTW fails to meet requirement 5.

To represent the possible route that fulfills temporal uniqueness of requirement 5, we define $r_{i,j}$ as binary matrix elements, where each row and column has only one non-zero element, i.e., a route selection matrix, subject to:

In the second phase, we select the optimal route to fulfill Requirement 7. The FD metric can be calculated as follows:

The main idea in the proposed distance measure is to see how one time series can be reshaped into the other time series at a minimum cost. In contrast to the pointwise distance matrix built with ED and DTW shown in Fig. 2, our proposed method considers the cost of both the amplitude and temporal changes by the first phase, i.e. Eqs. (4) and  (5). For example, if we move the load consumption from time $i$ in the first time series to a different time $j$ in the other time series, the ED and DTW are unable to tell the difference, i.e., $C_{3,4}=0$ when $X_{3}=Y_{4}$ and $3\neq 4$, as shown by the number in boldface in Fig. 2. However, the FD metric can quantify this difference, as shown in Fig. 3. The second phase aims to minimise the cost of reshaping the time series $\mathbf{X}$ into $\mathbf{Y}$ by Eq. (7), which is a linear sum assignment problem (LSAP). The LSAP problem can be solved by the Kuhn-Munkres Hungarian Algorithm with $O(n^{3})$ time and $O(n^{2})$ space [13]. This process involves considering all possibilities and combinations in increment, decrease, shift and temporal sequence exchange, as depicted by the red arrows derived from the improved distance matrix in Fig. 3.

Table I summarises the capability of ED, DTW and FD to meet the requirements listed in Section III. It is important to note that, under certain circumstances, if one of the two time series is flat, the cost of reshaping using FD will be equivalent to the sum of diagonal elements in the distance matrix. This is similar to ED but with a different magnitude, as in Eq. (4). In addition, 0 temporal weights make it similar to Wasserstein distance. Furthermore, an important by-product of the proposed distance metric is to find the shortest path in Fig. 3, which provides an optimal solution to reshape time series $\mathbf{A}$ into time series $\mathbf{B}$. Hence, FD as a metric could provide a quantitative distance that presents the minimum cost for reshaping electricity time series and strategies for optimal reshaping. To this end, FD could be used in many smart grid applications, e.g., to design better home energy management systems and demand response programs.

To further evaluate the effectiveness of the proposed method, we conducted three experiments, including load scheduling, anomaly detection, and multi-label classification to assess and compare the performance of the ED, DTW and FD metrics.

One downside of the proposed distance measure is the higher complexity of solving an LSAP problem compared to ED and DTW. However, we believe that the effectiveness and accuracy of the proposed method outweigh the time requirements, particularly when it comes to electricity grids as critical systems. It should be noted that modern residential users’ data is typically measured at hourly or half-hourly intervals, which means that the amount of data to be processed for each computation is limited to a relatively small window of 24 or 48 data points. For example, it takes 11 ms, 6 ms, and 1 ms for FD, DTW, and ED, respectively, to compute the distance between two time series with half-hour intervals for two days. As a result, the computational requirements are not likely to be a major obstacle, especially for applications that do not require all subsequences’ similarity. Moreover, the increasing computational capacity and enhanced edge computing techniques available to modern smart grid systems can help compensate for the higher complexity of the proposed method when high-resolution data is required for certain applications.

In future work, the temporal weights can be further developed and elaborated by involving users’ preferences, e.g., decomposing the historical load demand and using the residuals of each time stamp to reflect their temporal effort more accurately in a user-centric manner.

The simulation studies conducted in this paper illustrate the effectiveness of the proposed method in three applications. In future work, we plan to apply our method to a large number of users with high-resolution data. Other smart grid applications, e.g., clustering, load disaggregation, and demand side management, could also benefit from FD and will be investigated in the future.

This paper proposed a new distance metric that can quantify the effort to reshape a time series to another. It can be used in time series analysis when users must consider reshaping and scheduling. Compared to ED and DTW, the proposed FD metric considers both the pointwise changes and the temporal shifting and temporal sequence exchange in the data. Therefore, it can play a critical role in many smart grid applications when energy utilities and participants analyse data on residential electricity usage collected by smart devices. This could result in improved energy efficiency, reduced green energy spillage, and greater customer satisfaction.