QI² - an Interactive Tool for Data Quality Assurance

Up to now ML systems are mostly used in areas, where wrong decisions only have minor consequences. The increasing performance of ML compared to classical systems especially for perceptions tasks in natural complex environments gives hope for e.g. driver-less vehicle operation in foreseeable future. The hopes of using ML also for perception tasks in areas of higher risk currently may not be fulfilled, due to the required trustworthiness of the systems including assured high quality of the used data sets may not be proven. To give a legal framework for the application of ML systems the European AI act (European Comission 2021) is currently under development. Simultaneously there are multiple projects from research and industry working on the topic of ML Systems in high risk areas like KI-Absicherung (Burton et al. 2022) (Mock et al. 2021) and safe.trAIn (Siemens 2022). High risk ML systems have to fulfill the requirements according to (European Comission 2021) Chapter 2 ”REQUIREMENTS FOR HIGH-RISK AI SYSTEMS” Article 10 ”Data and data governance” Point 3:

Training, validation and testing data sets shall be relevant, representative, free of errors and complete.

In this paper we present a novel approach to visually represent and quantify data quality, and demonstrate this approach with several different application examples. The aim is to provide new tools to understand and manage the data quality. The approach is a part of the QUEEN-method (Qualitätsgesicherte effiziente Entwicklung vorwärtsgerichteter künstlicher Neuronaler Netze, quality-assured efficient development of neural networks) (Waschulzik 1999) a high-level procedure for quality assured development of neural networks. QUEEN gives guidance for developing explainable artificial intelligence (XAI) based on the development of dedicated prepossessing. It is supported with quality indicators that are used as key performance indicators for the achieved complexity reduction for the given task. Quality assuring the processing of every single step before the data is used as input for a neural network supports the trustworthiness for the data bases and is combined with lightweight AI systems that can be interpreted in detail. QUEEN defines two data quality assurance methods called QI² (integrated quality indicator) (Geerkens 2021) and ECS (equivalent classes sets) (Sieberichs 2021). In this paper we want to show the mathematical bases as well as the applications of QI² and its underlying methods in terms of data quality assurance. The method quantifies the complexity of input-output relationship defined with a data set. The term complexity herein is representative for a measurement of the amount of non-linearities in input-output relationships. It is based on the pairwise measurement of input and output distances what makes it attractive for a wide variety of applications due to simply adapting the distance metric. The ECS is not part of this paper but is covered in the submission (Sieberichs et al. 2023).

The QI² and its respective methods are a handy tool in terms of quality assurance due to a low dimensional representation of data. It quantifies neighborhood input-output relationship behaviors over a set of data points. High dimensional anomalous structures and relationships in data are represented by anomalous structure in a simple visualisation. It is possible to directly and accurately address several quality aspects - e.g. global linearity, outlier, simple and difficult sub-structures, clustering structure, inconsistent data, discontinuities - by simply interacting with its visual representation.

Data quality in general is a widely researched topic due to the growing necessity and usage of data in our every day life. But there are still a lot of problems to be tackled. In (Wang and Strong 1996) Data quality is defined with respect to the intended use of the consumer. Although data should be intrinsically good they state that data quality needs to be a context dependant term in order to be appropriate for the desired task. The term of data quality is furthermore split into many dimensions like accuracy, consistency, completeness, safety, timeliness etc. in (Sidi et al. 2012). (Gualo et al. 2021) considers the ISO/IEC 25012, an accepted industry standard for data quality and gives a model based approach. In addition to that (Pipino, Lee, and Wang 2002) states a separation into subjective and objective quality assessment with three functional forms. The diversity of the definition of data quality has to be resolved in the future to give a guidance to implement the European AI Act. A consensus seems to exist, that data quality has intrinsic properties - accuracy, completeness, consistency, etc - as well as properties that are more dependent on use - timeliness, presentation quality etc.

As a first glance in data quality assurance descriptive statistics (Holcomb 2016) is used. Herein statistical methods for measuring central tendencies, dispersion and association are used for basic understanding of variables in data. The most common displaying method for those statistics are visualisations via scatter plots and histograms. Our proposed method extends descriptive statistical methods with a new visualization that gives deeper inside to the spatial relationships in the typically high dimensional data to support dedicated quality assurance and enables direct interactions between different visualisations of the data.

Dimensional reduction methods like PCA (Jolliffe 1990), tSNE (Maaten and Hinton 2008) or UMAP (McInnes, Healy, and Melville 2020) are a further visually representative quality assurance. These methods map the high dimensional data into low dimensions that may be easier analysed and understood by humans. Nevertheless they suffer from a high loss of information and might impose misleading interpretations. The herein proposed method of QI² is a human interpretable visual representation of the data according to neighborhood relationships. It therefore represents high dimensional information of the data that are necessary for efficient quality assurance. It does not contain the data itself in the visualisation but it is computed on the original values and the structural behavior.

Further metrics claiming approaches for general data quality assurance try to cover as many dimensions of data quality as possible by testing data against predefined rules and assumptions. In contrast to descriptive statistics those methods do not define data quality due to a statistical and distributional structure that is met by the overall data. They assess data in order to meet certain rules and assumptions. (Iannone and Vargas 2022) published the pointblank R package for agent based data quality assurance. This package contains quality assessments on a higher level basis with testing values in specific columns or rows against defined functions. One can test if the data is correct in terms of the values either being greater, lower, equal or between certain values or follow structural or informational rules defined by the user. Another general quality assurance method is DEEQU published in (Schelter et al. 2018a) and (Schelter et al. 2018b). This package allows the user to define specific assumption based unit tests for data at large scale. Similar to the previous one this package also adds row and column based checks to a pipeline for data quality assurance. In addition to unit tests on a single data set there is the possibility of anomaly detection over time. A further assumption based approach for data quality assurance is shown in (Heinrich et al. 2018). The probability-based method herein outputs a value representing the probability of a data set being free from internal errors with respect to given rules. In comparison to the previous rule based approaches this method does not apply rules row or column based. The rules defined for this assurance are more like relationships between individual columns of data sets representing correlations. Despite a more complex application of rule based checks all mentioned methods are heavily assumption and rule based and therefore require high knowledge of the data set and accurate assumptions. Assumptions on data although will never allow full coverage of quality dimensions because one can not know what one does not know. Due to a representative visualisation of input-output relationship behavior our proposed method does not need any assumptions for certain quality aspects. It can be used without any knowledge about the data and does not need thresholds or possible coincidences between data dimensions.

For deeper quality assessment more specific methods need to be taken into account. Herein methods developed for a specific purpose will be the key. In the field of outlier detection for example there exist a variety of different approaches all tackling the same problem. A first strategy is using density-based outlier detection algorithm (Breunig et al. 2000). As a first glance this method outputs a value similar to a probability determining if a data point being an outlier. This value is computed based on how locally isolated the data point is. For this the local density of each point is compared to the local density of their $k$-nearest neighbors using the local reachability density (LRD). Another clustering based approach uses DBSCAN (Ester et al. 1996) as clustering algorithm and repeats a specific clustering and cluster merging algorithm to a state in which every cluster has its own $\epsilon$ value regarding DBSCAN. After that $minPts$ value of DBSCAN will be computed for each cluster with respect to the smallest $minPts$ of that cluster. Regarding $minPts$ clusters will be classified as anomalous (Tran Manh Thang and Juntae Kim 2011).

In further development density based clustering was used as a preprocessing in outlier detection combined with inter cluster distance based classification of clusters as anomalous. Methodologically similar approaches differing in clustering, distance computation and anomaly detection are (Fawzy, Mokhtar, and Hegazy 2013) (Samara et al. 2022) (Samara et al. 2023). The first one is using the fixed width clustering algorithm for finding clusters in a data set followed by computing inter-cluster distances. Clusters are classified as anomalous by looking at the average inter-cluster distance and its deviation from the mean inter-cluster distance. The second one above is using the previously mentioned DBSCAN clustering algorithm in combination with a anomalous classification based on inverse distance weighting (IDW). The third one is using OPTICS (Ankerst et al. 1999) algorithm for clustering and an inter cluster distance based anomalous classification with kringing methods. The methods of QI² could as well be seen as a clustering-based analysis method but differs from the mentioned methods due to the fact that outlier detection is not based on thresholds and clustering as preprocessing directly. Clustering as preprocessing can lead to loss in information regarding individual data points. Furthermore the proposed method gives a wider variety in quality assurance than only tackling one specific problem.

Since the algorithm presented visually represents the relationships between input and output changes within a data set $P$, there must first be a division of the data set into input space and output space. Therefore every data point can be described by

$$\vec{p^{T}}\coloneqq\begin{pmatrix}vi_{1}\ldots vi_{I}vo_{1}\ldots vo_{O}\end{pmatrix}$$

(1)

with $I$ values of the input-space $vi_{i}\epsilon\mathbb{R}$ and $O$ values of the output-space $vo_{o}\epsilon\mathbb{R}$.

With this separation the method of QI² can be applied. First of all it is necessary to calculate the pairwise distances regarding input-space and output-space throughout every possible pair $P^{2}\coloneqq\{x\coloneqq(\vec{p_{1}},\vec{p_{2}})\mid\vec{p_{1}},\vec{p_{2}}\epsilon\mathbb{R}^{(I+O)}\}$ of data points. For this every possible metric $M$ can be applied. For example euclidean or cosine distance for scalar values and SSIM (Wang et al. 2004) or mutual information (Russakoff et al. 2004) for images. Despite using a certain metric for the input-space the output-space can have a different metric applied. Take as an example a quality assurance of an image classification data set. In the input-space (images) the SSIM can be applied to compare the structural information as distance metric but in the output-space (classes as scalar values) the euclidean distance can be applied.

$$d_{RI}(x)\coloneqq M(x)$$

(2)

$$d_{RO}(x)\coloneqq M(x)$$

(3)

After that the distances will be normalized by mean for comparability. In the following this will be shown exemplary for the input-space.

$$d_{NRI}(x)\coloneqq\nicefrac{{d_{RI}(x)}}{{\frac{\sum_{y\epsilon P^{2}}d_{RI}(y)}{|P^{2}|}}}$$

(4)

Having the pairwise normalized distances in input- and output-space it is possible to compute a first representative value for a data set described by the data points taken into account. This value describes the complexity and quality of this data set by comparing the relationship between output changes following on certain input changes.

$$QI^{2}R(P)\coloneqq\frac{1}{|P^{2}|}\sum_{x\epsilon P^{2}}(d_{NRI}-d_{NRO})^{2}$$

(5)

This representative value gives a first shot in quality assurance of a data set by describing the non-linearity. The higher the value, the more non-linear parts are in the data set. A second handy property of this value is that it describes a two dimensional data set with random normal distribution with a value of one. If the QI²R of a data set is higher than one, the input-output behaviore of the data set is more complex than a random normal distribution. This occurs e.g. if parts of the data set are more likely to have more linear relationships between input and output but are separated by a sudden change in direction of the gradient.

The next step in quality assurance of a data set is a local complexity representation. Therefore the QI²R can be repeatedly computed over subsets of the original data set. For this the data set needs to be sub sampled by e.g. building subsets based on increasing k-nearest neighborhoods around every data point. This means for every data point in the data set there will be a number of subsets depending on how much neighbors should be taken into account. Another possibility is to divide the data set into subsets with respect to a high dimensional sphere around each data point with increasing diameter. With those subsets it is now possible to compute a matrix of local QI² (MLQI²).

$$mlqi^{2}_{i,k}(P)\coloneqq QI^{2}R(KNN(P,p_{i},k))$$

(6)

$KNN(P,p_{i},k)$ represents the buildup of subsets based on $k$-nearest neighbors around the respective data point $p_{i}$ in the data set $P$. The computation of MLQI² considers every pairwise distance between possible pairs of points in the subset.

Visualizing a matrix human interpretable is hardly possible in this case. Thus a visualization via a histogram of local QI² (HLQI²) was chosen.

$$hlqi^{2}_{v,k}(P)\coloneqq\sum_{i=1}^{|P|}{I^{3}({mlqi^{2}}_{i,k}(P),v)\cdot{blqi^{2}}_{i,k}(P)}$$

(7)

The term $blqi^{2}_{i,k}$ (boolean matrix of local QI², BLQI²) is responsible for preventing boundary effects by comparing neighborhoods. It checks whether the QI²R for the current neighborhood has already been computed, since this neighbourhood already exists on the basis of another data point. If it has already been computed this calculation will be dropped in the computation of $hlqi^{2}_{v,k}$.

$$blqi^{2}_{i,k}(P)\coloneqq\begin{cases}1,&\text{for }\forall{j<i}\\ &|KNN(P,p_{j},k)\neq KNN(P,p_{i},k)\\ 0,&\text{else}\end{cases}$$

(8)

$I^{3}(h,v)$ is responsible for sorting the MLQI² into the desired histogram by checking if the receiving value $h$ lies in the bin $[v|v+1)$.

$$I^{3}(h,v)\coloneqq\begin{cases}1,&\text{for }v\leq\frac{h-min_{hi}}{binsize_{hi}}<v+1\\ 0,&\text{else}\end{cases}$$

(9)

As a last step the histogram is scaled to its maximum value in every $k$ so that it contains relative values for the bins and it is gamma calibrated due to visual reasons for lower values (scaled histogram of local QI², SHLQI²).

$$shlqi^{2}_{v,k}(P)\coloneqq\left(\frac{hlqi^{2}_{i,k}(P)}{\sum_{s=1}^{|P|}{hlqi^{2}_{s,k}(P)}}\right)^{gamma_{hi}}$$

(10)

As stated before the SHLQI² can be used to determine certain quality aspects in data sets with respect to a local analysis. Certain aspects need special treatments regarding the identification via interaction with SHLQI². We want to present some exemplary identifications with examplary and state-of-the-art data sets.

Structural and individual distribution. The overall distribution has a huge impact in data quality. The SHLQI² can be used in order to determine the overall structure due to homogeneity of clusters as mentioned above. As an example figure 3(a) visualizes the identification of homogeneous clusters. Herein a classification data set was analysed due to its data points being in a homogeneous cluster in which every data point has 60 direct nearest neighbors with the same class. The identification therefore needs to be a check if for a certain range of $k$ the values of SHLQI² for a point is not leaving the characteristic area of homogeneous clusters.

In contrast to analysing a data set in terms of clustering it is possible to identify out of distributional data points individually. Out of distribution therefore needs to be defined as a data point being outside of its respective classes cluster but not interfering with another class. While interfering with another class this data point would be a misclassificational outlier. The identification is quite similar to the one of homogeneous clusters. The only difference is, that due to average normalization of the input distances the value of QI²R will be higher within a defined boundary of $mlqi^{2}_{i,k}\epsilon[1|2]$. This is caused by considering significant higher normalized distances between the out of distribution data point and every other point inside the respective cluster in equation 5. Therfore this identification is a check if for a certain local range of $k$ the values of SHLQI² for a point is within the range slighlty above the homogeneous characteristic and below two as presented in figure 3(b).

Locally simple input-output relationship. Data sets with multiple thousands of data points can be very complex when it comes to interpretation of the whole data set. Its input-output relationship behavior can have quite non-trivial changes or interruptions due to nonlinearities in the real world. The data points can and will have heavily complex relationships especially with increasing input dimensions. Therefore it is nice if a data set has a subset that can be specially treated due to a rather incomplex input-output relationship behavior. For those points one can apply the divide-and-conquer principle. The simpler subset can be used to get further interpretability and stability into the system the data is applied on. For example an ML system can generate high robustness against inputs near or inside this sub structure of the data. As an exemplary visualisation of identification of simple subset, an artificially created data set representing a simple sine wave with heavy noise due to undefined reasons was created. The noisy parts follow no specific rules in terms of input-output correlation. In contrast to that, the sine is following a more strict relationship between input and output dimensions. The more linear this relationship is, the lower is the local complexity for the data points in those parts. A simple subset therefore will be visible due to persistent and falling low complexities in local areas. The longer the complexities are in a low area the more data points form a simple subset. Visually this case is shown in figure 4.

It can be seen clearly that the marked lower complexities in the SHLQI² directly refer to the sine based parts of the data set. Those data points are following defined simpler structural rules and can be treated like a simple subtask.

To explain the use of the new method the popular MNIST handwritten digits training data set has been chosen. The data set consists of 60.000 different 28x28 gray scale images. We applied the computation of SHLQI² onto the whole data set with a maximum neighborhood of 3000 neighbors. At this point the expectation was that nearly the whole histogram will be covered by the previous mentioned steep rises with extremely high value due to an average amount of 6000 data points per class. According to this amount of points per class we expected the set to have a pretty decent clustering and therefore homogeneous groups with couple hundreds or thousands of neighbors from the same class. The next data point with another class would then cause an explosion of the value of $mlqi^{2}_{i,k}$. As shown in figure 5 this assumption is totally met. The complexities of MNIST reached pretty high values with a peak at $mlqi^{2}_{i,k}\approx 1100$ as well as high coverage of the histogram with classification characteristic steep rises in complexities.

Structural distribution. After first overall quality assurance a deeper analysis in order to determine the structural distribution can be done. In order to get a good knowledge of how well the data is homogeneously clustered the proposed identification of structural distribution was applied. Figure 6 shows the correlation between the proposed identification and its respective black marked data points.

Interesting to see is that for the classes ”one” (bottom left cluster), ”zero” (rightmost cluster) and ”two” (left cluster below ”zero”) almost every data point is part of a homogeneous cluster with respect to 300 nearest neighbors. In contrast to that the cluster ”nine” has only two representative data points. This already shows the difference in quality assurance between SHLQI² and dimension reduction algorithms. While UMAP represents a qualitative homogeneous cluster for class ”nine”, the SHLQI² shows that it is build up by quantitative rather small homogeneous clusters within local areas. The UMAP visualisation is misleading in terms of overall structural distribution. Furthermore table 1 shows the amount of data points building homogeneous cluster $p_{cluster}$ with at least $k$ nearest neighbors having an identical class.

Outlier detection. An outlier in classification tasks can be detected due to low MLQI² at $k=1$ and a steep rise in complexity for local areas $k\epsilon[5|25]$. A low complexity at $k=1$ represents data points having one direct neighbor with another class. This might as well be the case in terms of classification boundaries but in combination with steep complexity rises in local areas these classification boundaries will be filtered out. The filtering happens because of an evenly distributed amount of data points with another and the same class in local areas for boundaries. This would result in lower complexity values. As stated in above classification tasks have steep rises in complexity due to one or a few examples being in an homogeneous cluster which is the case for outliers. Examples for outliers, their class according to the data and their detection in MNIST training data regarding euclidean distance are shown in figure 7.

In this paper we presented a novel quality assurance method based on visually representing the neighborhood relationship. We gave concrete examples for assurance of defined quality aspects and tested them exemplary with the MNIST training data. The QI² can be used to get deeper structural and individual insights regarding qualitative and quantitative aspects of the data. The new tool-set based on QI² enables quantitative requirements for the quality assurance process. As an example a requirement for the quality assurance of the data set may be formulated, that for all potential outliers identified with SHLQI² in neighbourhoods of 10 to 50 examples the output of every example has to be validated. With this measure the probability of wrong class labels will be reduced significantly with relative low costs for quality assurance activities. The range for which neighbourhoods the validation is required may be e.g. defined according to the required safety integrity level of the application. Especially in combination with a representative visualisation of the data this tool has many handy features for an overall efficient interactive quality assurance process.