A Novel Data Pre-processing Technique: Making Data Mining Robust to Different Units and Scales of Measurement

In databases, data objects represent real-world entities defined by a set of selected features or properties. For example, people can be defined by their name, age, salary etc. and cars can be represented by model, price, fuel efficiency etc. Each data record/instance represents an instance of real-world entity defined by their values of the selected features. Features can have numeric or categorical values. In this paper, we consider the case where features have numeric values. Let $D$ be a collection of $N$ data instances $\{{\bf x}^{(i)}\}_{i=1}^{N}$. Each data instance ${\bf x}$ is represented as a vector of its values of $M$ features $\langle x_{1},x_{2},\cdots,x_{M}\rangle$ where $\forall_{j}\mbox{ }x_{j}\in\mathbb{R}$ and $\mathbb{R}$ is a real domain.

In real-world applications, numeric features are often measured or recorded in some units or scales [1, 2, 3, 4]. For example, annual income of people can be recorded in integer scale as $x=100,000$ or using a logarithmic scale of base 10 like $x^{\prime}=5$. Sample variability can be measured in terms of standard deviation ($std$) or variance ($var$) ($std=\sqrt{var}$ or $var={std}^{2}$). Similarly, fuel efficiency of cars can be measured in km/ltr as $x=9.0$ or ltr/100km as $x^{\prime}=11.11$ where one is the inverse of the other.

Most of the existing data mining algorithms use feature values of data directly to build their models. They assume that data are embedded in an $M$-dimensional Euclidean space and use the spatial positions of data in the space. Many of them compute distances between data points and/or model density distribution of data. Because feature values depend on how features are measured, existing algorithms are sensitive to units and scales of measurement of features. They might give different patterns if data are measured in different units and/or scales. For example, Fig 1 shows the distributions of annual incomes of 500 individuals in two scales - normal integer scale and logarithmic scale of base 10. The two distributions look very different from each other. Similarly, distances between data may also be changed if they are expressed/measured differently. For example, if $x=1,y=2$ and $z=3$, $y-x=z-y=1$. However, if the same data are provided as their squares ($x^{\prime}=1,y^{\prime}=4$ and $z^{\prime}=9$), $y^{\prime}-x^{\prime}<z^{\prime}-y^{\prime}$.

The task-specific performances of many existing data mining algorithms depend on how data are measured. They may perform poorly if data are not measured in appropriate units or scales. Unfortunately, information regarding units and scales of feature values are often not available during data mining, instead only the magnitudes of feature values are available. Even if known, the units and/or scales used for measurement may not be appropriate for the task at hand. Data need to be pre-processed to transform into an appropriate scale before using existing data mining algorithms. The simplest way to identify appropriate scale is to try different transformations and test which one produces the best task specific result. Because there is an infinite number of possible transformations, it is not feasible to find best transformation from trial and error [5].

One simple solution is to use ranks of feature values. Rank transformation [6] is robust to units and scales of measurement as it either preserves or reverses the ranks of data even though they are expressed differently. The main issue with the rank transformation is that it makes the distribution of resulting data uniform because the rank difference between two consecutive points is always 1 regardless of difference in the magnitude. Since the relative differences of data instances are lost, rank transformation may not be appropriate in some data mining applications.

To overcome the above mentioned limitation of the rank transformation, we proposed a new variant of rank transformation using an ensemble approach. Instead of computing rank of data point $x$ among all $N$ points in $D$, we propose to compute the ranks of $x$ in $t$ sub-samples $D_{j}\subset D$ ($j=1,2,\cdots,t$) where $|D_{j}|=\psi<<N$ and use the average rank as its final transformed value. We call the proposed transformation technique Average Rank over an Ensemble of Sub-samples (ARES). Similar to the rank transformation, ARES is also robust to units and scales of measurement. However, it also preserves the relative differences between data points to some extent.

Our empirical results in the classification and anomaly detection tasks using different algorithms in a wide range of data sets show that pre-processing using ARES results in either better or competitive task specific results compared to widely used min-max normalisation and rank transformation based pre-processing of data. Furthermore, rank and ARES transformations are both robust to units and scales of measurement of raw data whereas min-max normalisation is not.

The rest of the paper is organised as follows. Section 2 provides a brief review of previous works related to this paper. The proposed method is discussed in Section 3 followed by experimental results in Section 4 and conclusions in the last section.

Psychologist S. S. Stevens (1946) [1] discussed four types of scales of measurement and their statistical properties which have been largely argued by many researchers. Since then, the scales of measurement has been a subject of discussion mainly in the measurement theory and among psychology communities. Their argument is that raw numbers can be misleading as they could have been measured/presented in different ways. One should not conclude anything from a given set of numbers without understanding where they come from and the underlying process of generating them [7, 8, 9]. Although this issue is equally important if not more in automatic pattern analysis, it has been hardly discussed in the data mining literature.

Velleman and Wilkinson (1993) [9] argued that good data analysis does not assume data types and scales because data may not be what we see. Joiner (1981) [10] provided some examples of ‘lurking variables’ — data appear to have one type whereas in fact hide other information. However, in data mining, numeric data are assumed to be in ‘interval scale’ [2, 5] — a unit difference has the same meaning irrespective of the values. This assumption may not be true when data are represented as a non-linear transformations, such as logarithm and inverse.

Different features may have different ranges of values. For example, annual income of people can be in the range of tens of thousands to millions of dollars whereas age is in the range of 1 to 100 years. Since many data mining algorithms use feature values directly to solve different tasks, their analysis can be dominated by features with larger range of values. To address such dominance, data values in each features are re-scaled to be in unit range [0,1] [11] before passing them into data mining algorithms. This is referred to as min-max scaling or normalisation in the literature. Though it resolves the differences in the ranges of values across features resulted due to linear scaling (e.g., pounds vs kilograms or ^∘C vs ^∘F) where interval scale assumption holds, it does not resolve the differences in ranges resulted due to non-linear scaling of data (e.g., km/ltr and ltr/100km) where interval scale assumption is violated.

Recently researchers started to investigate the issue of units and scales of measurement in data mining [2, 3]. One potential solution suggested in the literature is to do rank transformation [6] and use ranks instead of actual values. It is robust even to non-linear scaling because the rank is either preserved or reversed. Some prior work [5, 12] show that rank transformation results in better task specific performances than using raw values with some data mining algorithms particularly those using distance/similarity of data instances. The main issue with the rank transformation is that it makes the distribution of resulting data uniform which can be detrimental in some data mining tasks.

Another line of research related to this area is representation learning [13, 14] where the task is to map data from the given input space into a latent space which maximises the task specific performance of a given data mining algorithm. Representation learning can be viewed as learning appropriate transformation that suits best for the data set and algorithm at hand. However, this approach has some issues: (i) requires extensive learning which can be computationally expensive in large and/or high-dimensional data sets; (ii) learns representation appropriate for the given algorithm, representation learned for one algorithm may not be appropriate for others in the same data set; and (iii) one cannot interpret the meaning of new features and what type of information they capture.

Previous researches in this area are primarily focused on the algorithm level and developed new algorithms which are robust to units and scales of measurement [2, 3, 5, 12, 4]. In this approach, different set of algorithms need to be developed for each data mining task. We believe a better option would be to work in the data pre-processing level and develop a pre-processing technique robust to units and scales of measurement so that existing data mining algorithms can be used as they are. In this paper we propose one such technique to transform data such that the resulting distribution is similar even though raw data are given in different units and/or scales.

To address the issue of rank transformation resulting in uniform distribution of the transformed data, we propose a new variant of rank transformation using an ensemble approach. In each feature or dimension $i$, instead of computing rank of $x_{i}$ among all $N$ values, we propose to aggregate ranks of $x_{i}$ in $t$ sub-samples of values in dimension $i$. We call the proposed method as ARES (Average Rank over an Ensemble of Sub-samples) transformation.

To make the explanation simple, we assume $D$ is a one-dimensional data set where $|D|=N$. We use $t$ sub-samples $D_{j}\subset D$ ($j=1,2,\cdots,t$) where $|D_{j}|=\psi<<N$. The transformed value of $x$, $\tilde{x}_{ARES}$, is the average rank of $x$ over all $t$ sub-samples.

where $r(x|D_{j})$ is the rank of $x$ in $D_{j}$:

Based on the definition of $r(x|D_{j})$ given by Eqn 2, all $x\in D$ have ranks in $\{0,1,\cdots,\psi\}$ in $D_{j}$. If $s_{j}^{(1)},s_{j}^{(2)},\cdots,s_{j}^{(\psi)}$ are sorted samples in $D_{j}$:

All $x\in D$ such that $s_{j}^{(k)}\leq x<s_{j}^{(k+1)}$ will get the same $r(x|D_{j})$ of $k$ regardless of the their differences in magnitudes. They can not be differentiated. However, some of them will have different ranks in other sub-samples. Thus, averaging over different sub-samples will maintain their differences to some extent. For example, even the case of $|D_{j}|=\psi=1$ where all $x\in D$ have ranks either 0 or 1 depending upon either they lie on the left or right of the sample selected in $D_{j}$, doing it multiple times, say $t=10$, maintains the differences between data points to some extent. Fig 2(a-c) show the distribution of an example data set and its transformations using traditional rank and ARES. It is clear that ARES (Sub-figure c) preserves the differences in data values in the original distribution (Sub-figure a) better than the rank transformation using the entire $D$ (Sub-figure b).

Like the traditional rank transformation, ARES is also robust to the changes in the scales and units of measurement. It is a variant of rank transformation using sub-samples in ensemble. Fig 2(d-i) show the same data as in Sub-figure (a) in logarithmic¹¹1The default base of logarithm in this paper is $e$, i.e., natural logarithm, unless specified otherwise and inverse scales and their transformations based on the traditional rank and ARES. From Sub-figures c, f and i, it is clear that ARES results in the same distribution. Note that the resulting distribution in the case of inverse scale (Sub-figure i) is the reverse of that in the original and logarithmic scales (Sub-figure c and f). There is some small differences in the ARES transformations of samples selected in $D_{j}$ (i.e., $x=s_{j}^{(k)}$) in the case of inverse scale because of the ‘$<$’ sign in Eqn 2.

In terms of run-time, ARES requires $O(t\psi\log\psi)$ time to generate and sort $\psi$ samples $t$ times and $O(Nt\log\psi)$ to compute ranks of all $N$ instances in $D$. The traditional rank transformation requires $O(N\log N)$ time to sort and rank $N$ instances in $D$.

In this section, we present the results of experiments conducted to evaluate the effectiveness of ARES against the widely used min-max normalisation and the traditional rank transformation. We evaluated them in terms of their task specific performances and robustness to the changes in the units/scales of measurement in the classification and anomaly detection tasks using a wide range of publicly available data sets.

In order to mimic the real-world scenario of possible variations in units/scales in data, monotonic transformations using logarithm, square, square root and inverse as $\log x,x^{2},\sqrt{x}$ and $\frac{1}{x}$ were applied to data in each dimension. To cater for $x<0$ where $\log x$ and $\sqrt{x}$ are not defined, all transformations were applied after rescaling feature values in the range of [0,1] in all dimensions. Further, to cater for $x=0$ in the case of $\log x$ and $\frac{1}{x}$, all monotonic transformations were applied on $x^{\prime}=b(x+a)$ where $a=0.0001$ and $b=100$. Scaling using $b=100$ was used to change the inter-point distances significantly. We used the same settings of monotonic transformation as discussed in [5].

The proposed method ARES and the traditional rank transformation were implemented in Python using the Scikit Learn library [15] which has the implementation of the min-max normalisation. Two parameters $\psi$ and $t$ in ARES were set empirically as default to 7 and 10, respectively.

We discuss the experimental setups and results in the classification and anomaly detection tasks separately in the following two subsections.

Many existing data mining algorithms use feature values directly in their model making them sensitive to units/scales used to measure data. Pre-processing of data based on rank transformation has been suggested as a potential solution to overcome this issue. Because the resulting data after pre-processing is uniformly distributed, it may not be very useful in many data mining tasks.

To address the above mentioned issue of the traditional rank transformation using the entire data set, we propose to use ranks in multiple sub-samples of data and then aggregate them. The proposed approach preserves the differences in data values in the given original distribution better than the traditional rank transformation. Our results in classification and anomaly detection tasks using widely used data mining algorithms show that data pre-processing using the proposed approach produces better results than the pre-processing using the traditional rank transformation. In comparison to the most widely used min-max normalisation based pre-processing, it produces more consistent results with the change in units/scales of data. Its results are either better or at least competitive to the min-max normalisation of the resulting data after the best scale mimicked by the monotonic transformations considered in this paper.

Therefore, it is recommended to pre-process the given data using the proposed approach for a good and consistent results of existing data mining algorithms particularly when there is uncertainty on how data are measured or presented.

In future, we would like to test the effectiveness of the proposed data pre-processing technique in: (i) other data mining tasks such as clustering, (ii) application areas such as IoT, sensor networks and health care where data could be in different units/scales as they are measured by different sensors/devices.