Adaptive Variance Thresholding: A Novel Approach to Improve Existing Deep Transfer Vision Models and Advance Automatic Knee-Joint Osteoarthritis Classification

Our study utilized knee joint X-ray images derived from the 2019 Chen, et al. study[39], obtained initially from the Osteoarthritis Initiative (OAI). The OAI is a multi-center study focused on biomarkers for knee osteoarthritis that involved 4796 participants aged between 45 and 79. We used the pre-processed primary cohort data[40] from the Chen 2019 study, which had undergone automatic knee joint detection, bounding, and standardization of zoom to 0.14mm/pixel. This process yielded 8260 images (224 x 224 pixels) extracted from 4130 X-rays, each encompassing both knee joints. The images were classified using the Kellgren and Lawrence (KL) system[41], as illustrated in Figure 2. The distribution of KL grades comprised 3253 images for Grade 0, 1495 for Grade 1, 2175 for Grade 2, 1086 for Grade 3, and 251 for Grade 4.

We adjusted each right knee joint image to replicate the orientation of a left knee. We subsequently located and inverted any negative channel images, leading to 189 instances. The contrast of the image histograms was then equalized using equation 1. In this equation, we took a given grayscale image I of $m\times n$ dimensions and used the cumulative distribution function $cdf$ and pixel value $v$ to derive an equalized value $h(v)$ within the range $[0,255]$. Here, $cdf_{min}$ represents a non-zero minimum value of the image’s cumulative distribution, while $m\times n$ refers to the total number of pixels.

In order to externally validate our approach, we mirrored the exact models and data utilized in the Chen et al. 2019 study[39]. We chose the study’s best-performing VGG19 - Ordinal model (publicly available[40]). We extracted the feature vectors from this model from the pre-output dense layer, effectively replicating the conditions under which the original model achieved its top performance. This step was crucial to ensure a fair comparison and assessment of the effectiveness of our approach when applied to an external model.

Convolutional Neural Networks (CNNs) [42], a cornerstone in the renaissance of deep learning [3], have often found applications in the realm of computer vision. The strength of CNNs lies in their ability to execute convolution operations between an input and a filter-kernel, thereby highlighting distinctive features captured in a response known as a feature map. As these filters slide across inputs, they create layers of complex feature maps, each representing more abstract concepts from the preceding maps. Mathematically, given an image I with dimensions $u\times v$ and a filter-kernel H of $s\times t$ dimensions, the generation of a feature map G by convolving I and H across axes $u,v$ is expressed as:

Usually, the resulting values of the feature map are filtered through an activation function, which acts to re-map these values using a pre-defined function. One common example is the Rectified Linear Unit activation function[43] (ReLu), which sets negative values to zero, thereby offering computational efficiency by eliminating less essential information. For any feature map value $z$, the ReLu activation function is defined as:

Further, a max pooling operation is often employed to down-sample the convolution result. Consequently, cascades of max pooling and convolution yield a gradually decreasing length of features. For an image I with dimensions $u\times v$, the max pooled value ${g(u_{I})}$ for dimension $u$ can be defined as:

Here, $u_{I}$ is dimension $u$ from image I, $r$ denotes the pooling window size, and $h$ signifies the stride value. This streamlining process contributes to an efficient and effective model for identifying the necessary classification task features.

A notable deep learning model, EfficientNet[44] , adopts the strategy of compound scaling, which systematically adjusts depth (number of layers), width (size of the layers), and resolution (size of the input image). This scaling process is mathematically encapsulated as follows:

Here, $\alpha,\beta,\gamma$ denote constants, $\phi$ stands for a user-selected coefficient, and $d_{0},w_{0},r_{0}$ represent the depth, width, and resolution of the original model, respectively.

A distinctive feature of EfficientNet is the MBConv block, which chains together transformations in the order of a $1\times 1$ convolution, a depth-wise convolution, a Squeeze-and-Excitation (SE) operation [45], and a subsequent $1\times 1$ convolution:

In this equation, $\textbf{{K}}_{1}$ and $\textbf{{K}}_{2}$ are $1\times 1$ convolutional filters, D stands for the depth-wise convolutional filter, and $SE(\cdot)$ signifies the Squeeze-and-Excitation operation. The EfficientNetV2[46] model extends its predecessor by incorporating a Fused-MBConv block that fuses the initial $1\times 1$ and depth-wise convolutions into a single $3\times 3$ convolution, followed by an SE operation and a concluding $1\times 1$ convolution:

Here, $\textbf{{K}}_{f}$ is the $3\times 3$ convolutional filter that merges the initial $1\times 1$ and depth-wise convolutions, and $\textbf{{K}}_{2}$ is the final $1\times 1$ convolutional filter. An activation function follows each convolution and may include a skip connection. The EfficientNetV2 base model (B0) and the modifications implemented in our study are illustrated in Figure 3.

We utilized the EfficientNetV2-M architecture for our model and further updated it by introducing flattening and a dense layer consisting of 240 neurons. The model was trained over 25 epochs using the Adam optimizer, with the dataset being partitioned into Training (75%), Validation (15%), and Testing (15%) sets (patient-wise splits). The minimum validation loss governed the early stopping criterion. The CNN training and online affine augmentations (’advanced augmentation preset’) were executed using the open-source Deep Fast Vision library[47].

Variance Thresholding is a popular feature selection method[48, 49] used to enhance machine learning model performance by removing excess features. It calculates the variance of each feature and drops those whose variance is below a specific user-defined threshold. However, the conventional method requires manually setting the threshold, which is not adaptable to different datasets and does not generalize across datasets or feature vectors of different architectures trained even on the same dataset. To overcome this, we propose Adaptive Variance Thresholding (AVT), which automatically sets the threshold from a user-defined percentile of the calculated feature variances. This approach makes the method flexible, as it adapts to the unique characteristics of each dataset without requiring manual changes. The following mathematical steps describe the process of Adaptive Variance Thresholding (AVT):

1. 1.

   Compute the variance of each feature in the feature set, F. We denote this variance as v, where each element $v_{i}$ represents the variance of the $i$-th feature:

   $$\mathbf{v}=\text{Var}(\textbf{{F}}_{i}),\quad\forall i\in{1,2,...,w}$$

   (8)

   Here, $\textbf{{F}}_{i}$ denotes the $i$-th feature in the feature matrix F with dimensions $w\times z$ (where $w$ is the number of features and $z$ is the number of samples). $\text{Var}(\textbf{{F}}_{i})$ calculates the variance of the $i$-th feature, with $\mathbf{v}$ storing these variances.

2. 2.

   Set the threshold $j$ to be the $p$-th percentile of these computed variances:

   $$j=\text{percentile}(\mathbf{v},p)$$

   (9)

   The function $\text{percentile}(\mathbf{v},p)$ computes the $p$-th percentile of the variances in $\mathbf{v}$, where $p$ is a user-defined parameter.

3. 3.

   Form a new feature matrix F’, consisting of only those features $i$ whose variance exceeds the threshold $j$:

   $$\textbf{{F}}_{i}\in\textbf{{F'}}\text{ if }v_{i}\geq j$$

   (10)

In the final step, the feature $\textbf{{F}}_{i}$ is included in the new feature matrix F’ if its corresponding variance $v_{i}$ is greater than or equal to the threshold $j$. Otherwise, the feature is removed. The power of the adaptive approach is in its capacity to derive the threshold directly from the data, allowing it to adjust to the distinct characteristics of each dataset. In our study, we examined three threshold levels, referred to as Low, Mid, and High, each representing different percentiles - 1.5%, 50%, and 98.5%, respectively, given the computational demands associated with this approach; conducting an exhaustive search was prohibitively computationally demanding. In essence, we preserved the top 98.5% varying features in the Low setting, the top 50% in the Mid setting, and only the top 1.5% in the High setting. These settings explored various scenarios from low to significant dimensionality reduction. The Python class implementing this method can be found in the provided materials. It constitutes a modification of the Scikit-Learn class[50], using NumPy[51] for computations.

Neural Architecture Search (NAS) [37], a method in machine learning, offers an automated way to design neural network architectures, diminishing the need for exhaustive manual tuning. By systematically exploring a range of potential architectures, NAS pinpoints the ones that yield the best performance for a specific task. The process involves defining a search space of potential architectures and using a controller model to generate, assess, and train these candidate architectures. In our experiment, we applied the AutoKeras[38] Structured Classifier search approach to identify suitable architectures for the Adaptive Variance Thresholding (AVT) sets. Notably, the evaluation of NAS was performed exclusively using validation data. We permitted a maximum of 55 consecutive trials with validation accuracy guiding the early stopping criterion. Each trial was given a training space of up to 25 epochs. All identified architectures are detailed in the appendix, and trained versions of these architectures are available under data availability.

Accuracy, a common measure of model performance, is the proportion of correct predictions (true positives and true negatives) to the total number of observations. It’s mathematically defined as:

Precision, also referred to as the positive predictive value, denotes the fraction of correct positive predictions in relation to all positive predictions made. The calculation for precision is as follows:

Recall, alternatively known as sensitivity or true positive rate, represents the fraction of actual positive instances that the model correctly identified. The formula for recall is:

The F1 score aims to balance precision and recall by taking their harmonic mean. It’s calculated using the following formula:

In these expressions, TP stands for True Positives, TN represents True Negatives, FP is for False Positives, and FN refers to False Negatives.

This study utilized adaptive variance thresholding (AVT) and neural architecture search on pre-trained vision neural networks. Table 1 compares the performance across AVT thresholds on the best pre-trained Chen 2019 model (baseline), highlighting the impact of these techniques. We observed that the model’s NAS input dimensionality significantly decreased from 4096 in the ’Baseline’ condition to 62 in the ’High AVT’ condition. Accuracy generally improved, peaking at 71.14% in the ’High AVT’ scenario. Precision increased to 70.98% in the ’Mid AVT’ condition and slightly declined to 69.76% in the ’High AVT’ setting. Recall remained relatively steady across all conditions with a minor increase to 68.95% in the ’Mid AVT’ scenario. The F1-score, crucial in handling uneven KL distributions, increased until ’Mid AVT’, then slightly decreased in the ’High AVT’ condition. Figure 4 further details the confusion matrixes for each condition.

In the provided confusion matrices and accuracy results, the impact of varying Adaptive Variance Thresholding (AVT) levels, namely Baseline, Low AVT, Mid AVT, and High AVT, on the performance of a neural network model is presented. A general improvement in accuracy was observed as the AVT level increased, rising from approximately 69.63% at Baseline to 71.14% at High AVT. A detailed look into the normalized confusion matrices revealed a consistent rise in correct predictions, notably for the first and third classes, across increasing AVT levels. An exception was observed for KL2, where correct predictions declined as AVT increased. On the other hand, the fourth and fifth classes exhibited stability in model performance with varying AVT levels after an initial improvement from Baseline to Low AVT.

Different trends surfaced when we focused on the recall values for individual classes (as presented in Table 2). The model’s proficiency in identifying instances of KL 0 consistently strengthened with increasing AVT, leading to the highest recall across all classes and conditions at High AVT (91.39%). Conversely, KL 1 experienced a peak in recall at Low AVT (36.82%), followed by a decrease at High AVT (14.86%), lower than its Baseline value. KL 2 also benefited from higher AVT levels, as indicated by a higher recall at High AVT (75.39%), despite slight decreases at Low and Mid AVT. The KL 3 class maintained a relatively stable recall across all AVT levels, peaking slightly at Low AVT (78.92%). Finally, the recall for KL 4 initially dropped from Baseline to Low AVT but then held relatively steady for Mid and High AVT. These recall trends underline the complex interplay between AVT levels and KL grade-specific performance.

The following section will focus on a different neural network model - the EfficientNetV2M. When looking at Table 3, it was clear that the dimensionality of the NAS input reduced dramaticaly as we moved from the Baseline to the High AVT level. Regarding model performance, Accuracy, and Precision peaked at the Mid AVT level, with Accuracy reaching 65.48% and Precision achieving 67.18%. However, Precision took an upward leap at High AVT, reaching 79.72%, the highest among all levels. In contrast, Recall and F1-score exhibited a slight downward trend after the Baseline, with the lowest values observed at the High AVT level - 58.99% and 58.35%, respectively. Based on Accuracy and Precision, the model performance seemed optimal at Mid AVT rather than High AVT, suggesting that extreme dimensionality reduction might not necessarily lead to better overall performance for this set-up.

When analyzing the confusion matrices (Figure 5), we observe an overall upward trend in the diagonal values, indicating an increase in correct predictions on a higher variance threshold (AVT). Notably, KL 0 and KL 2 experienced consistent improvement in accuracy. The performance, however, varied for other classes. While KL 1’s score declined post-Baseline and droped to zero at High AVT, KL 3 showed a slight inconsistency, and KL 4 displayed a fluctuating trend, peaking at Baseline and Mid AVT but falling at High AVT. The off-diagonal elements reveal changes in KL confusion across different AVT levels. Mainly, confusion between KL 0 and KL 1 decreased with an increasing AVT, while the confusion between KL 1 and KL 2 increased. The recall table 4 further supplements these findings.