Decorrelative Network Architecture for Robust Electrocardiogram Classification

Adversarial attacks are small input perturbations that do not change the semantic content yet cause massive errors in a network output; for example, imperceivable noise patterns that, when added to an image, cause a model misclassify the image [7]. Given a target model and input, projected gradient descent (PGD) is the most common algorithm for finding adversarial perturbations under an $\ell_{\infty}$ bound [8, 9]. Various other algorithms exist, including the use of generative adversarial ensembles [10, 11, 12]. Impressively, universal adversarial perturbations can be crafted to fool a network when added to any sample [13, 14].

The understanding of adversarial attacks has rapidly developed over the past several years. Akhtar and Mian wrote a broad survey on adversarial attacks in computer vision [15]. Although adversarial instability may relate to overfitting, DNNs often generalize well to unseen data yet fail on previously seen data that are only slightly altered [16]. Furthermore, it has been shown that linear models and other machine learning methods are also vulnerable to adversarial attacks [17]. Early research attributed this phenomenon to lack of data in high-dimensional problems, which leaves large portions of the total ‘data-manifold’ unstable [18, 19]. Literature also reported a relationship between large local Lipschitz constants (with regards to the loss function) and adversarial instability [20, 21, 22]. To our knowledge, the most unifying explanation is the robust features model, where it is shown that data distributions often exhibit statistical patterns that are meaningless to humans but correlate well with different classes [23]. From a human’s perspective, these patterns are arbitrary and easily perturbed, but since models are trained to only maximize distributional accuracy, they have no reason to prioritize human-favored features over these patterns.

Training models for defending against adversarial attacks remains an open problem, affecting nearly every application of machine learning. Early attempts at defense methods by obfuscating the loss gradient were found to beat only weak attackers, proving ineffective for sophisticated attackers [24, 25, 26, 27, 28]. Adversarial training, in which a model is iteratively trained on strong adversarial samples, has shown the best results in terms of adversarial robustness [8, 29]. However, the network size and computational time required is considerable for small problems, and improving adversarial robustness appears to sacrifice performance on clean data [30]. Satisfactory performance on larger problems has not been achieved due to these limitations.

Another troubling, well-documented characteristic of these attacks is their transferability: models trained on the same task will often be fooled by the same attacks, despite having different parameters [7, 17, 31]. This phenomenon is largely congruent with the robust features model, since these models are likely learning the same useful, but non-robust features. Nevertheless, transferability makes black-box attacks viable, where a malicious attacker does not need detailed knowledge of the target model.

Han, et al. has shown that models trained for ECG classification are concerningly susceptible to natural-looking adversarial attacks [32]. In short, the authors observed that traditional $\ell_{\infty}$-bounded PGD attacks produce square-wave artifacts that are not physiologically plausible in ECG; to rectify this, the perturbation space was modified by applying Gaussian smoothing kernels in the attack objective, rendering plausible yet still highly effective adversarial samples.

ECG is a front-line, noninvasive tool for monitoring heart health. Skin electrodes measure electrical signals originating from the heart; the behavior of these signals over time correspond to various events during the cardiac cycle. A healthy rhythm consists of a P wave, QRS complex, and T wave, which correspond to atrial depolarization, ventricular depolarization, and ventricular repolarization, respectively. Clinical ECGs have traditionally used 12 or 5-7 (Holter) leads, but single-channel ECGs have become more prevalent for continuous monitoring [33]. These devices are either external or implantable loop recorders (ILRs) designed to record for multiple years.

A variety of downstream analytical tasks are associated with ECG, including biometric identification, respiratory estimation, emotional monitoring, and even fetal heartbeat monitoring. However, the most common application is detection of various arrythmias, which can indicate disorders or disease risk [33]. Atrial fibrillation (AFib), or an abnormally rapid atrial firing rate, is commonly assessed (such as in the 2017 PhysioNet Challenge [4]), but many arrythmic classses exist, including left or right bundle branch block, premature atrial or ventricular contraction, ventricular fibrillation, tachycardia, and myocardial infarction (heart attack) itself. Certain classes require immediate and serious medical intervention while others, such as AFib, are not immediately harmful but could still indicate risk of disease. The clinical utility of AFib detection is still an active area of investigation: a systematic review found AFib to be associated with increased risk of myocardial infarction in patients without coronary heart disease and increased risk of all-cause mortality and heart failure in all patients, implying value in detecting it [34]. On the other hand, a randomized control trial using ILR in patients with at least one risk factor for stroke concluded that continuously monitoring for AFib in this population did not reduce risk of stroke [35]. This suggests that detecting AFib early may not provide additional information for managing stroke in patients that are already known to be at risk of the event. Nevertheless, ECG and its subsequent analysis is widely applied and investigated for its implications on patient cardiovascular health.

Clinician review is often required for ECG analysis; this process is resource-consuming, especially in the case of continuous monitoring or large patient samples. Thus, automatic classification of these signals is desirable, but simple rule-based classifications often fail to generalize due to data heterogeneity between patients and the non-stationary nature of the signal within patients. As such, researchers have turned to data driven techniques and machine learning to build ECG classification models. Convolutional neural networks (CNNs) have been the most dominant architecture for ECG arrythmia classification, but deep belief networks, recurrent neural networks, long short-term memory, and gated recurrent units have all been investigated for the same task [36]. For extensive background, Merdjanovska, et al. provides a comprehensive review of applications, public datasets, and deep learning research for ECG, and Ebrahimi, et al. further surveys common deep learning architectures for ECG [33, 36].

As misdiagnosis in healthcare contexts can cause serious harm, the standard of trust required for AI to operate in this space is high. Rather than replacing clinicians, we envision AI tools augmenting clinical workflows by monitoring inputs over a large population, flagging alarming or low-confidence instances for human observation. Figure 1 broadly illustrates this scenario, where AI could allow a few experts to analyze ECG signals from a large patient population, continuously or transiently monitored. To achieve this synergy, models must be capable of gauging their own confidence, recognizing conditions where they can and cannot perform well [37]. Bayesian deep learning (BDL) is a promising field that models the parameters of a DNN as a distribution rather than a point estimation; sampling this distribution at inference time then allows one to estimate model certainty in an inference [38, 39]. Approaches for approximating and sampling the parameter distribution, including variational inference and Markov Chain Monte Carlo with Hamiltonian Dynamics, are often difficult to scale to large spaces [40, 41]. One simple approach is to train an ensemble of networks for the same task, with each network acting as a sample of the parameter space [42]. However, this approach does not guarantee robustness: adversarial attacks in particular are known to transfer between different models because these models (even with vastly different parameters) often learn the same unstable features. Furthermore, in high dimensional problems with large parameter spaces, training, storing, and running inferences from numerous models quickly becomes infeasible. As such, the goal in training such ensembles should be to achieve adequate robustness and feature diversity with a small number of models.

To our knowledge, prior work on adversarial robustness has primarily quantified either white-box accuracy or black-box transferability, but have not evaluated uncertainty via BDL. Furthermore, works in this field primarily test methods on lower dimension datasets, such as MNIST or CIFAR10. Our goal is to efficiently train small but diverse deep ensembles capable of gauging uncertainty in worst-case scenarios, i.e., adversarial attacks. We contextualize this in the aforementioned ECG classification, a problem that is much higher in dimension. Adversarial training as a means of diversifying an ensemble is explored, and we also introduce two novel diversification methods that do not require adversarial sample computations, adding almost no overhead to the regular training process.

According to the robust features model, simply training networks with different parameters in isolation does not achieve adversarial robustness, as networks trained under the same conditions tend to converge toward the same learned features and vulnerabilities [23, 43]. As such, rather than achieving diversity in the parameter space, we turn the conversation to diversity in the feature space. A mechanism for incentivizing networks to learn different features is necessary. To this end, Yang, et al. conceived DVERGE, which diversifies the learned features and adversarial weaknesses in a classification ensemble [44]. However, this method requires full or partial computations of adversarial samples and round-robin style training of networks, which adds considerable compute. In this work, we experiment with ensemble diversification methods that are based on adversarial gradients and other methods that are agnostic to these calculations.

Two ECG datasets are used for all experiments: the 2017 PhysioNet challenge data (single-channel, four classes) [4] and the 2018 China Physiological Signal Challenge (CPSC) data (twelve-channel, nine classes) [5]. The following ensemble training strategies were tested:

• •

  baseline: Conventionally trained ensemble, where each model is identically and independently trained.

• •

  dec: Ensemble trained with the proposed linear feature decorrelation to diversify the model features.

• •

  part: Ensemble trained using the proposed Fourier partitioning scheme.

• •

  adv: A baseline ensemble that undergoes additional ensemble adversarial training.

• •

  dec+part: Ensemble that employs both linear feature decorrelation and Fourier partitioning.

• •

  dec+adv: A decorrelated ensemble that undergoes additional ensemble adversarial training.

• •

  dverge: A baseline ensemble that undergoes additional DVERGE training.

Ensembles produce multiple inferences, which can be processed in various ways to gauge epistemic and aleatoric uncertainty[45, 46]. Here, we adopt a normalized uncertainty approach from [47], which calculates a normalized measure of certainty $I_{norm}$ based on the mutual information between the sample and the model parameters (see Methods: Ensemble Training and Inference).

We test each ensemble using validation data perturbed by both PGD and physiologically feasible SAP attacks of varying magnitude $\varepsilon$, targeting the first model in each ensemble [32]. Figure 2 displays several example attacks from the PhysioNet 2017 data along with inferences, probability, and uncertainty values outputted by the baseline, dec, part, and dec+part ensembles.

Notably, we found that implementing the decorrelation step only slightly increased training time: conventional training took about 352 and 358 minutes per model on average for PhysioNet and CPSC, respectively; decorrelation only added about 10 minutes on average to this time in both cases (see Methods: Experimental Details for training parameters). Fourier partitioning did not noticeably increase training time.

To classify an ensemble inference as either certain or uncertain, a threshold $I_{T}\in[0,1]$, can be applied to differentiate certain $I_{norm}\leq I_{T}$ and uncertain $I_{norm}>I_{T}$ predictions. A robust model is generally correct when it is certain and uncertain when it is incorrect. Thus, in addition to inference accuracy, we also adopt the following three evaluation metrics from [47]:

• •

  Correct-certain ratio $R_{cc}(I_{T})=P_{I_{T}}(correct|certain)$: Probability the model inference is correct when it is certain.

• •

  Incorrect-uncertain ratio $R_{iu}(I_{T})=P_{I_{T}}(uncertain|incorrect)$: Probability the model is uncertain when it is incorrect.

• •

  Uncertainty Accuracy $UA(I_{T})=P_{I_{T}}(correct\cap certain\bigcup incorrect\cap uncertain)$: Probability of a desired outcome (either correct and certain of uncertain and incorrect).

All three measures depend on the variable uncertainty threshold $I_{T}$. Thus, similar to a binary classifier, the overall efficacy can be found by integrating the measure as a function of $I_{T}\in[0,1]$ (i.e., finding the area under the curve, where larger values are more desirable).

Table 1 summarizes the average prediction accuracy and areas under the curve (AUCs) for the correct-certain ratio, incorrect-uncertain ratio, and uncertainty accuracy for natural, PGD, and SAP adversarial datasets for the PhysioNet 2017 experiments. Similarly, Table 2 reports the same metrics for the CPSC 2018 experiments. Some initial observations from these numbers are as follows: 1) For the PhysioNet 2017 experiments, dec, part, and dec+part generally have comparable or superior metrics to the baseline on natural ($\varepsilon=0$) samples, but achieve better performance on stronger $\varepsilon=50,75,100$ PGD and SAP attacks. Adversarial training (adv) improves performance on all adversarial attacks, but the combination of dec+adv generally leads to better performance in these instances, while dverge does not seem to improve from the baseline in this instance. 2) On the CPSC 2018 experiments, adv and dec+adv achieve improved robustness on perturbed data but sacrifice considerable accuracy on natural samples. dverge experiences a mild reduction in natural performance for a moderate performance increase on perturbed data, although this advantage diminishes with higher magnitude attacks. The combination of dec+part poses better performance on perturbed data relative to the baseline without sacrificing performance on natural samples.

Since uncertainty $I_{norm}$ should be a useful discriminative feature for distinguishing correct and incorrect predictions, it is desirable that incorrect predictions, on average, output higher uncertainty than correct predictions. Thus, we simply define the average difference in uncertainty between incorrectly and correctly classified samples:

Figure 3 compares this $\Delta I_{norm}$ for all experiments and ensembles as a function of attack strength $\varepsilon$. In all instances, $\Delta I_{norm}$ decreases sharply for the baseline ensemble as perturbation strength increases. dverge follows this same trend on the PhysioNet data but maintains better robustness on the CPSC data. part, dec, dec+part, and dec+adv generally maintain higher $\Delta I_{norm}$ on perturbed data in both cases.

In a clinical setting, deep models should be used to augment clinical workflows, as shown in Figure 1. When the amount of data needing analysis outstrips the available time of qualified clinicians, deep ensembles can initially assess all inputs and defer the most uncertain samples to human readers. This begs the question: If a deep ensemble is budgeted a certain number of cases that it can refer to human experts, then how many cases would still be misclassified? To investigate this, we run the following hypothetical experiment: 1) A dataset of all natural samples and a partially perturbed dataset are drawn. In the partially perturbed dataset, only 50% of the data are unperturbed, and 25%, 15%, and 10%, of the data have $\varepsilon=10,50,75$ and $\varepsilon=0.025,0.05,0.15$ perturbations for the PhysioNet and CPSC data, respectively. 2) Each ensemble evaluates all samples, ordering the inferences from most to least confident. 3) Starting with the most confident samples, varying amounts of samples would defer to the deep model’s classification, while all other (less confident) samples would be correctly classified, presumably reviewed by clinicians. Note that this experiment is not meant to exactly reflect the actions and metrics of a true clinical workflow; rather, it is to investigate and compare the potential benefit of the aforementioned deep ensemble methods in augmenting human workers, particularly in the face of adversarially corrupted data.

Figures 4 and 5 plot the percentage of misclassified instances in the sample as a function of the percentage of cases referred to the deep ensemble for the natural and partially perturbed datasets, respectively. Additionally, the total area-under-curves are shown, with lower values being better in this instance. Initially, one can see that adv and adv+dec perform poorly on the natural datasets, particularly the CPSC 2018 data. dec, part, and dec+part perform better than the baseline on the partially perturbed datasets while still performing comparably or better than the baseline on the natural PhysioNet 2017 dataset.

Basic ensembles consist of multiple deep neural networks trained for the same task. An ensemble inference on sample $x$ is simply the average output of each model in the ensemble. Thus, for an ensemble with models $f_{1},f_{2},...f_{K}$:

Note that for a classification task, this output is a discrete probability distribution.

For estimating epistemic uncertainty, we adopt the approach from [47], which defines the uncertainty of sample $x$ as the mutual information between the inferred label $\hat{y}$ and the underlying parameter distribution. In other words, how much additional information sample $x$ tells us about the true parameters:

$\mathcal{D}$ is the training data. The first term is intractable, but can be estimated using the network ensemble as the entropy of the expected inference [47]. Thus, for ensemble with $f_{1},f_{2},...f_{K}$ models, each of which output a discrete probability distribution over $C$ classes:

The scale of $I$ is relative and can vary between models and ensembles. Thus, we normalize the uncertainty with the minimum and maximum uncertainty values found during training.

Note that test samples can have greater or less uncertainty than any sample encountered in the training set. Thus, values for $I_{norm}$ are not necessarily limited to $[0,1]$. For a threshold $I_{T}$ which classifies samples as either ’certain’ or ’uncertain’, metrics $R_{cc}$, $R_{iu}$, and $UA$ were calculated empirically over an adversarial dataset based on the number of correct & certain, correct & uncertain, incorrect & certain, and incorrect & uncertain samples.

For the baseline ensemble, each model was trained independently in sequence. Other ensembles were trained using the methods described below.

With the Fourier partitioning scheme, the first ensemble model has no modification. The other models were trained normally but inputs were filtered during both training and inference (Figure 7). This approach is inspired by [30], which showed that neural networks can achieve high classification accuracy in many computer vision tasks with only a portion of an input’s frequency data, and that models often overfit to discriminative features in specific frequency bands.

In practice, filter convolution was done by pointwise multiplication in the Fourier domain, computed using the fast Fourier transform.

Adversarial attacks are formulated by maximizing the loss objective $L$ of the model f by modifying $x$ (with the paired label $y$) within a set of valid perturbations $\Delta$:

Two algorithms were used to craft adversarial attacks: projected gradient descent (PGD) and smoothed adversarial perturbations (SAP). PGD is widely used as a strong attack with an $\ell_{\infty}$ bound through iterative optimization [8]:

where $x^{\prime}_{i}$ is the sample at the $i^{th}$ iteration, $y$ is the corresponding label, $\alpha$ is a step size, $L$ is the loss function, and the clipping operation clips all values to be within the $\ell_{\infty}$ ball of radius $\varepsilon$ around $x$, as well as any implicit bounds on the domain of $X$.

SAP is a variation of PGD designed to craft smooth attacks for ECG signals. The details of SAP can be found in [32]. To summarize, perturbation $\theta$ is iteratively optimized in a fashion similar to PGD, but is also convolved with a sequence of $M$ Gaussian kernels at every step, each of which are parameterized by their width s and standard deviation $\sigma$:

The convolution with Gaussian kernels smooths high frequency perturbations, removing unrealistic square wave artifacts. PGD and SAP attacks were optimized over 20 steps in total. For both attacks, $\alpha$ was scaled as $\varepsilon/10$. All adversarial attacks were crafted from the validation to target one of the models in an ensemble. For experiments with PhysioNet 2017 data, the convolution kernels are identical to those used in [32]: $s=(5,7,11,15,19),\sigma=(1,3,5,7,10)$. For the CPSC 2018 experiments, we used $s=(9,11,15,19,21),\sigma=(5,7,10,13,17)$.

The two ECG datasets used in this work are from the PhysioNet 2017 and CPSC 2018 challenges. The PhysioNet dataset contains 30-60 second duration single channel ECG recordings sampled at 300Hz. All samples were were zero-padded to be 60 seconds. Class labels are Normal, A. Fib., Other Rhythm, Noise. Scaling of the signal magnitudes were identical to those in [32]. The CPSC 2018 data contains 6-60 second ECG recordings at 500Hz with nine class labels: Normal, A. Fib, 1^st-Degree Atrioventricular Block, Left Bundle Branch Block, Right Bundle Branch Block, Premature Atrial Contractions, Premature Ventricular Contraction, ST-segment depression, ST-segment elevated. All samples were truncated or zero-padded to be 48 seconds. For simplicity, the minority of samples labelled with multiple diagnoses were not used. Additionally, all channels were normalized from -1 to 1.

For all experiments, each ensemble consisted of $K=3$ classification networks, each with the architecture in used [6] for experiments using the PhysioNet 2017 data. For experiments using the CPSC 2018 data, this architecture was simply modified to have 12 input channels and 9 output classes. Each network was trained for 80 epochs (batch size of 64) using the Adam optimizer with a learning rate of $10^{-3}$. Pytorch 1.8.1 was used with two NVIDIA Titan RTX GPUs, and a 90/10 training/validation split. For all ensembles using decorrelation training, hyperparameters $r=32$ and $\lambda=0.2$ were used (the uncompressed latent dimension $D$ was 64).

For ensembles that underwent additional ensemble adversarial training, each model in the ensemble was sequentially trained using adversarial samples. In practice, this is identical to regular training, except each sample batch is perturbed using PGD (Equation 6) prior to the forward training step. The perturbations target the model being trained, and use $\varepsilon=10$ and $0.025$ for the PhysioNet 2017 and CPSC 2018, respectively. Each model is trained for an additional two hours, translating to six extra training hours total for each adversarially trained ensemble.

DVERGE training is similar to adversarial ensemble training, in that both use PGD to perturb a sample $x_{s}$ within a small $\ell_{\infty}$ bound. Differences are that in DVERGE 1) this optimization is used to maxmize similarity between the distilled features of $x_{s}$ and some other randomly drawn sample $x$ at some randomly selected feature layer $l$ of the network, and 2) the samples perturbed using network $i$ are used to train other networks $i\neq j$. Thus, we use the feature distillation objective and training procedure from [44]:

Values for $\varepsilon$ were identical to those used in adversarial ensemble training. The DVERGE training also ran for the same total training time (6 additional hours) as adversarial ensemble training. For each batch, feature layer $l$ was randomly, uniformly selected from the post batch-norm layers of the networks.