Learning Clinical Concepts for
Predicting Risk of Progression to Severe COVID-19

We use retrospective observational data collected by a major healthcare provider in Southwestern Pennsylvania from January 1st, 2020 to January 12th, 2022. Out of 171,009 patients who were tested for COVID-19, we extract the 40,190 who tested positive. Of those, we remove individuals who were already mechanically ventilated or admitted to the ICU within 30 days prior to the time $t_{0}$ of testing positive for the first time. This leaves a cohort of 31,336 individuals (Table 1). Note that this study seeks to predict the risk of progressing to severe COVID-19 upon testing positive for the first time, and so features and outcomes are defined relative to time $t_{0}$.

Features are extracted no later than the date of each patient’s first covid positive test. These include testing location (inpatient/outpatient), demographics, labs, medications, vaccines, symptoms, and problem history. The most recent value of each feature is extracted, and symptoms are limited to a one-day window around $t_{0}$. Since there are tens of thousands of distinct medications, labs, diagnoses, vaccines, etc. in our data, the feature pool is limited to the top 20 of each data type except for labs (top 50). Upon clinician review, 45 more features are extracted. After removing low-variance features, converting categorical values to indicators, and normalizing continuous values, this yields a fixed-length 139-dimensional feature vector (see acmilab.org/severe_covid) for patient information known at $t_{0}$.

Since patients are right-censored upon leaving the hospital system, the outcome of interest is a time-to-event, where the time is computed as the time elapsed between $t_{0}$ and severe COVID-19 (mechanical ventilation, ICU admission, or death) or censorship (when the patient was last seen in hospital records), whichever is first.

To learn these concepts, we use the “anchor-and-learn” framework.[9] For each concept of interest, we identify some key informative observations (“positive anchors”) relating to that concept. In this work, we only consider binary-valued concepts (present vs. not present). An observation is an anchor for a concept if it is conditionally independent of all other observations conditioned on the concept. When the presence of an anchor almost certainly implies the presence of the concept, this is known as a positive anchor.

Consider a patient with covariates $x\in\mathbb{R}^{d}$. Suppose we want to extract a concept $c$ with positive anchor $x_{c}\in\{0,1\}$ (e.g. extracting a diabetes concept with a diabetes diagnosis code as a positive anchor). Let $y_{c}\in\{0,1\}$ be the true binary label for whether concept $c$ is present. Note that in most observational health data, we observe the presence of a clinical condition, but not the absence of it. For example, when extracting the diabetes concept, we can be fairly confident that a patient marked as diabetic does indeed have diabetes, but patients unmarked do not necessarily not have diabetes. Said differently, we have positive and unlabeled (PU) data rather than positive and negative data. Since only positive examples are labeled, $y_{c}=1$ is certain when $x_{c}=1$, but when $x_{c}=0$, then $y_{c}$ could be either 0 or 1.

Thus, we leverage algorithms designed to learn from positive and unlabeled data, or “PU learning” algorithms. Let $x_{\bar{c}}$ refer to all covariates except for $x_{c}$. Since anchors are conditionally independent of all other observations conditioned on the concept, we have that $p(x_{c}|y_{c}=1)=p(x_{c}|y_{c}=1,x_{\bar{c}})$. Now, consider $p(x_{c}=1|x_{\bar{c}})$. We have that:

where $\delta_{c}=p(x_{c}=1|y_{c}=1)$. The first equality follows from the fact that $y_{c}=1$ is certain when $x_{c}=1$, and the second equality follows from Bayes rule. In words, the expression indicates that true probability of the concept being present is proportional to the probability of the positive anchor being present by a factor of $\delta_{c}$. Thus, if we can train a PU classifier $g(x_{\bar{c}})=p(x_{c}=1|x_{\bar{c}})$ that learns the probability that a positive anchor is present given the remaining covariates, we need only scale the probability by $\delta_{c}$ in order to get the probability of the underlying concept being present. As noted in Elkan and Noto,[6] for the set $P$ of positive labeled examples, one can construct an empirical estimate of the constant $\delta_{c}$ as $\hat{\delta}_{c}=\frac{1}{n}\sum_{x_{\bar{c}}\in P}g(x_{\bar{c}})$ , due to the observation that $g(x_{\bar{c}})=\delta_{c}$ for $x_{\bar{c}}\in P$:

Finally, this yields the following procedure for learning clinical concepts:

1. 1.

   Identify clinical concepts of interest, and corresponding positive anchors.

2. 2.

   Learn a positive vs. unlabeled classifier. Use logistic regression to learn a classifier $g(x_{\bar{c}})$ that outputs the probability of the positive anchor given the other covariates.

3. 3.

   Estimate the scaling constant. On a validation set, estimate $\hat{\delta}_{c}$ by averaging the output of $g(x_{\bar{c}})$ on all positive labeled examples (i.e. examples with the positive anchor).

4. 4.

   Scale predictions from the PU classifier by the estimated scaling constant to get the probability that the underlying concept is present. That is, compute $p(y_{c}=1|x_{\bar{c}})=g(x_{\bar{c}})/\hat{\delta}_{c}$ for all examples where the positive anchor is not present. If the positive anchor is present, leave the probability as 1.

This procedure is also used in Halpern et. al.,[9] except instead of drawing the concept from a Bernoulli distribution parameterized by $p(y_{c}=1|x_{\bar{c}}$), we directly use the computed probability $p(y_{c}=1|x_{\bar{c}})$ since it can provide more granular information. The scikit-learn[14] python package was utilized for its logistic regression implementation.

In order to define clinical concepts of interest, we surveyed several clinicians in the healthcare provider network about the main concepts they would look for when assessing risk of severe COVID-19. The survey yielded 21 concepts: old age, inpatient, outpatient, diabetes, shortness of breath, fever, cough, fatigue, COVID-19 vaccination, flu vaccination, obesity, hypertension, immunocompromised, COPD, congestive heart failure, chronic kidney disease, hyperglycemia, transplant, cancer, lung disease, and myalgia. We identify positive anchors for these concepts (precise definitions at acmilab.org/severe_covid) and apply the PU algorithm to extract a more complete representation of the concepts. Learning Severe COVID-19 Risk Using the lifelines python package,[4] a Cox proportional hazards model with L1 regularization (Lasso-Cox) is used to model risk of progression to severe COVID-19. For a patient with covariates $X$, their hazard $h$ at time $t$ is given by:

where $h_{0}$ is a baseline hazard function, and $\beta$ are learned coefficients. The regularization penalty is given by $\lambda||\beta||_{1}$, where regularization strength $\lambda$ is selected using 5-fold cross validation and grid search over penalties between 0 and 0.2, with a step size of 0.001 between each penalty. For stability of training, features with variance $<0.01$ are removed.

In order to explore the marginal effect of incorporating learned concepts versus the original set of 139 features, we analyze and evaluate Lasso-Cox models learned from five different sets of features:

1. 1.

   Raw positive anchors: only the positive anchors identified in the data, without learning the corresponding clinical concepts (e.g. mention of ”diabetes” in a note, ICD code for diabetes, etc.)

2. 2.

   Learned concepts (LC): only the learned clinical concepts (e.g. the diabetes concept)

3. 3.

   LC + Numeric: the learned clinical concepts and numerical features (e.g. diabetes concept and labs)

4. 4.

   LC + All Features: the learned concepts, as well as all of the original 139 features

5. 5.

   All Features: all 139 original features, no learned concepts.

In real-world settings, hospital systems may want to use updated data to revise their models. To emulate this process, we re-train models (including PU concept classifiers) up to the end of each 3-month season (spring, summer, fall, winter), and evaluate their performance on subsequent seasons. Spring is March 20th until June 21st, followed by summer until September 22nd, followed by fall until December 21st, followed by winter until March 20th of the following year. For each 3-month period, a 70-30 split designates train and test sets, where test data is never included in any model training. To keep the risk score interpretation simple, for each time period a grid search on the Lasso-Cox penalty is done to choose a model with approximately ten features. We additionally train models on the entire study time range, with train and test sets that aggregate the respective 3-month datasets.

Since the concepts are only positively labeled or unlabeled, it is not possible to compute precision of the concept classifiers.[2] However, we can compute recall as the proportion of known positives recovered by the classifiers. Additionally, we examine the number of previously unlabeled samples predicted to be positive.

The Lasso-Cox model coefficients are in terms of original features as well as learned concepts. In addition to listing the Lasso-Cox coefficients, we create an interactive Sankey diagram to visualize how raw features translate into concepts, and how the resulting models pull from both. This gives the user a birds-eye view of how each concept is defined, the strength and sign of the coefficients, and which concepts are used in different models.

The concordance, or C-index, is used to evaluate the model’s discriminative ability. To evaluate calibration, both one-calibration at 14 days and D-calibration are used.[8] Additionally, low, medium, and high-risk strata are defined and their 14-day Kaplan-Meier survival curves are inspected.

We compare our model performance to that of the Covichem[1] and Galloway[7] risk scores. For Covichem, in order to conduct a fairer comparison than directly applying their logistic regression coefficients learned on a different population, we extract the same features and re-train logistic regression on our own training data. For Galloway, we extract all but one of their twelve features (radiological severity is not available in our data), and compare performance against two versions of their model: (1) directly applying their proposed count-based risk score (Galloway count), and (2) re-training a logistic regression model using the twelve variables (Galloway reweighted).

The strongest coefficients for each concept classifier are often not features that immediately come to mind, but nevertheless match clinical intuition. For the old age concept, we observe that a high PF flu vaccine has a large coefficient (3.31), and is a vaccine only given to patients 65 and older. The outpatient shortness of breath symptom (3.23) is the second highest coefficient for the inpatient concept, possibly indicating that outpatients with this symptom are at high risk of becoming an inpatient. The shortness of breath concept depends on dexamethasone (0.75) which relieves inflammation, and albuterol sulfate (0.67), prescribed for lung conditions. For the obesity concept, sleep apnea, which is often caused by excess weight, has the largest coefficient (0.96).

Note, however, that the learned concepts may not be a perfect representation of the underlying concept. None of the concept classifiers perfectly recover the originally known positives, with recall ranging from 0.381 to 0.974 (Table 2). This could be due to insufficient signal in the remaining covariates or underfitting due to the simplicity of the logistic regression model class. Additionally, some concepts learn substantially more positives than were available in the original data. For example, obesity originally has 433 positives in the data but the concept classifier marks 2,157 patients as having obesity with probability greater than 0.5. It is difficult to verify the faithfulness of the concept classifiers to the true concepts without manual review, but it is possible that the learned concepts may mark patients as “obesity-like” based on their other covariates rather than learning whether they truly have obesity. Additionally, while learned concepts are amenable to interpretation through the coefficients of the concept classifier, they still require domain expertise to manually define positive anchor variables. Finally, the conditional independence assumptions of the selected anchors may not hold in practice, and these assumptions are difficult to verify.

The coefficients for the survival models trained on LCs, All Features, and LC + All Features are mostly consistent with clinical intuition. Inpatients are more likely to experience adverse outcomes than those not hospitalized, old age is well-documented to be associated with higher COVID-19 death rates,[5, 13, 7] shortness of breath indicates respiratory involvement, and medications such as dexamethasone, acetaminophen, and intravenous saline are given to hospitalized patients. Higher BUN indicates worse liver and kidney function, and COVID-19 vaccines are designed to protect against severe COVID-19. While it is surprising that some of the learned concepts for fever and cough symptoms have negative HRs, upon inspection we find that these are most reliably recorded for outpatients and may encode some additional information about outpatient status. From exploring the interactive visualization, some of the features selected by the All Features model are used by LCs in the LC + All Features model, possibly indicating that they serve as proxies for higher level concepts. For example, the saline IV bolus, present only in the All Features model, is used in the inpatient concept classifier with a positive coefficient. We also note that the coefficients are likely shrunken towards zero due to the Lasso penalty, and the non-informative or independent censoring assumption of the Cox proportional hazards model may not hold since censoring occurs upon discharge.

Our Lasso-Cox models all outperform the baselines (Covichem, Galloway count, Galloway reweighted) in terms of aggregate, inpatient, and outpatient concordance (Table 3). As measured by aggregate concordance, we observe that the learned concepts provide a boost in performance over the raw positive anchors (C-index 0.858 vs. 0.844). This boost in performance places the LC model approximately halfway between the performance of the raw positive anchors and the LC + All Features and All Features models, which both achieve a C-index of 0.872. For LC + All Features and All Features models, the C-index on the outpatient subpopulation (0.879 and 0.880) is higher than that on the entire cohort, whereas the C-index on the inpatient subpopulation (0.715 and 0.717) is lower. For all remaining models, the performance on the inpatient and outpatient subpopulations is lower than in aggregate, possibly indicating that it is easy to order the relative risks of inpatients versus outpatients. The LC model appears slightly better calibrated than the LC + All Features model, but when used to stratify patients into high, medium, and low-risk strata, both models yield groups with clear separation between their survival curves. Finally, while there is some loss of discriminative performance going from the All Features models to the LC only model, when tested under the back-testing framework this gap seems to close quickly on subsequent time periods and the LC model even eventually surpasses the performance of the All Features model. Thus, models with LCs might be more resilient over time than models learned only from All Features. If the set of important high-level concepts themselves change over time, however, new concepts may need to be learned accordingly.

In the future, we plan to integrate our models with the healthcare system, and to continue to monitor the performance of the models over time. It will be important to further study how high-level clinical concepts perform across different settings, and to quantify the extent to which classifiers learned on top of these concepts might be transferable across hospitals. As COVID-19 continues to evolve over time, we will also investigate whether new concepts become relevant for prediction.