COVIDx CXR-3: A Large-Scale, Open-Source Benchmark Dataset of Chest X-ray Images for Computer-Aided COVID-19 Diagnostics

As the COVID-19 pandemic continues to progress internationally, the push for effective and robust screening methods is imperative. One method that has seen significant increase in usage in the COVID-19 clinical workflow is chest X-ray (CXR) imaging, which offers complementary screening information to transcriptase-polymerase chain reaction (RT-PCR) testing with the benefits of more readily accessible and mobile healthcare equipment [1]. In addition, CXR imaging is routinely conducted in parallel to tedious viral testing for patients with respiratory complaint [2] to reduce patient volume and provide more clinical detail. As such, multiple works have investigated deep learning-driven computer-aided diagnostics to both detect positive SARS-CoV-2 cases from CXR images [3, 4, 5] as well as provide severity assessments that can be leveraged in patient triaging [6, 7, 4]. Nevertheless, as a result of the rapid progression of the pandemic and drive for results, such studies have been limited in terms of quantity and/or diversity of patients as well as standardization between datasets for higher public accessibility. In this work, we introduce COVIDx CXR-3, a large-scale benchmark dataset of CXR images that has been carefully curated with expert radiologists and clinicians for computer-aided COVID-19 research comprising of 30,386 CXR images from a multi-institutional, multinational cohort of 17,026 patients across at least 51 countries. To the best of the authors’ knowledge, COVIDx CXR-3 is the most extensive, accessible COVID-19 CXR dataset in open-source form. Additionally, we examine the dataset diversity in regards to patient findings, demographics, and imaging view, and explore areas of potential bias and dataset imbalances.

The COVIDx CXR-3 benchmark dataset comprises carefully curated CXR images from a multi-institutional, multinational patient cohort, in collaboration with expert radiologists and clinicians. More specifically, COVIDx CXR-3 comprises patient cohorts from the following organizations and initiatives: (1) RSNA International COVID-19 Open Radiology Database (RICORD) [8], (2) Stony Brook University (COVID-19-NY-SBU) [9], (3) Valencian Region Medical Image Bank (BIMCV) [10], (4) RSNA Pneumonia Detection Challenge dataset [11], (5) COVID-19 Image Data Collection [12], (6) Fig.1 COVID-19 CXR Dataset Initiative [13], (7) ActualMed COVID-19 CXR Dataset Initiative [14], and (8) COVID-19 Radiography Database [15]. Expert feedback was used to evaluate the quality and applicability of the images from each source, leading to removal of some images from the listed sources. A test set that has no patient overlap with the training set was created from randomly sampled patients from the Radiological Society of North America (RSNA) RICORD [8] and Pneumonia Detection [11] initiatives as a result of their high image and annotation quality. Each collected patient case is documented with one of the following findings: (A) normal or no pneumonia and no SARS-CoV-2 infection, (B) non-SARS-CoV-2 pneumonia infection, and (C) positive SARS-CoV-2 (COVID-19) infection. Example images from COVIDx CXR-3 for each infection case are shown in Figure 1.

Table 1 shows the distribution of CXR images and patients in regard to infection type. The final COVIDx CXR-3 benchmark dataset is composed of 30,386 CXR images from 17,026 unique patients, of which 16,194 CXR images (53.3%) and 2,986 patients (17.5%) come from SARS-CoV-2 positive cases, creating a relatively balanced training and testing set for SARS-CoV-2 positive and negative detection in terms of image count. The smaller balanced test set is the result of sampling an 80%/20% patient split from the RSNA RICORD [8] initiative for positive SARS-CoV-2 cases to ensure networks are evaluated against expertly annotated positive samples.

The distribution of demographic patient data for age and sex as well as for CXR imaging view is shown across all infection types in Table 2 and for SARS-CoV-2 positive cases only in Figure  2. The dataset does not indicate a bias for youth or senior vulnerable age groups with the majority of patients (66.2%) falling within the range of 18 to 59. Moreover, in terms of patients with documented sex, there is a relatively similar fraction of male (53.2%/37.7%) and female (40.1%/27.5%) patients across all infections as well as across SARS-CoV-2 positive cases respectively. In terms of imaging view, a majority of the CXR images are captured in anterior-posterior (AP) (67%) view as opposed to posterior-anterior (PA) (28.1%) view.

Systems trained on the COVIDx CXR-3 benchmark dataset have the opportunity to detect each individual documented infection class or group normal and pneumonia cases together to form a negative SARS-CoV-2 label for targeted SARS-CoV-2 screening. The hope is that the release of COVIDx CXR-3 in an open-source manner will help encourage researchers, clinical scientists, and citizen data scientists to accelerate advancements and innovations in the fight against the pandemic.

Finally, to serve as a baseline reference for comparison purposes, Table 3 provides benchmark test performance for three different deep neural network architecture designs on the COVIDx CXR-3 test dataset, as conducted in [4, 5]. We hope that COVIDx CXR-3 can assist scientists in advancing research against both the COVID-19 pandemic and related diseases.

While the aim with this large-scale benchmark dataset release is to support researchers, clinicians, and citizen data scientists in advancing research, one negative societal impact that can potential arise from this release the misuse of the collected data. For example, the benchmark dataset may be utilized by insurance companies to construct machine learning algorithms for forecasting future medical expenses, and as such lead to increased insurance premiums by insurance companies that could be inappropriate for a given patient given imperfect predictive analytics.

We would like to thank the Canada Research Chairs program and the the Natural Sciences and Engineering Research Council of Canada (NSERC).