Statistical analyses of ordinal outcomes in randomised controlled trials: protocol for a scoping review

Randomised controlled trials (RCTs) typically aim to estimate the average causal effect of one intervention relative to at least one other intervention. The average causal effect can take many forms, such as a risk or odds ratio (binary outcome) or difference in means (continuous outcome). The outcome of interest in a trial may be nominal, ordinal, interval or a ratio [1]. Nominal outcomes refer to outcomes that are categorical and unranked, such as the blood type of a patient. If the outcome is measured on an interval scale, the outcome can be categorised, ranked, and the difference between any two proximate values are equally spaced (e.g. body temperature). In addition to these properties, the ratio scale has a true zero point (e.g. weight). Ordinal scales can be considered to inhabit the space between nominal and interval/ratio scales; they are categorised and ranked, but the distance between any two categories is not necessarily meaningfully quantifiable or equally spaced. For example, a change from a disease-free state to hospitalisation would not be considered to be equivalent to a change from hospitalisation to death. An ordinal outcome is therefore a variable that measures a single characteristic that comprises of multiple, monotonically ordered categories that are not separated by a meaningful distance [2]. The categories should also be mutually exclusive (such that the categories are non-overlapping), detect improvement and deterioration, and be unambiguously defined (so that categories can be clearly distinguished from each other) [3]. An additional condition of an ordinal outcome, if it measures a change between two points in time, is that the scale should be symmetrical in structure to avoid bias [3]. That is, there should be an equal number of categories that represent both improvement and deterioration.

Ordinal outcomes have become increasingly common in trials, particularly during the COVID-19 pandemic. For example, the World Health Organisation (WHO) developed the WHO Clinical Progression Scale [4], an ordinal scale that describes disease severity of COVID-19 that has been adapted in various treatment trials [5, 6]. The categories reflect patient states that include being uninfected with COVID-19, ambulatory mild disease (asymptomatic or symptomatic), moderate disease (defined by patient hospitalisation and whether oxygen therapy is required or not), severe disease (ranging from a hospitalised patient who requires oxygen by non-invasive ventilation or high flow, to patients requiring mechanical ventilation with any of vasopressors, dialysis or extracorporeal membrane oxygenation for treatment), and death [4]. A review of clinical trials on the management of COVID-19 found that over half of the trials that evaluated disease severity and progression used an ordinal outcome [7], where the majority were used as secondary outcomes. Ordinal outcomes are also commonly used in stroke trials that often use the modified Rankin scale, a measure of the degree of disability among individuals who have suffered a stroke, as an outcome of interest [8, 9, 10, 11].

Although it can be easier to interpret a clinically important effect using a dichotomised or continuous outcome, various disease states measured on an ordinal scale can represent meaningful distinctions that may be of clinical importance. Ordinal outcomes are also appealing as they can retain information and increase statistical power compared to dichotomised outcomes, allowing a smaller sample size to be used [12]. They can also answer important clinical questions regarding specific patient states that cannot be answered using continuous outcomes, and can allow multiple clinical outcomes to be comprised in a single endpoint. Although there are advantages to using ordinal outcomes, the required analyses can be complicated and important considerations need to be made in the design phase of the trial. For example, the number of categories in the ordinal scale (fewer categories may reduce power and increase the sample size needed to detect an effect [13]), and explicitly defining an appropriate target parameter to compare the intervention groups.

There are a number of different target parameters that may be used to compare interventions with an ordinal outcome. For example, one could use an odds ratio that is assumed to be constant across all of the dichotomisations of the ordinal scale, known as the proportional odds assumption. Such a statistic can be estimated using the cumulative logit model, commonly referred to as the ‘proportional odds model’ [14]. Alternatively, the target parameters of interest might be odds ratios that use a baseline category as the reference level that can be analysed using multinomial regression. Ordinal outcomes can also be dichotomised for analysis, with the target parameter of interest possibly being the difference in proportions between the intervention groups which can be estimated using binomial regression. Finally, the ordinal outcome can be treated as continuous data with the target parameter of interest being a difference in means, estimated using a linear regression model.

There are pros and cons to the different target parameters and methods of analysis for ordinal outcomes. When an ordinal outcome has been dichotomised for analysis, the analyses are simple and the interpretation of the effect of interest can be easily understood. Ordinal outcomes that have been dichotomised, however, can discard potentially useful information on the levels of the scale and can lower statistical power compared to the original ordinal outcome [15, 16, 17]. Armstrong and Sloan found that the variance of the odds ratio using a logistic regression model is between 25–50% higher than the variance of the odds ratio estimated from a cumulative logit model [18], therefore reducing the power to detect a clinically important effect. When an ordinal outcome is treated as a continuous outcome, a linear regression model can be used to estimate a difference in means. Such an analysis assumes that the difference between any two categories of the ordinal outcome is uniform and separated by a quantifiable distance, the outcome has unbounded support, and that the outcome follows a normal distribution. Although the analyses are straightforward, these assumptions are likely to be violated as ordinal outcomes often have few categories that is insufficient to approximate a normal distribution and, more importantly, the distance between any two categories can not be described quantitatively. Any treatment effect estimated from such an analysis would therefore be difficult to interpret.

If the target parameter is an odds ratio from a proportional odds model, then the target parameter has a fairly simple clinical interpretation (usually being the odds of a better outcome). However, in practice, the proportional odds assumption may not hold [19, 20], in which case the odds ratios across each binary split of the ordinal outcome are not equal. Instead, the treatment comparison can be extended to be odds ratios that are not constant across the binary splits of the ordinal scale, which can be estimated using a partial proportional odds model [21]. Alternatively, adjacent-category logit and continuation ratio models could also be used to estimate the odds ratios, though these models have different model assumptions and interpretations of the target parameter(s). All these models also have natural extensions to account for repeated measures over time (e.g. mixed models [22] or Markov transition models [23], [24]).

There has been some methodological research that describe how ordinal outcomes can be used in in specific settings, such as vascular prevention trials [25] and comparative studies [26]. However, these studies focussed on a small number of statistical models and are not reflective of more general settings. With the increasing use of ordinal outcomes in randomised trials should come an improved understanding of how ordinal outcomes are used in practice. Better understanding will ensure that any issues in the use of ordinal outcomes in RCTs are identified and improvements to the reporting and analyses of such outcomes can be discussed. We aim to improve understanding by conducting a scoping review to systematically review the literature to (i) identify which target parameters are of interest in trials that use an ordinal outcome and whether these are explicitly defined; (ii) describe how ordinal outcomes are analysed in RCTs to estimate a treatment effect; and (iii) describe whether RCTs that use an ordinal outcome adequately report key methodological aspects specific to the analysis of the ordinal outcome.

We anticipate that the expected start date of this review is 15 August 2022 and the anticipated completion date is 1 February 2023. We will systematically search, identify, and describe RCTs that have used an ordinal outcome that have been published in the top four medical journals between 1 January 2012 and 31 July 2022. The four medical journals that will be included in this search are British Medical Journal (BMJ), New England Journal of Medicine (NEJM), The Lancet, and the Journal of the American Medical Association (JAMA). These journals have been selected because they are the top journals in the medical field that publish original, peer-reviewed research from RCTs and have been used in other reviews of trials [27, 28]. It is expected that these journals will capture the current best practice in the use of ordinal outcomes in RCTs.

We will identify RCTs to be included in the review by searching PubMed. Our search terms will employ search strategies developed to identify RCTs [29], and terms that are used to describe ordinal outcomes in the title and abstract of relevant published articles. Since we anticipate that varied terminologies are used to describe ordinal outcomes, we first examined various RCTs that use an ordinal outcome to determine the type of terminology that is used to describe ordinal outcomes. This enabled us to develop and refine our search strategy. The full search strategy to be used in the review is outlined in Table 1.

There is no pre-defined sample size. We plan to include all eligible studies that appear in our search using our pre-defined search strategy.

Titles and abstracts identified by the search strategy will be extracted into Covidence, a web-based tool for managing systematic reviews [30]. The review began with a piloting phase, where two authors (CS, RM) independently assessed 20 abstracts to ensure that the application of the inclusion and exclusion criteria was consistent between reviewers. If there was more than minor disagreement, then the criteria were further refined following discussion between the three reviewers (CS, RM, KL).

The review will be conducted by two authors (CS and one of RM or KL) through a two-phase screening process. In the first phase, all abstracts will be screened by one author (CS). A 10% random sample of the identified abstracts will be screened by a second author (either RM or KL) to identify those for inclusion. If there is disagreement over whether a study should be included between reviewers, then the study will move to the second phase of screening where the full text will be evaluated against the eligibility criteria by both reviewers and inclusion will be determined via discussion among the 3 reviewers. Studies that are found to have met all the inclusion and none of the exclusion criteria will be included.

Covidence will be used to extract and store the data from the review. A data extraction questionnaire was developed (Supplementary Material 1) and was piloted for use by CS and RM using a sample of 10 studies, with changes being made to the questionnaire where necessary. CS will extract data from all eligible studies in the review. Double data extraction will be performed by either RM or KL on a random sample of 10% of eligible studies and additionally when there is uncertainty about studies to ensure consistency throughout the data extraction process.

A full list of the data extraction items is provided in Table 2. We will only extract data that is reported in the article and supplementary material. We expect some data will be challenging to extract. The assumptions and simplifications that we will make under these conditions are summarised in Supplementary Table 1. If any post-hoc assumptions or simplifications are made, these will be reported as part of the analysis.

Once data extraction is complete, the data will be exported from Covidence. The extracted data will be cleaned and analysed using R [31]. The data will be summarised using descriptive statistics. Frequencies and percentages will be reported to summarise categorical data. Medians and interquartile ranges will be used to summarise continuous data. The data and code will be made available publicly on Github.

There will be no patients or public involvement in this review.

The findings from this review will be reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist [32].

A targeted review of RCTs in top medical journals publishing original and recent research will highlight the current state of practice for analysing ordinal outcomes. We have a priori specified eligibility criteria and strategies to handle anticipated challenges with data extraction. The screening and data extraction process will be conducted systematically, in which the pilot tests and double data extraction ensure the consistency and reliability of the extracted data. The search strategy, dataset and code will be made publicly available to ensure that the review is reproducible. The PRISMA-ScR checklist will be used to ensure that reporting is conducted to the highest standard.

This review does, however, have its limitations. Although there is an exclusive focus on the PubMed database and on the top four medical journals, we made this decision given the scoping nature of the review, to make it as reproducible as possible, and to ensure that the total number of studies included in the review was manageable. We assume that these journals capture the current best practice in the use of ordinal outcomes in RCTs, and therefore reflect the current use of ordinal outcomes in practice. Our search strategy may miss certain phrases or variants (particularly related to an ordinal outcome), although our piloting phase has hopefully mitigated this to a large degree. We deliberately avoided including the names of specific scales in our search strategy as this would provide a non-representative sample.

In addition to critically appraising and examining the literature regarding the use of ordinal outcomes in RCTs, this review will identify areas of improvement for the use and the analysis of ordinal outcomes for future trials to ensure the reliability and transparency of reporting of such outcomes. We also hope the results will be used to inform methodological research in the analysis of ordinal outcomes.

Funding sources/sponsors This work forms part of Chris Selman’s PhD, which is supported by an Australian Postgraduate Award administered through The University of Melbourne, Australia and a scholarship from the Australian Trials Methodology Research Network. Research at the Murdoch Children’s Research Institute is supported by the Victorian Government’s Operational Infrastructure Support Program. This work was supported by the Australian National Health and Medical Research Council (NHMRC) Centre for Research Excellence grants to the Victorian Centre for Biostatistics (ID1035261) and to the Australian Trials Methodology Research Network (ID1171422). Katherine Lee is funded by an NHMRC Career Development Fellowship (ID1127984).

Authors’ contributions CS, RM and KL conceived the study and CS wrote the first draft of the manuscript. All authors contributed to the design of the study, revision of the manuscript, and take public responsibility for its content.

Competing interests statement None declared.

Acknowledgements Chris Selman acknowledges the support of the University of Melbourne, Murdoch Children’s Research Institute and the Australian Trials Methodology Research Network. Robert Mahar acknowledges the support of the Australian Trials Methodology Research Network Seed Funding Program.

Ethics and Dissemination As data and information will only be extracted from published studies, ethics approval will not be required. Conference presentations and peer-reviewed publications will be used to disseminate the results.