Closed testing procedures for treatment-versus-control comparisons and multiple correlated endpoints

Multiple endpoints are used in some randomized trials in bio-medicine, whereas the majority are simply independently evaluated univariately, each to level $\alpha$, controlling the comparisonwise error rate (CWER) only. Here, we consider the joint evaluation of a few (e.g., 2, or 3), possibly different-scaled, correlated endpoints. (Notice, currently manifold published tests for high-dimensional problems, e.g. in genetics, represent a different issue). Two scenarios should be distinguished: i) co-primary endpoints, where the effect evidence is only available if both endpoints contribute with some noncentrality to the desired effect, and ii) primary endpoints, where any endpoint contribute to the effect and a multiple testing procedure allows the interpretation of the marginal hypotheses, i.e. per endpoint (which are the actual hypotheses of interest). This approach is investigated in the context of higher-level global tests for multiple correlated endpoints. Co-primary endpoints (case i)) are analysed be means of an intersection-union test (IUT), where the global claim is proved if each marginal test is under $H_{1}$, each at level $\alpha$. Primary endpoints (case ii)) are analysed almost differently by means of an union-intersection test (UIT), e.g., by a maxT-t test [10]. Here, the claim is proved if at least one, anyone, marginal test is under $H_{1}$, each at an adjusted level $\alpha_{i}$ whereas the entire procedure controls the FWER in a strong sense. Interestingly, the criterion ’at least one, anyone’, also includes ’all’ marginal tests, which allows a link between case i) and case ii) using the all-pairs power concept).
By means of a simulation study several tests are considered for both size and power, whereas first a simple two-sample design with 2 and 3 correlated endpoints is used, followed by a three sample design $[C,T_{1},T_{2}]$ and 2 endpoints. This dimension limitations result from the properties of the closed testing procedure (CTP), which is the focus of the paper.
Notice, in biology and genetics, multivariate tests are used to combine the information of many low informative endpoints as stand-alone approaches.

The closed testing procedure (CTP) [16] and the maxT-test use almost contrary principles for a similar objective. While the maxT-test tests each elementary hypothesis to an adjusted $\alpha$-level (e.g. $\alpha/\xi$)- based on the union-intersection (UIT) principle (’or, ,…, or’), in the CTP each elementary hypothesis is tested to the full $\alpha$-level, but not alone, rather within an intersection-union (IUT) principle (’and,…,and’), where all hypotheses in the intersection branch of the closed system tree must also be under $H_{1}$ (each at $\alpha$-level). The maxT test requires a complex multivariate t-distribution and their correlation matrix, whereas the CTP requires only quantiles of an univariate (F, or t-) distribution (without considering correlation). Central to the CTP is the definition of the elementary hypotheses, which alone are of interest and are then to be interpreted.

First, the two-sample design is discussed $H_{0}:Y_{0}^{1}-Y_{0}^{2}$. Based on this, all subset intersection hypotheses up to the global hypothesis are constructed, involving $H_{i0}$. Following the IUT principle, $H_{i}$ is rejected at level $\alpha$ if and only if $H_{i}$ itself is rejected and all hypotheses which are include them. Each of these many hypotheses is tested individually with a level $\alpha$-test. This may be any appropriate level $\alpha$-test, making the CPT flexible. Even within a tree, different tests, e.g. F-tests and t-tests, can be used. Each of these branches tests of the tree (determined by the $\xi$ elementary hypotheses) represent an IUT, i.e. $T^{CTP}=min(T_{1},...,T_{\xi})$, or $p^{CTP}=max(p_{1},...,p_{\xi})$. Comparing of endpoints $Y_{ii^{\prime}}$ form a complete family of hypotheses [21] avoiding complex subset intersection hypotheses due to the non-overlapping hypotheses. For the simple bivariate $Y_{1},Y_{2}$ and trivariate $Y_{1},Y_{2},Y_{3}$ 2-sample design, the family include the elementary, intersection and global hypotheses, as shown in Tables 1 and 2

Secondly, considering treatment-versus-control comparisons with multiple endpoints, e.g. the bivariate case, the properties of a complete family are lost and the structure of the CTP tree becomes complex including the partition hypotheses, i.e. the special non-overlapping intersection hypotheses, e.g. for the simple design with $[C,T1,T2],[Y1,Y2]$:

Thus, not only uni-, bi- and tri-variate tests for the 2 and k sample problem are necessary, but also for the specific partition hypotheses for non-overlapping hypotheses, e.g. $H_{01}^{12}:H_{02}^{2}$, which are tested here using Fisher’s product criterion. The already complex system of this simple example shows that CTP is practically limited to only a few comparisons to control (2, 3, or 4) and only a few endpoints 2,3, or 4. In the elementary hypotheses, instead of pairwise t-tests, multiple contrasts for $\mu_{i}-\mu_{0}$ are used [4, 9], which exploit the entire $df$, a significant power advantage especially for small $n_{i}$. To avoid directional errors, CTP’s were formulated for one-sided hypotheses here.

Both CTP and maxT-test become more conservative with increasing number of elementary hypotheses. Controlling FWER, this implies an inherent power loss compared to a single test. In the maxT-test, this is determined by the number $q$ of individual tests in the $\xi$-variate t-distribution, with the correlation between the tests having a conservatism-reducing effect. In contrast, the conservativeness of the CTP is due to that of the underlying IUT in the branches of the CTP decision tree, whereas the correlation between the individual tests is not considered in that univariate distribution. Furthermore, the number and type of individual tests in the IUT’s depends on the specific structure of the CTP tree. If the number of individual tests is too high within each IUT (and e.g. 10 are quickly reached in real applications), it becomes hopelessly conservative - a limitation of the CTP in principle to low dimensional cases.

Presumably, the maxT test in connection with multiple endpoints was first proposed for analysis of multiple binary endpoints as a resampling test [22] (albeit as a min-p test; see the comparison between maxT and min-p test [3]). Here, only the parametric maxT test is considered as an asymptotic test version. The extension to multiple contrast tests with correlated endpoints was introduced as max(maxT) test, where one part of the approximate correlation matrix consists of the calculated correlations between the treatment contrasts and the other part consists of the empirical Pearson correlation between the endpoints [5, 6, 7, 8]. Thus, adjusted p-values or simultaneous confidence intervals for both differences and ratios (for control) can be estimated with the CRAN package SimComp. An alternative approach for estimating the variance-covariance matrix for the multivariate normal distribution of underlying maxT tests represents the multiple marginal models approach [19]. Obtaining the asymptotic joint distributions of parameter estimates from multiple marginal models (glm’s up to glmm’s) fitted to the same observations is a very helpful and general feature of this approach, available as the function mmm within the CRAN package multcomp [11]. In the context of multiple endpoints, this approach was used for multiple binary endpoints in both 2- and k-sample design [13], for multiple and repeated measures endpoints [12, 18], for multiple normal distributed, proportions and different-scaled endpoints [9](in chapters 2.4.4., 3.6.1. and 5.4), for composite and their marginal endpoints [20], as well for multiple time-to-event endpoints by the Wei-Lachin test [2]- probably the first mmm approach.

There are some principal properties: i) the power of the max(maxT)) test is equal to that of the Bonferroni test for correlations near to 0 (otherwise for correlations near to 1 of the marginal tests), ii) the more endpoints, the more conservative the procedures, and iii) in the closed test are $p_{adjust}=p_{marginal}$ when the p-values in all subset global tests of the IUT branch are smaller.

Single-step tests for $q$ co-primary endpoints use commonly the IUT, with the global claim when $p_{adjusted}^{IUT}=max(p_{1},...,p_{q})<\alpha$. This test has two disadvantages, it is conservative with increasing number of endpoints (because the correlation between the endpoints is not taken into account) and it has only a global statement, i.e. there are no inference statements about the effect of the individual endpoints. Therefore, the alternative considered is the UIT (i.e., exactly the tests for multiple correlated endpoints), but with the all-pairs power concept. The UIT is under $H_{1}$, if at least one, any single hypothesis is under $H_{1}$. And this includes the case of interest here, that all single hypotheses are under $H_{1}$. Using UIT’s with the all-pairs power concept provides both the global claim of co-primary endpoints and claims on the individual endpoints by means of adjusted p-values or simultaneous confidence intervals. A simulation study for practically relevant designs must clarify whether the quality disadvantage compared to the IUT is still tolerable.

Familywise error rate (in weak and strong sense) as well as 4 different power estimates were considered in a simulation study with 5000/10000 independent replications per setting using a two or three sample balanced design ($n_{i}=5;20$) with normal homoscedastic errors with two (or three) correlated endpoints ($\rho=0.9,0.7,0.5,0$, where $\rho=-0.5$ is without any practical relevance for the unidirectional bivariate tests). Normal distributed, homogeneous errors, complete multivariate data without missing values, endpoints of different magnitude (so that standardization is needed) and low dimensions (i.e., 2 or 3 treatments, 2,3 endpoints) are used. The non-overlapping hypotheses in the CTP were tested by Fisher product criterion.
The aim is to interpret the elementary hypotheses when correlated, normal distributed multiple endpoints are considered. Therefore, the following tests are compared: i) max(maxT))-test using multivariate t-distribution with Pearson/calculated correlation matrix [5] (’maxT’), ii) max(maxT))-test using estimated correlation matrix from multiple marginal models [9] (’mmm’), iii) max(maxT)test using Bonferroni-adjustment for multiple endpoints (’Bon’), iv) closed testing using global parametric OBrien-type summary test (’CTPps’), v) closed testing using global rank-transformed OBrien-type summary test (’CTPns’), vi) closed testing using squared Euclidean distance based summary test (’CTPd’), vii) closed testing using maximum absolute component distance based summary test (’CTPD’) and viii) IUT (’IUT’) when considering all-pairs power only. Furthermore, the marginal tests per endpoint are considered (’Y1,Y2,Y3’) as well the individual estimates when considering the individual power (using the extensions 1,2,3).