Integrative Sparse Partial Least Squares

With the rapid development of technology, comes the need to analyze data with high dimensions. Partial least squares, introduced by Wold et al. (1984), has been successfully used as a dimension reduction method in many research areas, such as chemometrics (Sjöström et al., 1983) and more recently genetics (Chun and Keleş, 2009). PLS reduces the variable dimension by constructing new components, which are linear combinations of the original variables. Its stability under collinearity and high-dimensionality gives PLS a clear superiority over many other methods. However, in high dimensional problems, noise accumulation from irrelevant variables has long been recognized (Fan and Lv, 2010). For example, in omics studies, it is wildly accepted that only a small fraction of genes are associated with outcomes. To yield more accurate estimates and facilitate interpretation, variable selection needs to be considered. Recently, Chun and Keleş (2010) propose a sparse PLS technique to conduct variable selection and dimension reduction simultaneously by imposing Elastic Net penalization in the PLS optimization.

Another challenge that real data analyses often face is the unsatisfactory performance generated from a single dataset (Guerra and Goldstein, 2009), especially for data with a limited sample size. Due to the recent progress in data collection, a possibility exists for integration across multiple datasets generated under similar protocols. Methods for analyzing multiple datasets include meta-analysis, integrative analysis, and others. Among them, integrative analysis has been proved to be effective both in theory and practice and have better performance in prediction and variable selection than other multi-datasets methods (Liu et al., 2015; Ma et al., 2011), especially including meta-analysis (Grützmann et al., 2005).

Considering the wide applications of PLS/SPLS to high dimensional data, we propose an integrative SPLS (iSPLS) method to remedy the aforementioned problems of the conventional SPLS technique caused by a limited sample size. Based on the SPLS technique, our method conducts the integrative analysis of multiple independent datasets using the penalization method to promote certain similarities and sparse structures among them, and further improve the accuracy and reliability of variable selection and loading estimation. Our penalization involves two parts. The first penalty conducts variable selection under the paradigm of integrative analysis (Zhao et al., 2015), where a composite penalty is adopted to identify important variables under both the homogeneity structure and heterogeneity structure. The intuition of adding the second penalty comes from empirical data analyses, that is, datasets with comparable designs may have a certain degree of similarity, which may help further improve analysis results. Our work advances from the existing sparse PLS and integrative studies by merging the dimension reduction technique and integrative analysis paradigm. Furthermore, we consider both similarity and difference across multiple datasets, which is achieved by our introduction of a two-part penalization.

The rest of the paper is organized as follows. In Section 2, for the completeness of this article, we first briefly review the general principles of PLS and SPLS, and then formulate the iSPLS method and establish its algorithms. Simulation studies and applications to TCGA data are provided in Section 3 and 4. Discussion is organized in Section 5. Additional technical details and numerical results are provided in the Appendix.

We simulate four independent studies each with sample size 40 and 120, and 5 response variables. For each sample, we simulate 100 predictor variables, which are jointly normally distributed, with marginal means zero and variances one. We assume that the predictor variables have an auto-regressive correlation structure, where variables $j$ and $k$ have correlation coefficient $\rho^{\left|j-k\right|}$, and $\rho=$ 0.2 and 0.7, corresponding to weak and strong correlations, respectively. All the scenarios follow the model $Y^{(l)}=X^{(l)}\beta^{(l)}+\epsilon^{(l)}$, where $\epsilon^{(l)}$ is normally distributed with mean zero. Following the data-generating mechanism in Chun and Keleş (2010), the columns of $\beta_{i}^{(l)}$, for $i=2,...,5$, are generated by $\beta_{i}^{(l)}=1.2^{i-1}{\beta}_{1}^{(l)}$. The sparsity structures of direction vectors $w^{(l)}$ are controlled by $\beta_{1}^{(l)}$. Within each dataset, the number of variables associated with the responses is set to be 10. The nonzero coefficients $\beta_{1}^{(l)}$ range from 0.5 to 4. We simulate under both the homogeneity and heterogeneity models.

Under the homogeneity model, direction vectors have the same sparsity structure, with similar or different nonzero values, corresponding to Scenario 1 and Scenario 2, respectively. Under the heterogeneity model, two scenarios are considered. In Scenario 3, four datasets share 5 important variables in common, and the rest important variables are dataset-specific. That is, direction vectors have partially overlapping sparsity structures. In Scenario 4, direction vectors have random sparsity structures with random overlappings. These four scenarios comprehensively cover different degrees of overlapping in sparsity structures.

To better gauge performance of the proposed approach, we also consider the following alternative approaches: (a) meta-analysis. We analyze each data set separately using the PLS or SPLS approaches and then combine results across datasets via meta-analysis; (b) a pooled approach. Four datasets are pooled together and analyzed by SPLS as a whole. For all approaches, the tuning parameters are selected via 5-fold cross-validation. To evaluate the accuracy of variable selection, the averages of sensitivities and specificities are computed across replicates. We also evaluate prediction performance by calculating mean-squared prediction errors (MSPE).

Summary statistics based on 50 replicates are presented in Tables 1-4. The simulation indicates that the proposed integrative analysis method outperforms its competitors. More specifically, under the fully overlapping (homogeneity) case, when the magnitudes of nonzero values are similar across datasets (Scenario 1), iSPLS-Homo_M has the most competitive performance. For example, in Table 1, with $\rho=0.2$ and $n=120$, MSPEs are 49.062 (meta-PLS), 5.686 (meta-SPLS), 1.350 (pooled-SPLS), 2.002 (iSPLS-Homo_M), 2.414 (iSPLS-Homo_S), 3.368 (iSPLS-Hetero_M) and 3.559 (iSPLS-Hetero_S), respectively. Note that under Scenario 1, the performance of iSPLS-Homo_M and iSPLS-Homo_S may be slightly inferior to that of pooled-SPLS. Since with fully comparable datasets, it is sensible to pool all data together, thus, pooled-SPLS may generate more accurate results. However, when the nonzero values are quite different across datasets (Scenario 2), as can be seen from Table 2, iSPLS-Homo_S outperforms others, including pooled-SPLS. Under the partially overlapping Scenario 3 (heterogeneity model), iSPLS-Hetero_M and iSPLS-Hetero_S seem to have better performance, for example when $\rho=0.7$ and $n=40$, they have higher Sensitivities (0.821 and 0.821, compared to 0.675, 0.575, 0.800 and 0.800 of the alternatives), smaller MSPEs (24.637 and 23.734, compared to 268.880, 30.928, 84.875, 40.867, and 39.492 of the alternatives), and with similar Specificities. Even under the non-overlapping Scenario 4, which is not favourable to multi-datasets analysis, the proposed integrative analysis still has reasonable performance. Thus, our integrative analysis methods have the potential to generate more satisfactory results comparable to meta-analysis, when the overlapping structure of multiple datasets is unknown.

To sum up, under the homogeneity cases, iSPLS-Homo_M and iSPLS-Homo_S have the most favourable performance, and under the heterogeneity cases, iSPLS-Hetero_S and iSPLS-Hetero_M outperform the others. It is also interesting to observe that the performance of the constructed penalties depends on the degree of similarity across datasets. For example, in Table 2, iSPLS-Homo_S (iSPLS-Hetero_S), with a less stringent penalty, has relatively lower MSPEs than iSPLS-Homo_M (iSPLS-Hetero_M), while in Table 1, it is the other way around. This comparison suggests the sensibility of the proposed contrasted penalization.

PLS regression has been promoted in ill-conditioned linear regression problems that arise in several disciplines such as chemistry, economics, medicine, and psychology. In this study, we propose an integrative SPLS (iSPLS) method, which conducts the integrative analysis of multiple independent datasets based on the SPLS technique. This study significantly extends the novel integrative analysis paradigm by conducting a dimension reduction analysis. An important contribution is that, to promote similarity across datasets more effectively, two contrasted penalties have been developed. Under both the homogeneity and heterogeneity models, we develop the magnitude-based contrasted penalization and sign-based contrasted penalization. We develop effective computational algorithms for the proposed integrative analysis. For a variety of model settings, simulations demonstrate satisfactory performance of the proposed iSPLS method. The application to TCGA data suggests that magnitude-based iSPLS and sign-based iSPLS do not dominate each other, and are both needed in practice. The stability and prediction evaluation provides some support to the validity of the proposed method.

This study can be potentially extended in multiple directions. Apart from PLS, integrative analysis can be developed based on other dimension reduction techniques, such as CCA, ICA, and so on. For selection, the MCP penalty is adopted and can be potentially replaced with other two-level selection penalties. Integrative analysis can be developed based on SPLS-SVD. Moreover, iSPLS is applicable to non-linear frameworks such as generalized linear models and survival models. In data analysis, both the magnitude-based iSPLS and sign-based iSPLS have applications far beyond this study.