The Role of Positive and Negative Citations in Scientific Evaluation

Our experiments are conducted on the APS dataset, which contains 71,287 publications published in two different subsets of APS: Physical Review C and Physical Review E (http://publish.aps.org/datasets), between 1970 and 2013 (43 years). Each record in the dataset includes the paper s title, DOI, author(s), date of publication, affiliation(s) of authors and publisher. A list of citations is provided by an independent dataset within the APS dataset.

The structure of PANDORA is illustrated in Fig. 2. PANDORA quantifies publication impact by differentiating positive COI, negative COI, positive suspected COI and negative suspected COI between the citing and cited publications. Furthermore, PANDORA computes the authoritative score of each paper based on CAJTRank [17]. CAJTRank was a graph-based ranking method and used PageRank [20] algorithm and HITS [21] algorithm simultaneously to rank scientific papers by relying on the four factors: citations, authors, journals (conferences) and publication time. PANDORA adopts the weighted PageRank to fairly evaluate the impact of scholarly papers, and it also leverages the credit allocation algorithm [19] to allocate the impact of a paper to its signed authors. In this paper, we consider citations, authors, journals and time factors, mainly because these factors can influence the impact of papers.

If the authors of a citing paper and a cited paper have been co-authors of one or more papers, COI relationships exist. Although the COI citations may distort the appraisal of research impact, some of these citations are positive and necessary. Consider a scenario where the authors of the citing paper $P_{i}$ and the cited paper $P_{j}$ have published papers together. If $P_{i}$ and $P_{j}$ are co-cited by the papers from different authors, we consider the citation of $P_{i}$ and $P_{j}$ is reasonable, and we refer the citation as a positive COI, with its citation strength $W_{i,j}^{P-PCOI}$ set as 1.

If COI relationships exist between the authors of a citing paper $P_{i}$ and a cited paper $P_{j}$, and no paper cites $P_{i}$ and $P_{j}$ simultaneously, we define cases like this as negative COI. The citing strength is quantified by the correlation strength between the authors, and we introduce two factors: collaboration times, and collaboration time span, for adjusting the weight of citation strength. The negative COI strength of co-authors in PANDORA is defined as follows:

where $W_{x,y}^{A-NCOI}$ is the negative COI strength of co-authors between the $x$th author and the $y$th author. $N_{x,y}^{Co-author}$ indicates the cumulative number of papers coauthored by the $x$th author and the $y$th author. $N_{x,y}^{Co-author}=|S_{x}\cap S_{y}|$, where $S_{x}$ is a set of papers published by the $x$th author, and $S_{y}$ is a set of papers published by the $y$th author. $T_{x,y}^{N}$ is the last year of the $x$th author citing the $y$th author. $T_{x,y}^{1}$ is the initial year of the $x$th author starts citing the $y$th author. $\Delta T_{c}=T_{x,y}^{N}-T_{x,y}^{1}+1$ indicates the number of year between the first and the last collaborations of authors $A_{x}$ and $A_{y}$.

The relationship strength of each two publications based on the co-author’s negative COI is calculated by:

where $W_{i,j}^{P-NCOI}$ is the negative COI strength of the $i$th paper and the $j$th paper. It is the sum of negative COI between each two authors of citing paper and cited paper. $N_{i,j}^{Co-paper}$ indicates the cumulative number of papers coauthored between each two signed authors in the $i$th paper and the $j$th paper. $N_{i,j}^{Co-paper}=|S_{i}\cap S_{j}|$, $S_{i}$ is the set of all papers with signed authors of the $i$th paper. $S_{j}$ is the all papers set of signed authors of the $j$th paper. The citation strength of each two publications is defined as follows:

where an exponentially decaying formula is proposed to calculate the citation strength between paper $P_{i}$ and paper $P_{j}$. We consider the citing strength ranges from 0 to 1. As scholars usually intend to cite recent work, if a paper fails to attract citations in early years after being disseminated, the chance of attracting more citations over time will be limited. $W_{i,j}^{P-NCite}$ is the citation strength of the $i$th paper citing the $j$th paper. $W_{i,j}^{P-NCite}$ is defined within a range between 0 and 1 in our work, instead of assuming the citation strength as 1 (i.e. without regarding the COI relationships) in previous works. The value of citation strength is reasonable with the range between 0 and 1, and the formula favors current citation. $\rho$ is a constant value illustrating predefined decay parameter [17]. $T^{Current}$ stands for the current time, $T^{Cite}$ is the time of paper $P_{i}$ citing paper $P_{j}$ , and $T^{Current}-T^{Cite}+1$ represents the time duration of paper $P_{i}$ citing paper $P_{j}$.

Given paper $P_{i}$ cites paper $P_{j}$, although the authors between the two papers have not collaborated in producing any joint paper, the authors of $P_{i}$ and $P_{j}$ belong to the same affiliation, we consider that suspected COI relationship exists between them. If the two publications are co-cited by one or more publications (i.e. demonstrated relevance), we regard paper $P_{i}$ citing paper $P_{j}$ as positive suspected COI, and the citing strength $W_{i,j}^{P-PSCOI}$ is set as 1.

Given paper $P_{i}$ cites paper $P_{j}$, the authors between the two papers have not co-authored any paper, and the authors of $P_{i}$ and $P_{j}$ belong to the same affiliation, however the two publications are not co-cited by other publications (i.e. without demonstrated relevance), we regard paper $P_{i}$ citing paper $P_{j}$ as negative suspected COI. The citing strength is quantified by introducing the two factors: citing times and citing time span. The strength of negative suspected COI relationship of each two authors is defined as follows:

where $W_{x,y}^{A-NSCOI}$ is the negative suspected COI strength of co-authors of the $x$th author and the $y$th author. $N_{x,y}^{Cite}$ is the cumulative number of papers of the $x$th author citing the $y$th author. $\Delta T_{s}=T_{x,y}^{N}-T_{x,y}^{1}+1$ indicates the number of years between the first and the last citing of authors $A_{x}$ and $A_{y}$. The strength of suspected COI relationship of each two articles is calculated by:

where $W_{i,j}^{P-NSCOI}$ is the negative suspected COI strength of the $i$th paper and the $j$th paper. It is a summation of all the negative suspected COI between authors of paper $P_{i}$ and paper $P_{j}$. $N_{i,j}^{Cite}$ is the cumulative citing number between the authors of the $i$th paper and the $j$th paper. The strength of citation relationship of each two articles is defined as:

In CAJTRank, all citing weights are set as 1, and the abnormal citations are ignored, which leads to the potential issues in article impact evaluation. PANDORA is proposed to address these issues to improve objective and accurate impact assessment of research articles. The core idea of PANDORA is that the prestige of a publication is quantified by the scores of its weighted-PageRank, authors, journal published and references, namely the impact of a publication. Compared to CAJTRank algorithm, PANDORA method constructs a weighted-PageRank to capture the authority score of each publication in citation networks. While the initial score of each paper is set as $1/N$, $N$ indicates the total numbers of scholarly papers in the experiment. Meanwhile, PANDORA also considers the authority scores of each author, journal and references, which are calculated by HITS algorithm. Particularly, CAJTRank algorithm assumed all co-authors’ contributions are equal to a paper. The CAJTRank algorithm neglects a fact that contributions of different authors of a paper are never equal. In order to resolve this problem, credit allocation is introduced in order to reasonably distribute the influential score of different authors of individual paper [19]. Concrete process of computing the prestige score of each publication is demonstrated as follows:

The weighted PageRank score of paper $P_{i}$ is computed by

where $IN(P_{i})$ indicates all the papers of linking to paper $P_{i}$, $|OUT(P_{j})|$ is the total number of publication $P_{j}$ linking out. $W_{j,i}$ illustrates the citation strength of paper $P_{j}$ citing paper $P_{i}$. $S(P_{j})$ is the original score of paper $P_{j}$ before iteration is updated.

The authors’ authority scores of single publication are determined by the influence score of each author, while the prestige score of individual author is related to the impact scores of his/her published papers, and can be calculated by the HITS algorithm. When computing the hub score of each author, credit allocation algorithm is used to distribute the proportion of impact of single paper to different authors.

Specifically, the authors’ prestige scores of each paper, $Author(P_{i})$, is calculated by

where $T(A)$ indicates a summation of all authors’ hub scores in the experimental data. $Neighbor(P_{i})$ denotes the co-authors’ list of paper $P_{i}$, $Neighbor(A_{j})$ shows all the papers published by author $A_{j}$, $CreditShare(P_{k},A_{j})$ illustrates the proportion of publication $P_{k}$ impact score by different authors of $P_{k}$. $S(P_{k})$ is the prestige score of paper $P_{k}$, $|Neighbor(A_{j})|$ is the number of publications of author $A_{j}$.

Likewise, the prestige score of a journal paper is computed by the HITS algorithm. Specific formula is shown as follows:

where $Journal(P_{i})$ demonstrates the prestige score of publication $P_{i}$ transmitted from its published journal, and the hub score of each journal includes the prestige scores of all the papers published in the journal. $T(J)$ is the sum of all journals’ hub scores, $Neighbor(P_{i})$ is the journal published by paper $P_{i}$, and each paper belongs to one journal. $Neighbor(J_{j})$ is the set of papers published by journal $J_{j}$, $|Neighbor(J_{j})|$ is the total number of publications in $Neighbor(J_{j})$.

The reference scores of each publication are also computed by the HITS algorithm:

where $Reference(P_{i})$ shows the publication score $P_{i}$ collected from the authority scores of its references. $T(P)$ indicates the total scores transmitted from all the hub papers. $Neighbor(P_{j})$ contains all the publications that $P_{j}$ links to. $|Neighbor(P_{j})|$ is the sum of publication references $P_{j}$.

The authority score of each article contains the four components: PageRank score of the paper, scores of authors, journal and references of this work. The authority score is defined as the weighted sum of these components plus a normalize term:

In our experiment, if the score gap between the present and previous prestige of each paper is less than 0.0001, we consider the presented PANDORA iterative algorithm converges. $S(P_{i})$ indicates the authority score of publication $P_{i}$. $\alpha$, $\beta$, $\gamma$ and $\delta$ are constant parameters, ranging from 0 to 1. The probability of random jump is set as 0.15 experimentally; meanwhile, the summation of $\alpha+\beta+\gamma+\delta$ is set as 0.85 for obtaining good experimental results. Currently, the typical approaches for parameter estimation include simple linear regression, multivariate linear regression and support vector machine regression [22]. Based on our experimental characteristic, multivariate linear regression is employed to estimate the parameters of PANDORA, CAJTRank and FutureRank, due to the characters of different factors and linear evaluation. We set the sum of all parameters as 0.85 in the three algorithms, and we find that a parameter captures relative high value, and other parameters receive relative low values by multivariate linear regress. We then estimate three groups of optimal parameters to compare the accuracy of RI and Spearman s rank correlation coefficients of the above mentioned three algorithms. More details can be found in reference [23].

In order to assess the impact of each institution objectively, the computing process is divided into the two parts in this section. The first part is to objectively allocate the impact of each publication to its co-authors by PANDORA. If a publication is only signed by an author, the prestige score of the paper belongs to the single author and this situation is relatively popular in decades ago. With the development of Internet, the collaboration among multiple authors also starts to prevail. Correspondingly, fair distribution of publication impact to multiple authors becomes an important practice to evaluate the impact of different scholars and their institutions.

To fairly distribute the impact of a paper, credit allocation algorithm is leveraged to resolve the coauthors’ contributions to a paper. In this algorithm, the contributive proportion of each author to a paper is determined by the total credit from all of his/her co-cited papers. The second part is to calculate the impact of different institutions. The impact of a scholar is defined as follows:

where $I_{A}$ refers to the set of individual scholar’s impact captured from his/her publications. $R$ refers to the proportion of credit allocation to different authors. $S(P_{i})$ indicates the prestige score of a scholarly paper.

The institutional impact is determined by the impact of all scholars’ publications in this institution:

where $I_{I}$ indicates the set of institutional impact, containing the impact of all scholars in this institution. $I_{A,i}$ denotes a scholar’s impact. For the authors with multiple affiliations, we consider the first affiliation as the primary institution.

Furthermore, the impact of a country is determined by the impact of publications of all institutions in this country. Given a country with $i$ institutions, the impact of a country is defined as follows:

where $I_{C}$ represents the set of country impact. $I_{I,i}$ denotes the institutional impact.

In order to objectively evaluate the impact of institutions, we first explore the COI relationships in citation networks. 71,287 papers with 755,902 author pairs are analyzed in our experiment. The author pairs are divided into the four categories: positive COI, negative COI, positive suspected COI and negative suspected COI. It is interesting to note that the author pairs of positive COI relationships and negative COI relationships are 59,388 pairs and 48,377 pairs respectively. There are 25,803 papers with positive COI, and 24,294 papers about with negative COI. We also find that there are 3,196 author pairs exist suspected COI relationships, in which author pairs of positive suspected COI relationships and negative suspected COI relationships are 1,645 and 1,551 respectively.

We conduct the experiments on two subsets of APS to compare PANDORA with FutureRank [16] and CAJTRank [17]. The experiments use two metrics: $RI$ [23] and Spearman s rank correlation coefficient [24]. Due to the lack of ground truth for ranking papers, previous research adopted the future PageRank score [16] or the future citations [17] as the ground truth. In our study, citations without COI are used as the ground truth instead. The reason is that both future PageRank scores and future citations contain negative citations. Therefore, the ground truth may be biased, and it likely results in the inaccurate appraisal of the impact of academic publications.

The concept of $RI$ is described as follows: let $R$ indicate the list of top-K returned papers of a ranking method and $L_{1}$ represent the list of ground truth. For any scholarly paper $P_{i}$ in $R$ with the ranked order $ro$, the $RI$ of $P_{i}$ at $k$ is defined as:

The above formula implies that for any article $P_{i}$ of the top-K ground truth list, if it is ranked higher, $RI$ of the article $P_{i}$ will be higher. Correspondingly, we are able to gain the $RI$ of the list $R$ at $k$. According to the $RI$ of each article, the $RI$ of the list $R$ at $k$ could be defined as follows:

To better understand the impact that PageRank, authors, journal, citations and time factors have on the prestige scores of each article, a multivariate linear regression is implemented to estimate the parameters of PANDORA, FutureRank and CAJTRank, to optimize the ranking results in terms of $RI$ performance. Fig. 3 shows different $RI$ accuracy rates of the three algorithms on top-K papers, and $K$ ranges from 10 to 300. The $RI$ accurate rates of CAJTRank and FutureRank are between 0.6 and 0.677, and between 0.5 and 0.633, respectively. In comparison, the $RI$ accurate rates of PANDORA are between 0.656 and 0.8. As shown in Fig. 3, PANDORA consistently outperforms CAJTRank and FutureRank in terms of $RI$ for all $K$ values.

Moreover, the Spearman s rank correlation coefficient is utilized to assess the similarity of these algorithms.

where $R_{1}(P_{i})$ and $R_{2}(P_{i})$ indicate the position of publication $P_{i}$ in the ground truth rank list and the corresponding algorithm rank list, respectively. $\overline{R}_{1}$ and $\overline{R}_{2}$ are the average rank positions of all publications in the two ranks lists, respectively.

In Fig. 4, we observe that Spearman’s rank correlation coefficients of PANDORA and CAJTRank are at around 0.5 and 0.42 respectively, while the coefficient changes from 0.267 to 0.519 for FutureRank. The best result could be obtained by considering all kinds of information: PageRank, authors, journal, references and time factors. PANDORA outperforms the other state-of-art methods on two subsets of APS.

By comparing $RI$ and Spearman’s rank correlation coefficients of CAJTRank and FutureRank, we find that journal factor is beneficial to assess the impact of papers. At the same time, by comparing PANDORA and CAJTRank, the preceding results indicate that by the using COI relationships, weighted PageRank could improve the performance of PANDORA. Next, we take dynamic evolutionary nature of citation networks, time information and COI relationships of authors into consideration, and jointly use them to determine the citing strength, which have important effect on generating a better $RI$ and Spearman’s rank correlation coefficient. Moreover, PANDORA’s author credit allocation scheme helps generating a fair and objective ranking list.

In order to capture the existence of COI citations at the institution-level and at the country-level, several characteristics are investigated for institution $m$ and country $n$: the institution size ($A_{m}$), represents the total number of authors from institution $m$ who published at least one publication. $P_{m}$ shows the number of publications published from institution $m$. $COIC_{m}$ is the cumulative COI citations derived from all publications in institution $m$. The country size ($A_{n}$), denotes the total number of authors from country $n$ who published at least a publication. $P_{n}$ represents the number of publications published from country $n$; $COIC_{n}$ shows the cumulative COI citations derived from all publications in country $n$.

Figs. 5a and 5b show the correlation between the institution size $A_{m}$ and both the average COI citations $COIC/A_{m}$ and the average publications COI citations $COIC/P$. Fig. 5a indicates that most institutions have a small number of scholars publishing on APS journals. It could be observed that COI citations exist in institution with different scales, and the larger the institution is, the more COI citations are likely to be. Fig. 5b shows the correlation between institution size $A_{m}$ and the average COI citations of each paper $COIC/P_{m}$. According to Fig. 5b, we can observe that COI citations of each paper range from $10^{-4}-10^{2}$, and the average COI citations of each paper range from 0.529 to 1.022. Figs. 5c and 5d show the correlation between the country size $A_{n}$ and both the average COI citations $COIC/A_{n}$ and the average publications COI citations $COIC/P_{n}$. Fig. 5c indicates that most countries have a small number of scholars in the APS publications, except for a few countries. For example, American scholars are the highest contributors to PRC and PRE, with 20,468 scholars. We also find that the country size $A_{n}$ positively correlates with COI citations, which indicates that large countries are likely to result in COI citations according to the analysis of each author’s COI citations. The potential cause of this phenomenon is that large institutions or large countries have more international and internal collaborations. According to Fig. 5c, we also observe that when country size falls in between ranks 60 and 100, $COIC/A_{n}$ presents a sudden fall. To obtain a better understanding of the decrease, we investigate these countries and their adjacent countries in the following three aspects: the number of papers, the number of co-authors and COI citations. We observe an interesting phenomenon: small COI citations exist in these countries, and the number of co-authors (about 3 authors) of each paper is less than their adjacent countries (around 4 authors). This may be the reason why impact of these countries is relatively small compared with other countries. Fig. 5d shows the correlation between the country size $A_{n}$ and the average COI citations of each paper $COIC/P_{n}$. It indicates that each publication contains more COI citations in large countries compared to small countries, COI citations of each paper range from 0.156 to 0.883, while the lower $COIC/P_{n}$ values are similar to the ones in Fig. 5c.

In order to investigate the impact of average institution and average scholarly papers in different institution sizes and country sizes, several external characteristics are examined. The institution impact ($I_{I}$) represents the total scores of all authors collected by all papers $P_{i}$ in this affiliation. The country impact ($I_{C}$), denotes the total summation of all the institutions collected by all the authors in this country. In Figs. 6a and 6b, the correlation between the institution size $A_{m}$ and both average institution impact $I_{I}/A_{m}$ and average scholarly papers impact $I_{I}/P_{m}$ are shown. We find that institution size influences the impact of each author in his/her affiliation, that is, scholars in large institution obtain higher average impact (Fig. 6a). We also notice that the institution size has little influence on the average impact of publications in institutions on the whole (Fig. 6b). In Figs. 6c and 6d, the correlation between the country size $A_{n}$ and both country average impact $I_{C}/A_{n}$ and average country scholarly papers impact $I_{C}/P_{n}$ are demonstrated, which indicates country size has some positive influence on the impact of country, however, the country size has little influence on the average impact of publications in countries on the whole. According to Figs. 5c and 6c, we observe that if the impact of country is small, its COI citations are small.

To investigate the relationship of COI citations and impact of papers evolving at the institution-level and at the country-level, previous studies have been focusing on citation distributions, yet little is concerned about the evolution of COI citations of individual institution and country. The trends of COI citations variations in different institutions and countries are well illustrated by the COI citations history of publications extracted from PRC and PRE subsets of APS corpus (Figs. 7a and 7b). In Fig. 7b, we analyze the relationship between the impact of top 100 institutions and their COI citations. The relationship distribution indicates that the higher the impact of institution is, the more COI citations it may contain. For example, the impact and COI citations of the top ranked institution are far beyond the other institutions. Most institutions with lower COI citations have a relative smaller impact than other institutions with higher COI citations. According to Fig. 7b, we observe that the top 20 institutions are from American national laboratory and university by the impact of institutions rank. In Fig. 7a, we observe that the countries with higher impact likely have more COI citations than the countries with lower impact. By COI citations of each country rank, top 20 countries from high to low in order are: America, Germany, France, Japan, Italy, China, UK, Spain, Australia, Canada, India, Netherlands, Russia, Belgium, Brazil, Israel, Poland, Switzerland, Denmark and Sweden.

According to Fig. 7c, we observe that the COI citations of some countries suddenly increase since 1993 year, the trend of high COI citations continues until 2005. After that, COI citations of country present decreased trend. The reason behind is that COI citations of a paper are a cumulative process. The published time of papers becomes shorter, and possible COI citations are relatively smaller, therefore, we observe that around 2013, the COI citations of country become less. Fig. 7d characters the dynamic impact of scholarly papers in country-level. We observe that the impact of countries presents a sudden increase since 1993, which is consistent with Fig. 7c. What fueled this increase in the impact of country? From the statistics of the yearly summation of national publications, we observe that the number of publications grows rapidly in 1993, and some countries even increase to 10 times, compared to 1992. We believe this shift has close relationship with the ”information superhighway” strategy of the United States originated from the Clinton period. In September 1993, shortly after Clinton became the president of the United States, he officially launched the cross-century national information infrastructure project plan. This program not only had a very broad impact on the world, but also created a brilliant future for the United States information economy. In addition, a journal paper published in Nature 2013 also reported that international collaboration increased more than ten-fold since the mid-1990s, which coincides with our experimental results from another aspect [25]. Top 20 countries from high to low in order are: America, Germany, France, China, Japan, India, Italy, UK, Spain, Brazil, Russia, Canada, Netherlands, Australia, Israel, Belgium, Poland, Argentina, Korea and Switzerland by impact of country rank. Specially, America is the most prominent country, and its impact is about 3.5 times of Germany. Through such sorting results, we observe that the ranking order of country impact correlates with scientific and technological competitiveness and more details for scientific competitiveness of nations can be found in the reference [6].