This paper was converted on www.awesomepapers.org from LaTeX by an anonymous user.
Want to know more? Visit the Converter page.

Towards Sustainability of Systematic Literature Reviews

Vinicius dos Santos, Anderson Yoshiaki Iwazaki
University of São Paulo (ICMC-USP)
São Carlos - São Paulo - Brazil
{vinicius.dos.santos, iwazaki.anderson}@usp.br
\AndKatia Romero Felizardo, Érica Ferreira de Souza
Federal University of Technology - Paraná (UTFPR)
Cornélio Procópio - Paraná - Brazil
{katiascannavino, ericasouza}@utfpr.edu.br
\AndElisa Yumi Nakagawa
University of São Paulo (ICMC-USP)
São Carlos - São Paulo - Brazil
[email protected]
Abstract

Background: The software engineering community has increasingly conducted systematic literature reviews (SLR) as a means to summarize evidence from different studies and bring to light the state of the art of a given research topic. While SLR provide many benefits, they also present several problems with punctual solutions for some of them. However, two main problems still remain: the high time-/effort-consumption nature of SLR and the lack of an effective impact of SLR results in the industry, as initially expected for SLR. Aims: The main goal of this paper is to introduce a new view — which we name Sustainability of SLR — on how to deal with SLR aiming at reducing those problems. Method: We analyzed six reference studies published in the last decade to identify, group, and analyze the SLR problems and their interconnections. Based on such analysis, we proposed the view of Sustainability of SLR that intends to address these problems. Results: The proposed view encompasses three dimensions (social, economic, and technical) that could become SLR more sustainable in the sense that the four major problems and 31 barriers (i.e., possible causes for those problems) that we identified could be mitigated. Conclusions: The view of Sustainability of SLR intends to change the researchers’ mindset to mitigate the inherent SLR problems and, as a consequence, achieve sustainable SLR, i.e., those that consume less time/effort to be conducted and updated with useful results for the industry.

Keywords Systematic Literature Review, SLR, Sustainability, Sustainability of SLR

1 Introduction

The community of software engineering (SE) has increasingly adopted Systematic Literature Review (SLR) in recent years [1] as a reliable method to summarize evidence from a number of studies and find out the state of the art in a given research topic [2, 3]. SLR has presented many important benefits, including the possibility of dealing with information from different studies in an unbiased manner [3, 4], producing auditable and repeatable results [5, 6], and identifying research gaps and also perspectives for future investigations [3].

At the same time, SLR has also suffered from diverse problems. Studies reported some of them, e.g., poor documentation of SLR conducted [7, 8] and lack of rigor in following well-experimented guidelines for SLR directly impacting the SLR quality [9]. Important solutions have been proposed for specific problems, such as techniques for better planning SLR [10, 11], conducting SLR [12], reporting [13], and adopting supporting tools [14]. However, two main problems still remain with no suitable solutions: (i) the SLR conduction is still a very time- and effort-consuming task [15]; and (ii) the lack of effective impact of SLR results in the industry [16]. This current scenario of SLR leads the SE community to question if it is worth continuing to invest in the SLR conduction. Hence, the research question addressed in this work is: Is it possible to reduce the high consumption of resources (time and effort) while assuring to achieve more accessible and useful SLR for the industry? Another question that intrigues us is: Should the SLR researchers change the mindset on how to better deal with SLR conduction and dissemination?

The main goal of this work is to introduce a new view — which we name Sustainability of SLR — to better deal with SLR and reduce those two main problems (high resources consumption and lack of impact in industry) and, as a consequence, achieve sustainable SLR (i.e., SLR that consume fewer resources to be conducted and updated with useful results for the industry). To achieve our goal and answer the research question, we carefully examined six reference studies published in the last decade [8][6][17][18][19][7] to identify, group, and systematically analyze the main problems in both SLR conduction and update. We found four groups of problems and 31 barriers (i.e., possible causes of the problems). Following this, we proposed the view of Sustainability of SLR, which connects these problems among them as well as their causes and encompasses three perspectives (social, economic, and technical) that could somehow deal with the SLR problems. The social perspective addresses human aspects, such as reviewers’ communication and stakeholders’ participation during the SLR conduction. The economic one is related to the resources (effort and time) to conduct and update an SLR. The technical perspective is related to the supporting tools and technologies used to conduct and update SLR.

It is worth highlighting that this prospective view does not intend to solve all current problems of SLR; instead, the main contribution of this work is to raise the awareness of researchers to change the mindset when proposing new solutions for SLR problems through a more holistic view.

The remainder of this paper is organized as follows: Section 2 presents the research method. Section 3 presents the SLR problems identified in the literature and their possible causes; Section 4 presents the view of Sustainability of SLR; Section 5 discuss some challenges for achieving the Sustainability of SLR as well as the threats to validity; and Section 6 presents the final remarks.

2 Research Method

To propose the view of Sustainability of SLR, it was necessary first to survey the literature and understand the current problems of both SLR conduction and update. For this, we selected six reference studies published over the last decade (2010 - 2020) [8, 6, 7, 19, 18, 17] based on our group experience in conducting SLR. We also used experts’ opinions to validate our choice and mainly avoid bias.

The six studies selected are tertiary studies (SLR of secondary studies) and experience reports from researchers’ perspectives to identify the main problems of conducting and updating SLR. Riaz et al. (2010) [19] presented an experience report involving three Ph.D. students that identified the main difficulties of novices while they conduct SLR and compared the results with SLR expert experience. Imtiaz et al. (2013) [17] conducted a tertiary study that evaluated 116 secondary studies published from 2005 to 2011, summarized the experience of researchers, and highlighted the challenges in conducting SLR. Kitchenham and Brereton (2013) [18] conducted a tertiary study that assessed 68 secondary studies published from 2005 to 2012 to summarize the researchers’ opinions about their experiences of conducting SLR and discuss techniques that could be used to improve the SLR process. Bugden et al. (2018) [6] conducted another tertiary study assessing 178 studies published from 2010 to 2015 summarizing lessons learned, researchers’ main problems, and focusing on how good SLR could be reported.

We can also interpret threats to validity as problems in SLR that could not be solved providing clues about the researchers’ difficulties during the SLR conduction. Hence, two tertiary studies [8, 7] identified and discussed the most common threats to validity and strategies to mitigate them. Zhou et al. (2016) [7] assessed 316 studies published from 2004 to 2015 and Ampatzoglou et al. (2019) [8] analyzed 165 studies published from 2007 to 2016.

Refer to caption
Figure 1: Research method

Based on the six studies aforementioned, we performed the four main steps, as shown in Figure 1, which summarized the research method adopted. In Step 1, we systematically identified the main problems that occur during the SLR conduction and reported in the six studies. We also deeply checked the studies cited by these six studies. In Step 2, to understand the source of those problems, we carefully identified the current barriers, i.e., the possible causes for these problems, that could be hampering authors to conduct good and effective SLR. In Step 3, we deeply analyzed and summarized those barriers removing duplication and classifying them111Replication package: https://github.com/CSM-Research/06-LR-TSInSLR according to three dimensions (or perspectives): social, economic, and technical. This classification was based on well-known sustainability dimensions already explored in other areas [20, 21, 22, 23, 24, 25], as described below:

  • •

    Social barriers are mainly associated with human aspects of SLR (e.g., communication, culture) and how SLR results can impact the society (academia and industry);

  • •

    Economic barriers are directly associated with the high consumption of time and effort while planning, conducting, or updating SLR; and

  • •

    Technical barriers address problems with tools and technologies used to support SLR conduction.

It is also worth highlighting that Steps 1, 2, and 3 were performed considering our long-term experience researching, conducting, and updating SLR. Besides, we based on evidence from the literature to make decisions. Finally, in Step 4, we analyzed the SLR problems and proposed a new view that connects these problems and their interactions as well as a possible means to deal with these problems. For this, we inspired ourselves on the views of sustainability from other areas, in particular, the ecology area [26] and SE area [20], to derive the view of Sustainability of SLR.

3 SLR problems and barriers

After conducting Steps 1 to 3 of our research method, Table 1 summarizes the four main problems were identified: (i) lack of industry’s interest in SLR results, (ii) SLR conduction is still very, time and effort consuming (iii) poor documentation of SLR, (iv) lack of SLR’s quality. For each large problem, we identified the barriers (i.e., possible causes for these problems and listed in column 4 of Table 1) and classified them in social, economic, or technical dimensions (shown in column 3).

Table 1: Problems of SLR and possible causes
Problems ID Dimension Barriers (Possible causes) Reference
Lack of industry’s interest in SLR results 1 Economic/Social Data synthesis, interpretation, and presentation of results are damaged when researchers have different interpretations of data extracted or when are conducted by a single researcher. [8] [19] [7]
2 Social Cultural differences may impact the validity results [19] [7]
3 Social Inappropriate research questions are defined [8] [17] [18] [19] [7]
4 Social SLR does not present generalizable results which can be used in industry or other areas [8][6]
SLR conduction/update is still very time- and effort-consuming 5 Economic Lack of standardization and incorrect/incomplete keywords hinder the search string construction [8][17] [18][19][7]
6 Economic Inadequate number of studies are evaluated hindering the validity of results [19] [7]
7 Economic/Technical Use of not credible or very specific/broad digital databases may return many irrelevant studies or miss relevant studies [8][17] [18] [19] [7]
8 Economic/Technical Primary studies may be duplicated in different databases [7]
9 Economic/Technical Inaccessibility of resources (papers/databases) [7]
10 Economic/Technical Database limitations and inefficiencies (e.g., interface, search string syntax) [17] [18][19]
11 Economic/Technical Need tool to support SLR [17][18]
Poor documentation of SLR 12 Economic Lack of information about the SLR conduction process (e.g, initial date, duplicated studies). [6] [18] [19]
13 Economic Lack information about the use of inclusion/exclusion criteria. [6] [18]
14 Economic Insufficient details/report about the quality evaluation [6]
15 Economic Additional search is not being reported correctly [6]
16 Economic Lack of details about the primary studies selected (e.g. context, participants, source material) [6]
17 Economic Lack of information about the exclusion of studies impacting SLR data extraction and synthesis [6]
18 Economic Studies do not specify important SLR details, causing problems to repeatability and replicability [8][19][7]
19 Economic/social The synthesis process is not clear [6] [19]
20 Economic/social Lack information about the reviewers’ participation during the SLR conduction process [6][17][19]
Lack of quality of SLR 21 Economic Many studies do not perform the quality evaluation or do not use it correctly in selection process. [6] [19]
22 Economic Few studies had a good search coverage. [6]
23 Economic Inappropriate search methods leading to problems in SLR coverage [17][7]
24 Economic Selection process can be difficult when inclusion/exclusion criteria are generic or inappropriate. [8][7]
25 Economic The wrong classification of the primary studies may cause the secondary study to lack robustness. [8][19][7]
26 Economic Delimiting a time span affect the coverage of SLR [7]
27 Economic Lack of unpartiality of researchers resulting in bias in study selection [7]
28 Economic Extraction process is difficult when quality assessment is biased by reviewer subjectivity or they do not completely understand the data extraction items [8][7]
29 Social Lack of expert evaluation of the results [18] [19] [7]
30 Economic Data model and data extraction forms may change during extraction [18]
31 Economic Difficulties in deciding when to stop the piloting process [19]

For the first problem “Lack of industry’s interest in SLR results”, four social barriers were identified and one can be also interpreted as economic. Studies reported that SLR are conducted using inappropriate research questions [8, 17, 18, 19, 7] and highlighted that the cultural differences among researchers threats the validity of results (e.g., preferences for the studies of some researchers’ nationality) [7, 19]. In addition, the lack of a clear synthesis of SRL results and lack of practical recommendations for SE practitioners impact somehow the generalizability of the results and hamper the use of these results in the industry and academia [6, 8].

For the second problem “SLR conduction/update is still very time- and effort-consuming”, seven barriers were found and all of them are economic barriers, while five of them can also be interpreted as technical barriers. A possible cause for this problem is the lack of standardization of terms to be used in the search string (as the case of SE area [7, 8, 18, 19, 17]) and as consequence, the search might return a large number of primary studies (including many irrelevant ones) or miss some relevant studies [8]. Another cause is the inefficiency of electronic databases [17, 19, 18] and, despite the improvements in many aspects over the years (e.g., usability, search engine, coverage, interface, etc), several studies still mention problems indicating that the choice of databases is still a challenge [8]. Five studies mention that non-credible or very specific or broad databases can return many irrelevant studies or can even miss relevant studies, increasing the effort for conducting SLR [7, 8, 18, 17, 19]. In this context, many supporting tools were developed to mitigate such problem [27], especially aiming at reducing the additional effort caused by databases inefficiency or integration problems (e.g., studies indexed by multiple databases). However, many of these tools solve only punctual problems. Hence, the lack of reliable and integrated tools is still considered an important factor for the excessive time and effort consumption.

For the third problem “Poor documentation of SLR”, nine economic barriers were found and two of them can also be interpreted as social barriers. Results indicate a lack of commitment of researchers in following the well-experimented guidelines proposed for reporting SLR [6]. Three studies reported that SLR do not report important details about the conduction process that allows SLR to be updated or replicated [6, 18, 19]. The poor documentation directly impacts academia and industry by creating a lack of credibility and problems in auditability and reproducibility of SLR [7, 8, 19]. In addition, this problem extends to the social dimension since a clear data synthesis is considered very important to disseminate useful results of SLR to industry and also academia [6].

For the fourth problem “Lack of quality of SLR”, 10 economic barriers and one social barrier were found. The economic barriers indicate that authors have difficulty in following the guidelines proposed for SLR and ensuring the quality of selection [7, 8], data extraction [19, 8], synthesis and documentation [6]. The wrong classification of the primary studies or application of inappropriate search methods, for example, cause problems in the SLR coverage and robustness. In addition, the lack of quality can also be considered a social problem because it is recommended that SLR include experts to evaluate the SLR results [18, 19, 7], because when they are not involved in the SLR process, results cannot be adequately synthesized, even generalized, and useful for SE practitioners.

Observing from a broader perspective, all problems and their possible causes mentioned previously are somehow interconnected among them. For example, using the economic perspective, the poor documentation hampers the auditability of SLR, resulting in a lack of confidence in the SLR results [7, 8, 19]. It also hampers the repeatability of SLR; hence, SLR cannot be easily updated [5], and researchers need sometimes to reconduct the SLR from “scratch” and, as a consequence, consuming extra resources (time and effort). This problem is also connected with a social problem because the low quality of SLR also directly impacts the confidence in the results and generates doubts in readers about biases that could be occurred during the SLR conduction. Jointly to the lack of practical recommendations for SE practitioners in such documentation [16, 6], in many cases, SLR are not used source of evidence for decision-making process [13], which could improve the productivity of industry and academia. Aiming to reduce the time and effort spent on the SLR conduction and deliver fast results, researchers have ignored important well-experimented guidelines [6], which could ensure such quality, and have created poorly documented SLR with shallow results, which are not useful in the end.

Based on the set of problems, their dimensions, and barriers, we proposed a new view that could suggest to the researchers how to better deal with these problems in an integrated way, as presented in the next section.

4 Sustainability of Systematic Literature Review

Before presenting the view of Sustainability of SLR, we provide a brief overview of the topic of sustainability that has been already researched in other areas and has served as a basis for our work.

4.1 Brief Overview of Sustainability

Sustainability started in the ecology context [23] and became popular in 1987 when the UN World Commission on Environment and Development published the report “Our Common Future” (Brundtland Report) defining “sustainable development” as a development that meets the needs of the present without compromising the ability of future generations to meet their own needs [22].

Next, computer science researchers realized that sustainability could be adapted to other contexts, such as SE [28]. In 2009, the International Conference on Software Engineering (ICSE) started a special track called “software engineering for the planet”222https://www.cs.uoregon.edu/events/icse2009/specialSessions/#planet discussing how software can interact with sustainability, including energy consumption and the impact of software production in society [28]. In the next years, the term “Sustainable Software” was interpreted in two ways [29]: (i) the software code being sustainable; and (ii) the purpose of the software being to support sustainability goals, i.e., to improve the sustainability of humanity on our planet. Both interpretations must coincide in a software system that contributes to a more sustainable life, i.e., sustainable software is energy efficient, minimizes the environmental impact of the processes it supports and has a positive impact on social sustainability and/or economics. In 2015, the Karlskrona manifesto333http://www.sustainabilitydesign.org/ was signed [20], creating awareness of the SE community on the need for sustainability and becoming a non-functional requirement for software [30] and aiming to support software engineers to reduce unnecessary resource consumption.

To adequately address sustainability, it is usually broken down in different dimensions [23]. The Brundtland Report [22] described sustainability in ecology with three dimensions: social, economical, and environmental. Although these dimensions have become mainstream throughout the literature, they are not universal [23]. Some authors consider necessary to have additional dimensions (e.g., institutional [24], cultural [25], or technical [21]). In the SE community, the Karlskrona Manifesto included the individual and technical dimensions to maintain human capital and ensure the longevity of information, systems, and infrastructure and their adequate evolution with changing surrounding conditions.

In summary, software construction has a well-defined process, consumes time and effort, uses human resources, and its success depends on efficient tools, human skills, and social aspects (e.g., participation of stakeholders). In this perspective, researchers in the SE area already understood that this problem must be solved using a systemic solution [20]. Coming from this context, we have used all knowledge previously accumulated and well-experimented in the environmental and SE area to propose a view of sustainability that could be valid to SLR.

4.2 Sustainability of SLR and their Dimensions

Considering that the problems and barriers found in this work are associated with economic, social, and technical perspectives, we broke down the Sustainability of SLR into these three dimensions, as described below:

  • •

    Social dimension: it addresses the human aspects associated with SLR, such as communication and collaboration, and ensures effective participation of stakeholders in the SLR process. This dimension focuses on ensuring that current and future stakeholders (including both, academic community and SE practitioners) have open access to the SLR results and ensuring that these results could be easily used.

  • •

    Economic dimension: it preserves the resources (time and effort) during the conduction, audition, and update or replication of SLR by creating high-quality SLR, i.e., this dimension ensures high-quality documentation, credible databases, efficient search engines, and rigor in following the guidelines to preserve reproducibility and auditability, minimizing unnecessary efforts.

  • •

    Technical dimension: it preserves the reliability of supporting tools used to conduct SLR, facilitating their use for the update or reconduction of those SLR. This dimension must assure technical ways to minimize the efforts by automating tasks and supporting their execution.

Based on these three dimensions, the view of Sustainability of SLR can be understood as “the process and a set of actions to make it possible to preserve SLR that endure over time (i.e., longevity) with less possible time and effort consumed and an effective impact to the industry.”

Regarding other sustainability dimensions, we believe they are not directly related to SLR. For instance, the environmental dimension handles the impacts of human activities in natural resources (e.g., water, land, air) or, in SE, it manages the excessive amount of energy consumption caused by software. We did not include such a dimension because there is no clear connection between SLR and the environment. Therefore, for the while, we believe that the three dimensions seem to be sufficient to connect SLR problems and understand their connections and possible ways to achieve sustainable SLR.

Following this, we develop a discussion about how to better represent graphically the Sustainability of SLR. For this, we revisit studies in the literature from other areas [31, 22, 23] and observed that the sustainability dimensions are described interconnected among them [23]. Therefore, an important discussion in the sustainability context is how to make a fair graphical representation including all dimensions defined and how they interact with each other. The Brundtland Report [22] proposed a representation (i.e., a Venn diagram with interlaced circles and the intersection between them) that became the most popular one for sustainability in the ecological area. The key idea is that human society is only sustainable if it can be sustained in all dimensions.

Figure 2 shows a graphical representation of Sustainability of SLR using the Venn diagram. Figure 2.A represents the Ballanced View with an intersection of all dimensions. We can observe that attributing the same weight for all dimensions and balancing them to achieve sustainability may lead to questions about its feasibility, due to the large number of factors that impact each dimension. An alternative is the Unballanced View that considers the technical dimension as a common concern that cannot be dissociated from social and economic perspective, as shown in Figure 2.C.We consider that both representations (shown in Figures 2.A and 2.C) present a “Weak Sustainability” (which was previously defined in [32]) because solving all problems associated only with, for instance, social dimension does not assure to achieve a sustainable SLR.

In an opposite perspective, Giddings et al [31] proposed a different representation that considers that dimensions must be nested and with different weights. This representation assumes that it is not possible for the society and economy to develop outside the biosphere [33] and this perspective warns that a finite planet cannot sustain human life with an economy that intends unlimited growth. It is necessary to acknowledge that there are fundamental biophysical limits that constrain the natural resources on the planet [34]. This perspective arranges the sustainability dimensions providing a hierarchic view, i.e., problems of each dimension must be treated by different disciplines and each decision must consider the constraints of each dimension [20] and this representation was later called “Strong Sustainability” [33] .

Figure 2.B presents the Sociocentric View that nests the dimensions considering the social dimension is the broadest and technical one is the most specific, so attributing different weights for them. This representation assumes technical problems are essentially connected with economic problems, and the social constraints must guide the decision-making of researchers while conducting SLR. Despite this representation seems to be fair for SLR, it may be a problem to put social priorities higher than economic needs. For example, the software industry (which is tightly associated with the social dimension) needs to speed up the SLR conduction; however, it may cause a lack of rigor in documentation and reliability that could directly impact the feasibility and value the of SLR and, as a consequence, it may result in unsustainable SLR. Figure 2.D presents the Econocentric View that considers that social needs are constrained by economic aspects. Due to the resources constraints sometimes imposed by the economic dimension, researchers are prone to conduct SLR ignoring the social concerns. An example is not to include a given search question that could indeed answer the doubt from a wider community only to speed up the SLR process.

In the end, the graphical representations of Sustainability of SLR intend to provide us the opportunity for better understanding the inherent difficulties to create hierarchies among the dimensions and to better balance them. Considering the current scenario of SLR, we believe that it is necessary to move backward and recognize that the problems cannot be solved without integrated thinking.

Refer to caption
Figure 2: An View of Sustainability of SLR

5 Discussion

We believe that this paper provides a good starting point and raises awareness of the community about the current problems of SLR from a holistic panorama. Many of the barriers have been mentioned since 2010 but not solved yet, in particular, the need for adequate synthesis of SLR results to be suitable for the industry. Our work indicates there is also a lack of: (i) knowledge about the SLR conduction process; (ii) application of the good practices for SLR conduction; (iii) details and important information to reproduce and update SLR; and (iv) tools’ integration. We also observed many barriers found are related to the conduction process, while few social barriers are reported indicating a lack of knowledge about social problems.

We believe this paper is provocative in the sense that it could bring a disruptive view and, as a consequence, new and coordinated actions, and solutions to make SLR reduce the high consumption of resources while producing useful results for the industry and academia. In this context, create effective methods to include the stakeholders (i.e., members of the industry) could be one of the possible solutions for social dimension problems. Economic dimension must create feasible methods to support the production of high-quality SLR consuming fewer resources (time and effort) to the conduction and update. For this, it is necessary to overcome many challenges including the researcher’s mindset about the importance of correctly documenting SLR using the best practices available for reporting and include SLR sustainability as a quality criterion. The technical dimension must support tools and technologies used to plan/conduct/report aiming to ensure its quality, accessibility, and integration. Furthermore, this dimension must handle the slow and fragmented development of tools to support SLR and propose solutions towards integration. For this, it is necessary an international collaboration to provide long-term support and integration of SLR tools. Finally, the main challenge of Sustainability of SLR is to change the researchers’ mindset about the importance of considering this integrated and holistic view while conducting SLR and proposing new solutions to overcome the current problems.

Regarding the threats to the validity, we did not perform an exhaustive search for studies that could have reported other problems and barriers in SLR. It is possible some problems and barriers have been missed. Besides, our work considered a 10-year period (2010-2020); hence, it is possible that problems and barriers reported before this period have not been considered. To mitigate these two threats, we used the experts’ opinions to validate the set of studies considered in our analysis. Moreover, the results presented in this work could have been influenced by our knowledge and experience in researching and conducting SLR. To mitigate this threat, we strictly based on evidence from the literature to make any decisions.

6 Final Remarks

The academic community has increasingly conducted SLR, but they still suffer from critical problems, and the several punctual solutions being proposed for those problems have not been widely effective. This paper is placed in this context and intends to somehow be provocative to call the attention of this community to change its mindset regarding the conduction and use of SLR results. Derived from the sustainability concept from other areas, the Sustainability of SLR primarily intends to achieve sustainable SLR, i.e., low consumption of resources (time and effort) and with useful results for the industry. As future work, the community needs to work together to promote sustainable SLR and, as a consequence, to achieve the initial goal of SLR that is the support of the industry.

7 Acknowledgements

This study was supported by FAPESP (grants: 2015/24144-7, 2019/23663-1), Capes (grants: PROEX-11308091/D, PROEX-11357580/D), and CNPq (grant: 312634/2018-8).

References

  • [1] E. Mendes, Wohlin. C., K. Felizardo, and M. Kalinowski. When to update systematic literature reviews in software engineering. Journal of Systems and Software, 167:110607, 2020.
  • [2] K. Felizardo, E. Y. Nakagawa, S. Fabbri, and F. Ferrari. Systematic Literature Review in Software Engineering: Theory and Practice (In Portuguese). Elsevier, 2017.
  • [3] B. Kitchenham, D. Budgen, and P. Brereton. Evidence-based software engineering and systematic reviews, volume 4. CRC press, Boca Raton, USA, 2015.
  • [4] M. Niazi. Do systematic literature reviews outperform informal literature reviews in the software engineering domain? an initial case study. Arabian Journal for Science and Engineering, 40(3):845–855, 2015.
  • [5] B. Kitchenham, P. Brereton, Z. Li, D. Budgen, and A. Burn. Repeatability of systematic literature reviews. In EASE, pages 46–55, 2011.
  • [6] D. Budgen, P. Brereton, S. Drummond, and N. Williams. Reporting systematic reviews: Some lessons from a tertiary study. Information and Software Technology, 95:62–74, 2018.
  • [7] X. Zhou, J. Yuqin, H. Zhang, S. Li, and X. Huang. A map of threats to validity of systematic literature reviews in software engineering. In APSEC, pages 153–160, 2016.
  • [8] A. Ampatzoglou, S. Bibi, P. Avgeriou, M. Verbeek, and A. Chatzigeorgiou. Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Information and Software Technology, 106:201–230, 2019.
  • [9] M. Kuhrmann, D.M. Fernández, and M. Daneva. On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empirical Software Engineering, 22(6):2852–2891, 2017.
  • [10] L. Cairo, G. Carneiro, M. Monteiro, and F. Abreu. Towards the use of machine learning algorithms to enhance the effectiveness of search strings in secondary studies. In SBES, pages 22–26, 2019.
  • [11] K. Felizardo, E. Souza, R. Falbo, N. Vijaykumar, E. Mendes, and E. Nakagawa. Defining protocols of systematic literature reviews in software engineering: A survey. In SEAA, pages 202–209, 2017.
  • [12] K. Felizardo, E. Souza, S. Hesae, N. Vijaykumar, and E. Nakagawa. Analysing the use of graphical abstracts to support study selection in secondary studies. In CIbSE, pages 497–510, 2017.
  • [13] B. Cartaxo, G. Pinto, and S. Soares. The role of rapid reviews in supporting decision-making in software engineering practice. In EASE, pages 24–34, 2018.
  • [14] C. Marshall and P. Brereton. Tools to support systematic literature reviews in software engineering: A mapping study. In ESEM, pages 296–299, 2013.
  • [15] K. Felizardo and J. Carver. Automating systematic literature review. In Contemporary Empirical Methods in Software Engineering, pages 327–355. 2020.
  • [16] D. Badampudi, C. Wohlin, and T. Gorschek. Contextualizing research evidence through knowledge translation in software engineering. In EASE, pages 306–311, 2019.
  • [17] S. Imtiaz, M. Bano, N. Ikram, and M. Niazi. A tertiary study: experiences of conducting systematic literature reviews in software engineering. In EASE, pages 177–182, 2013.
  • [18] B. Kitchenham and P. Brereton. A systematic review of systematic review process research in software engineering. Information and Software Technology, 55(12):2049–2075, 2013.
  • [19] M. Riaz, M. Sulayman, N. Salleh, and E. Mendes. Experiences conducting systematic reviews from novices’ perspective. In EASE, pages 44–53, 2010.
  • [20] C. Becker, R. Chitchyan, L. Duboc, S. Easterbrook, B. Penzenstadler, N. Seyff, and C. Venters. Sustainability design and software: The karlskrona manifesto. In ICSE, pages 467–476, 2015.
  • [21] R. Hill and P. Bowen. Sustainable construction: principles and a framework for attainment. Construction Management and Economics, 15(3):223–239, 1997.
  • [22] B. Keeble. The brundtland report:‘our common future’. Medicine and war, 4(1):17–25, 1988.
  • [23] B. Purvis, Y. Mao, and D. Robinson. Three pillars of sustainability: in search of conceptual origins. Sustainability Science, 14(3):681–695, 2018.
  • [24] J. Spangenberg. Towards indicators for institutional sustainability: lessons from an analysis of agenda 21. Ecological Indicators, 2(1-2):61–77, November 2002.
  • [25] K. Soini and I. Birkeland. Exploring the scientific discourse on cultural sustainability. Geoforum, 51:213–223, 2014.
  • [26] B. Brown, M. Hanson, D. Liverman, and R. Merideth. Global sustainability: Toward definition. Environmental Management, 11(6):713–719, 1987.
  • [27] C. Marshall and P. Brereton. Systematic review toolbox: a catalogue of tools to support systematic reviews. In EASE, pages 1–6, 2015.
  • [28] L. Hilty, W. Lohmann, and E.M. Huang. Sustainability and ict - an overview of the field. Notizie di Politeia, 27(104):13–28, 2011.
  • [29] B. Penzenstadler, A. Raturi, D. Richardson, C. Calero, H. Femmer, and F. Franch. Systematic mapping study on software engineering for sustainability (SE4s). In EASE, pages 1–14, 2014.
  • [30] K. Raisian, J. Yahaya, and A. Deraman. Current challenges and conceptual model of green and sustainable software engineering. Journal of Theoretical and Applied Information Technology, 94:428–443, 2016.
  • [31] B. Giddings, B. Hopwood, and G. OB́rien. Environment, economy and society: fitting them together into sustainable development. Sustainable Development, 10(4):187–196, 2002.
  • [32] M. Gutés. The concept of weak sustainability. Ecological Economics, 17(3):147–156, 1996.
  • [33] I. Morandín-Ahuerma, A. Contreras-Hernández, D.A. Ayala-Ortiz, and O. Pérez-Maqueo. Socio–ecosystemic sustainability. Sustainability, 11(12), 2019.
  • [34] E. Neumayer. Weak versus Strong Sustainability. Edward Elgar Publishing, 2013.