Data Discovery for the SDGs: A Systematic Rule-based Approach

Data infrastructures are defined by (Kitchin, 2014) as ”…the institutional, physical and digital means for storing, sharing and consuming data across networked technologies”. Gray et al. (Gray et al., 2018) state that data infrastructures contain data sets and describe how data could be used and shared. ”Raw” data makes no sense unless people understand the context of the data, which is how it is generated and what it can be used for. Gray et al state that data infrastructures were not only data resources but were interpreted as a set of relations in terms of datasets, standards, policies, and more. Kitchin (Kitchin and Lauriault, 2014) points out that data infrastructures can be likened to recipes, where chefs describe how to use the ingredients. When considering these perspectives, we can think of data infrastructures as a means to make data available to others interested in it, such as data analysts, data scientists and software developers, and thus, to those who are likely to make the best use of the shared data.

UN ESCAP’s report describes how Big Data, IoT, and AI already have great power in their fields, and a combination of these three technologies would create more value, especially in the area of sustainable development (ESCAP, 2017). IoT serves to gather a large amount of data from loads of different sources apart from traditional methods, and the data of various types and sources are said to characterise Big Data approaches, which are analysed through AI methods to explore patterns and help with decision-making. MacFeely (MacFeely, 2019) states that if such methods were correctly applied, they would reduce the time spent on calculating SDG indicators and improve the accuracy of results. Not only have scholars noted that different types of data, and their combination thereof, have the potential for monitoring SDGs, but the UN has also realised the opportunities brought by the widespread abundance of data. There are various generalised research data sharing platforms such as Harvard’s Dataverse(King, 2007), as well as platforms for specific domains such as COPO for the life sciences (Shaw et al., 2020), and data catalogues such as FAIRsharing (Sansone et al., 2019) aim to facilitate finding relevant data within the data diaspora. Within sustainable development, the UN Global Platform was set up to support the worldwide collaboration for data sharing for the SDGs (United Nations UN-CEBD, 2022).

While efforts such as the UN Global Platform aim to provide a central resource for data sharing relating to the SDGs, it is still challenging to discover the relevant data for use in SDG activities. Even with these available data infrastructures, the there is still a gap towards organisations using data analytics to contribute towards the SDGs (El-Haddadeh et al., 2021). This paper aims to provide an approach to discovering data used in SDG research.

While data infrastructures may provide facilities to find data sources, they often rely on explicit contributions from a community or dedicated individuals to build and maintain such resources. For example, the infrastructures mentioned earlier (Dataverse, COPO and FAIRsharing) depend on researchers’ contributions to deposit or index their data, and the UN Global Platform relies on maintenance by its own staff. Research is often reported in the literature, published in journals and conferences, and indexed by bibliographic databases (e.g., PubMed, Web of Science, JSTOR). Such publications report the research outputs rather than the specifics of the data they may have used. Thus, some work is required for data infrastructures to be up-to-date and curate themselves to report on new data types and sources (for brevity, we will refer to text mentioning data types and sources as ”data entities”).

Automated data discovery has frequently been proposed in the literature (Michener, 2006; Fernandez et al., 2018; Färber and Leisinger, 2021), but its adoption in scientific research has yet to be established. There exist general-purpose search engines (e.g. Google, Bing), knowledge engines (e.g., Wolfram Alpha, Watson) and, more recently, large language model-based tools (e.g., ChatGPT, Bard) that can assist in data discovery. There also exists only one dedicated dataset search engine, Google’s Dataset Search. While these are powerful tools, they are also opaque in how they function and are designed to be general-purpose. When considering specific tasks within a specific domain, such as discovering data for particular SDGs, we often prefer a systematic and transparent approach to maintain trust in a given discovery. Thus, systematic research approaches remain relevant despite these powerful automated tools being available.

The Systematic Literature Review (SLR) is a method that typically aims to identify, evaluate, and synthesise a corpus research literature in order to address a research question that may or may not be related to the aims of the constituent research reported (Webster and Watson, 2002). SLRs are characterised by defining a reproducible method of selecting, appraising, and combining research evidence relevant to that question. This method aims to minimise bias by performing an exhaustive search and quality appraisal of the literature and by making explicit the methods used for data extraction and synthesis.

The Systematic Quantitative Literature Review (SQLR) is similar to the SLR in its approach to providing a reproducible method while being comprehensive in review and minimising bias. However, SQLRs, as the name suggests, are quantitative, aiming to quantitatively summarise and analyse the literature (Pickering and Byrne, 2014). This is through counting features of interest in the literature, such as the categories of studies published, geographical and temporal trends, types of methods used and characteristics of the results. In contrast to traditional SLRs, SQLRs do not aim to synthesise the published results but rather synthesise the studies’ characteristics.

Similar to SQLRs, but differing in aims, the Systematic Mapping Study (SMS) is a method used to provide an overview of a broad field of interest (Petersen et al., 2015). It helps to identify gaps in research, trends, and the type and quantity of evidence available, usually by quantitatively summarising and visualising a body of literature. The main aim of an SMS is to provide an overview and create a ’map’ of the research done in a particular field. SMSs do not usually delve into the relationships between study characteristics and outcomes, while SQLRs go further by quantitatively analysing these characteristics and their outcomes.

Finally, an emerging trend is in automating such systematic approaches, wholly or in part, with notable examples being the Computational Literature Review (Mortenson and Vidgen, 2016) and the Smart Literature Review (Asmussen and Møller, 2019). However, these are designed to automate performing SLR-like reviews on substantial bodies of literature by which a traditional human-performed review would be infeasible, rather than extracting specific features or entities from a corpus. Typically these use methods such as topic modelling (Blei, 2012) and bibliographic analyses (De Bellis, 2009).

While there exist various systematic approaches to discovering knowledge in literature, our research question focuses on extracting knowledge about data entities relevant to the SDG. We propose that this knowledge about data entities, which would typically be discoverable by using data infrastructures, can be extracted from the literature systematically. We describe our method for achieving this in the next section.

First, we must decide on the focus of our data entity discovery by choosing a particular SDG or SDG Target to constrain the document search. While it may be desirable to try and perform our data entity extraction for all SDGs collectively, we assume that a more valuable structure to the SDG data mapping is to place them within the constraints of each of the 17 SDGs or 169 SDG Targets. This places a practical constraint to make the analysis more feasible by limiting the scope of possible documents to analyse. One can then decide on a literature source (or sources) on which to perform the document search, as one would in a regular SMS, for example, using one or more bibliographic search engines such as the ACM Digital Library, Scopus or Web of Science. Two corpora are constructed - a first one that is manually analysed to develop the SDG data taxonomy, and a second larger corpus to apply the data taxonomy in the automated entity extraction pipeline to develop the final SDG data mapping.

After setting the search strategy and document selection criteria, and consequently compiling a corpus of documents to analyse, we perform a two-stage process to extract the data entities reported as being used in the published research. The first stage consists of manual qualitative coding on a literature sample to inductively build a taxonomy of data entity terms and their groupings. This is done through open coding of the literature followed by axial coding (Hadley, 2017) to develop the SDG data taxonomy. This taxonomy is then used to bootstrap a set of rules implemented in Python to label documents according to the taxonomy automatically. These rules are used in an automated entity extraction pipeline to label a more extensive corpus with the data entities relating to the same SDG to develop the final SDG data mapping.

Bootstrapping refers to a semi-supervised learning approach (Riloff, 2003) where a small set of manually annotated data (in our case the manually coded literature) is used to train an initial model (the resulting data taxonomy from which we develop rules), which is then used to annotate a larger set of data (the application of the rules to the expanded dataset).

Scopus was used as the bibliographic database, and the keywords ”SDG 7”, ”SDG7”, and ”data” were applied to the title, abstract, and keywords search fields, resulting in 87 candidate documents for inclusion. We then applied each selection criteria to focus the document corpus further to arrive at a final corpus of 53 documents to use in the first stage of our methodology, manual quantitative coding. A summary of the document search and selection that constitute steps M1 and M2 are shown earlier in Table 1, and the effect of applying each selection criterion is shown in Figure 1.

The final list of documents was exported from the Scopus database to an Excel spreadsheet containing the document title, abstract, keywords, and Web links to the Scopus record. The researcher then read each document online while performing open coding and added a short summary description about each in the spreadsheet, along with any data entity of interest noted (step M3). Once each document had been analysed, the open codes were reviewed via axial coding and grouped into higher-level classifications thus building up an initial SDG 7 data taxonomy (step M4).

This taxonomy organised data entities of interest into two broad groups: data sources and data types. The documents were then further organised into hierarchical classification levels based on shared traits to better understand the taxonomy.

Next, we use the SDG 7 data taxonomy from the first stage to prepare for the second stage of analysis, automated entity extraction, by developing a decision list that classifies each document entry based on the SDG 7 data taxonomy from the first stage. For example, in the manual qualitative coding stage it was found that the International Energy Agency was a relevant data entity (and data source), and this was identified by noting the words ”International Energy Agency” and its abbreviation ”IEA” within the context of a sentence about data or data sets. Based on this observation from the open coding, we model this classification computationally as a Boolean decision function in Python code. We build the decision list with similar functions for each code developed in the open and axial coding steps. The first step, A1, in the entity extraction pipeline, is to obtain the PDF file for each of the documents. This was done by manually downloading the PDFs from the relevant publisher and storing them locally in a folder on a local disk. Next, steps A2 to A4 are run where Python code loads each document from disk and converts the PDFs to text (step A2), applies the decision list classifier (step A3), and compiles the final mapping and corresponding visualisation (step A4).

In the results obtained from the automated entity extraction, the percentage of data sources and data types accounted for was nearly half, 55%, and 45%. A visualisation of the summary distribution of data entities and categories for the SDG 7 data mapping is shown in Figure 3

In the category of data sources, the number of organisations and databases that were referred to was more than twice that of traditional statistics. The survey was still the most popular conventional way of data collection among these studies. International organisations and databases were the traditional data sources, accounting for more than one-third of the total data entities. The World Bank, databases of the UN, and Eurostat were the top three frequently referred data sources, meaning they were the primary sources among the SDG 7 related papers.

Regarding the data types, the source and geographic data were discussed more often than the sensor data and weather. The proportion of diverse resource and geographic data chosen in the research was around 95% of data types. Data on electricity, land use, solar, and water use were the most popular being used to monitor the progress of SDG 7.