Geo-Spatial Data Visualization and Critical Metrics Predictions for Canadian Elections

For this case study, we require to extract validated dataset of Canadian elections from different resources. The database that is developed by Dr. Anthony Sayers at the Department of Political Science at the University of Calgary is one of the most reliable, consistent, and accessible databases for Canadian elections [6]. The details of the database are given here: http://canadianelectionsdatabase.ca/ This database contains information on federal, provincial, and territorial elections since 1867 and is arranged by-election, party, candidate, and district. The database allows users to explore the data in numerous ways. We have chosen this database as it contains around four million data points [6], and represents the difficulties of interpreting open data for users.

A challenge of working with this data is the lack of having a download option to retrieve and use this information (e.g., import to analysis tools). Therefore, we developed a scraper to scrape, extract necessary data, and store them in an appropriate format, to reduce further data cleaning and polishing, whereas storing data as it is found in the database may lead to exhaustive data-formatting for the re-usability purpose. The extracted data is intended to be fed to the other two components. An example of one of the tables on this data (from the database website) is shown in the top part of the Fig. 1. After the whole process of scraping, we are going to acquire all the data stored in the mentioned web-embedded tables and save them as .csv files (shown in the bottom part of Fig. 1) so that the data can be easily accessed later for the use of data interpretation or visualization tool.

We have chosen Python to develop the scraper component as it provides a powerful, robust package for web-crawling and data scraping, namely ”Beautiful Soup.” Beautiful Soup is a Python package for parsing HTML and XML documents (including having malformed markup, i.e., non-closed tags, so named after tag soup). It creates a parse tree for parsed pages that are used to extract data from HTML web pages [7]. Beautiful Soup package provides methods to return/download the HTML page upon sending a request to the server with the corresponding URL. From the downloaded HTML page, we can search, identify, and retrieve required components (document segments) of the page such as ”table” using their designated class or id. Once we locate the required element, the data is parsed into a list to be stored later in a .csv file. The intermediate list helps to modify huge chunks of data as per the requirement of other components in the project (for convenience and reusability). The scraped election data is stored as census-district-level data and province-territory-level-data, shown in the top part of Fig. 1. We have named different tables storing different election information in the following format ”$\langle election-type\rangle$ _ $\langle year\rangle$” where election-type refers to either Federal or Provincial election and year refers to the year when the election took place.

A particular challenge was that some pages were communicating with the database using dynamic AJAX and JS requests. Beautiful Soup does not provide support for such scenarios. These pages were inspected separately for the specific requests, which would, in turn, return the data that we require. To retrieve the data that is fetched by AJAX or JS request, our code listens on the port that communicates with the database and downloads the database response. To accomplish this task, we utilized another powerful, robust package Selenium that enabled us to capture all the AJAX and JS responses. Lastly, these methods are highly dependent on the ”connection-time.” If the response is not received within a specified period, the method terminates listening to the port. A try-catch block captures the method and converts the connection-time to be an incrementing one-hot-encoding variable to address this issue. As a result, for a specified connection-time, if the method can not capture a request, it would try again with an increased connection-time and returns to base value after each loop.

The static and dynamic pages are handled differently, as explained above. Each program extracts and stores the retrieved data separately.

For this part of the project, we have developed a Choropleth map (i.e., a map in which we shaded the areas in proportion to a statistical variable) of Canada and integrate it with the election information. The gist of the process is grabbing a .shapefile (geospatial vector data format for geographic information system software) and converting it to simple-features objects to be used in R.

The tidyverse is used as the core package to convert map data into a plot. Converting map data into a format (that R packages can use) requires a lot of different technical steps as they cannot be used directly with the provided methods in tidyverse libraries. Other packages: sf, rgdal, geojsonio, spdplyr, rmapshaper are also used, which provide functionalities for conversion and mapping process. As part of this process, we have built a separate function, theme_map(), as a ggplot theme that turns off insignificant pieces of the plot (so that it looks neat).

In the next step, the actual map data (.shapefile) of different regions is extracted for developing a Canadian map. We collected the data from the Canadian Central Statistics Agency: http://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2011-eng.cfm. This Shape File format (.shp) is the most popular and widely-used standard format for map data. In our tool, we selected ArcGIS .shp files as they contain the category Census divisions and cartographic boundaries, and it is convenient for the integration of the election data. These shapefiles are imported in our R script as an object using the readOGR function of rgdal package and then are converted into GeoJSON format to simplify the polygons. This GeoJSON file becomes the building block for further components. After that, we read the GeoJSON file back as an sf (simple features) object.

These steps of data importing, converting, and thinning take a long time to execute, the resulting data are saved for further use and are made available for the use of other researchers. the .geojson the file is also included in the repository, so for the test purposes, one can load the data and start working form this point.

Fig. 2, 3, 4 represent the generated maps without the merging of the election data. The first part, Fig. 2 shows the map with our putting any color code throughout the region; the second part, Fig. 3 shows the map with color-coded provinces and the third part, Fig. 4 shows a color-coded district throughout the Canadian region.

Finally, the scraped election data stored as census-district-level data and province-territory-level-data are converted into data frames and merged separately with the map-data produced as a sf (simple- feature) object, namely canada-cd (this is essentially a big data frame specifying a large number of lines that need to be drawn on the plot).

This merging step required careful attention while matching the key variables to avoid introducing missing values; otherwise, the lines on the map would not have smoothly joined. If missing values are introduced, it would have resulted in a shredded map as R tries automatically to fill the missing portion of the polygons [8].

That is why it is worth mentioning that for plotting data on maps, joining two datasets on the character attributed column should be avoided [8]. An example is provided to illustrate the issue. If we had taken PRNAME (province name) column to merge the data, it would introduce null values in the resulting- data frame as rows corresponding to the province ”British Columbia” is referred as ”British Columbia (BC)/Colombie britannique” in the election data frame but ”British Columbia (BC)” in the corresponding map data frame. Broken Maps are usually caused by these kind of merge errors. Another example can be, one of the province names could contain a leading or trailing space as a result of data- extraction limitations (like ”Alberta” and ”Alberta” which would cause the join to fail). Therefore, another attribute is used to merge the data (in our case province id- PRID) as it has a numeric value. Other issues that we have considered for plotting maps is converting the numeric columns back to factors and using specific data type (discrete or continuous) to avoid receiving errors.

Once the joining is complete, our R script generates the map with conventional ggplot by filling the PRID (Province ID) for representing federal elections or the CDUID (Census-District ID) for provincial elections. Our interactive platform lets the user choose any election (federal or provincial) and the year. We just select a specific table storing the specified information, merge the table with map-data, and generate the Choropleth map for visualization.

For demonstration purposes, we have randomly selected one map (for the sake of brevity) that colors the regions based on the winner parties in the latest Province and Territorial election and shown the representation in Fig. 5. This figure is produced from data shown in Table I. At the bottom of this figure, all the party names are assigned different colors and then different regions of the Geospacial map are painted with the same color of the associated party won in that particular region. From the bottom right corner of the figure, we have attached the associated table-data that was retrieved from the web-embedded database.

Please note that this map is selected randomly from over 400 maps generated from the scraped data (31 elections for each of the 13 provinces).

Visualizing election-data on maps is helpful, but it cannot provide a lot of the required information and analysis. We developed the last component of our tool architecture to mitigate the map limitations and provide precise predictions on particular metrics for both federal and provincial elections [9]. We have used Tableau to generate different graphs of users’ choice.

A demonstration on how user can choose from different data-interpretation or plotting techniques is provided in Fig. 6 and 7. The user can choose any of these illustrations from the platform. The mentioned figures represent the Canadian Election data from 1963. In Fig. 6, we have used a vertical bar chart to interpret 3 important metrics; ”percentage of seats won,” ”the number of votes won,” ”percentage of seats won.” The same metrics are also represented by the pie chart in Fig. 7. Using these representations, any user can quickly get the gist of the whole election and can easily identify the most and the least prominent players in the election. We do acknowledge the fact that every user would not be comfortable using the same representation. Keeping that in mind, in addition to these two representations, we have provided users lots of other conventional plotting techniques such as horizontal bar chart, and donut chart, scatter plot, and line plot. From our iterative user study, we understood that providing different options for the users to explore the data is essential, as the data has different aspects to it and also ranges quite a lot.

The next illustration in Fig. 8 shows the total candidate numbers in the federal elections since 1867. We can see that the number of candidates is ever increasing. We have implemented a simple Linear Regression algorithm to predict the number of candidates in the 2019 election [11]. Prediction of the illustration (Candidate Number in 2019 election:  336) is close to the original result. Here, we have also added information about the median and average of the number of candidates. If anyone wants details of an election since 1867, the user can hover the mouse over the specific point on the shown line, and the text-box pops up with all the interesting information about that election. For future predictions, the user can hover over any point on the best-fit line.

The illustration in Fig. 9 shows the federal parties’ history of winning in the past elections. We have chosen heat-map as it has previously been used to compare the volumetric difference using color intensities. Therefore, a user can easily grasp the information like the following: ”In the history of Canada, a specific party has won the most election” This is a review we got from our user during the testing phase of the tool. [12]. Users can also pick that the second and third most popular parties in Canada always went neck-to-neck when it comes to winning federal elections, although the user may need to possess certain knowledge about the party-names and which parties are still in play. But users can gain this knowledge from other graphs that are discussed in the project.

From our initial study, users revealed that having only year-wise or election-wise data representation is not helpful enough. Our users have requested a progression map where all the practical data should be present and accessible by the users at their convenience. In Fig. 10, a representation is shown where the information of all of the parties is put together (e.g. ”number of seats won,” ”percentage of seats won,” ”percentage of votes won”) along with the regression lines generated for each party. This graph can identify which party has been consistent over the year, which party has been trending over the past couple of elections, which parties’ popularity (a feature that is depicted by other attributes like percentage of seats and votes won) [13]. For example, from this illustration, we can see that although the Liberal Party has won most of the elections throughout history, its’ popularity has been slowly decreasing over the long run. In contrast, the Conservative Party of Canada is gaining tremendous support in recent decades, and Unionists have been holding their position steadily throughout the history [14].

The first part of Fig. 11 deals with provincial elections, unlike the other graphs. Here all the provincial parties participated to demonstrate which ones gained most of the votes throughout the history. The same information has been interpreted using the heat-map in the second part of 11, where the user has the independence to choose from any representation.