Urban feature analysis from aerial remote sensing imagery using self-supervised and semi-supervised computer vision

Self-supervised learning extracts knowledge from unlabeled datasets by setting up a pretext task on which the model can be pretrained in a supervised manner [3]. In self-supervised workflows, the focus is on the intermediate representation that is learned by the self-supervision pretext task, rather than maximising prediction accuracy. This intermediate representation is used in downstream tasks such as object detection, with the expectation that the representation learned during the pretext task is robust from a semantic and structural perspective.

There is currently a large body of work focused on learning task-independent representations using these techniques. For example, Noroozi and Favaro [4] formulated a jigsaw puzzle task by selecting several adjacent blocks of pixels. After shuffling the blocks, the model’s task is to recover the correct spatial order (see Fig. 1a). This task requires high-level reasoning based on the objects and details visible in the image. Therefore, a model that excels in the pre-training task is likely to contain a useful representation of the image. Similarly, Doersch et al. [5] designed the task of retrieving the relative position of a tile compared to a selected image section (see Fig. 1b).

Importantly, while self-supervised learning tends to reduce the labelling requirements for training neural networks, it does not provide a means for labelling large datasets. This is because self-supervised learning generally provides psuedo-labels for the model to build an initial representation of the world, which helps to reduce the number of labeled data points it needs to see to build a hypothesis about a particular category, but does not necessarily label data points related to those particular categories.

Semi-supervised learning corresponds to the class of machine learning techniques where a large amount of unlabeled data is available alongside a smaller collection of labeled data. These approaches attempt to use the small volume of labeled data to assign labels to the much larger volume of unlabeled data, in an iterative fashion. Thereby, the set of labeled data grows during analysis, leading to more accurate models.

Prior work has used semi-supervised approaches (also referred to as bootstrapping approaches in some research areas) to improve prediction accuracy of predictive models by generating more training data. However, very few operate in a fully automated manner. An early work in this paradigm of models learning by themselves is by Yarowsky [6], who investigated the possibility of using labeled sentences coupled with unlabeled data to perform word-sense disambiguation. Several works also explored applicability of this technique in computer vision. For example, Cui et al. [7] iteratively grew their dataset by merging in high-confidence predictions from their model. However, a manual vetting process was employed at each step. Huang et al. [8] used morphology and colour-based indices using predefined formulae, as well as openly available information sources to generate training sets and classify images into the classes of buildings, roads, soil, water, shadow, and vegetation. A key issue with classification approaches is often that classes are assumed to be mutually exclusive. However, in aerial images of urban scenes, it is possible for roads, vegetation, soil, water and buildings to co-exist within the same image.

In general, the key improvements due to semi-supervised learning strategies can be incorporated at two levels:

• 1.

  Model level involves improvements incorporated into model training processes and focus on providing models with a more robust representation from fewer initially labeled image samples.

• 2.

  Data level involves improvements in the semi-supervised labelling process itself, allowing model-independent improvements via techniques such as heuristics and morphological feature extraction.

Model level improvements generally include techniques which are useful even outside of the scope of semi-supervised learning as well. In fact, many of these techniques are used to improve supervised learning models. As examples, Miyato et al. [9] use adversarial training, Siddharth et al. [10] use disentangled feature learning and augmentation strategies such as RandAugment introduced in Cubuk et al. [11] are also commonly used.

Data level techniques operate outside of the scope of the model. These increase the probability of the model correctly labelling unlabeled image samples without human intervention. For example, Kothari and Meher [12] use unlabeled neighbourhood information to improve model performance.

As these two types of techniques apply at different levels, it is additionally possible to overlap them for potential combined improvements as well.

Most work in semi-supervised learning focuses on images captured in the horizontal perspective (images generated by cameras in non-aerial settings), due the abundance of labeled data which enables much easier model evaluation. By treating a large part of the dataset as unlabeled, it is still possible to easily evaluate model behaviour with small labeled datasets while also providing very robust accuracy, precision and recall metrics as required. Using an unlabeled dataset only enables the provision of estimates of such performance metrics, as the ground truth of a large part of the dataset is unknown. However, this type of analysis more accurately matches use of the technique with unlabeled datasets in the wild.

Table 1 contains a comparison of such techniques based on labeled set size, perspective and unlabeled set size. This comparison indicates reported results for the model using the lowest number of labeled images and not the number corresponding to the best results.

Many techniques compare performance based on a percentage of unlabeled data used as labeled data (for example, 1% of data used as labeled data). However, this is not necessarily representative of annotation effort, which is a function of the absolute number of labeled images. As evaluation is primarily carried out using labeled data which is treated as unlabeled data (by hiding the label from the model), it is straightforward to do so. However, for use in the wild with a new unlabeled dataset, data annotation effort is often the limiting factor. Additionally, most techniques report performance based on the training set size as a percentage of unlabeled/total set size, disregarding the labelling requirements for validation data. In this article, a major objective is to limit the total labelling requirement, and aim to work with smaller validation sets as well.

In machine learning, active learning refers to the class of techniques where the model can iteratively query a human user regarding the ground truth of a subset of input data. Based on the user’s input, the model then performs additional learning to improve its prediction accuracy. This requires manual intervention at each iteration of the learning process. Active learning has been successfully used for a multitude of tasks including crystal structure prediction [16], vehicle detection [17] and facial recognition [18]. These approaches work well in theory, by using an oracle or already annotated dataset for evaluation purposes. However, Settles [19] argues that, when attempting to bootstrap a new dataset in practice, it is generally not time efficient to wait for model training to finish before annotating more images. A key difference between semi-supervised learning and active learning is that the agent doing the annotation in semi-supervised learning is an automated model, whereas in active-learning it is generally a human.

A large image dataset was obtained using an aerial imagery provider. The image collection spanned 15 of the most populated cities across Australia for a total of 62.5 million images of size 256$\times$256 pixels taken at a zoom level of 21 (corresponding to 0.074 metres per pixel at the equator, a tile edge size of roughly 20 metres and an area of roughly 400 square metres covered by each tile). The total area covered by the study was $22,536km^{2}$. Data collection by city/state is indicated in Table 2.

For the exploration of cycling infrastructure in urban environments, an initial sample of labeled imagery was obtained through an observational study [39]. A total of 100 participants were recruited between March 2015 and January 2017 near on-road locations in Western Australia where a crash was observed previously. Cyclists were intercepted as they stopped at traffic lights and offered a slap-band for their wrist which had the study website recruitment address printed on it. Cyclists then completed an online questionnaire and were asked to leave their contact details within the questionnaire if they were willing to be contacted to be part of the study. Potential participants were contacted by phone to further explain the study and were sent a consent form by email. Cyclists were eligible to participate if they had not been involved as a cyclist in a bicycle crash requiring hospitalisation in the previous three years, were 18 years or older, lived in the Greater Perth area, spoke English, and cycled at least once per week. If the cyclist agreed to participate, an appointment was made to attach the GPS tracking sensors to their bicycle. Data collection included the recording of up to six hours of cycling video footage and associated GPS data per participant. Participants were asked to record any cycling they participated in and ride exactly as they normally would.

After the conclusion of the observation period, the recorded GPS information was allocated to specific participant trips. The most common routes that each participant travelled on covered a total road network of 1680 kilometres in Western Australia (see Fig. 3). This consisted of 280 kilometres of bicycle paths and 1400 kilometres of on-road routes. The recorded GPS tracks were then used to annotate aerial imagery in Perth with the presence of cycling infrastructure, leading to an initial dataset of labeled imagery.

An ablation test was performed on the above workflow, for validating its usefulness with the aerial imagery. This was achieved by sampling 100 images each for training and 1000 images each for validation for two classes (cycle infrastructure vs other) from the aerial dataset mentioned in Section 2.1.1. A ResNet50 model was then built and trained on this task following three separate configurations. The first was instantiated with the pre-trained weights from ImageNet[46] for ResNet50, which is a commonly used approach in computer vision. An objective of this experiment is to evaluate the suitability of such a technique for use with overhead imagery. A single fully connected layer was trained for the purposes of class prediction, and was placed on top of the final bottleneck layer of the ResNet network (identical to ImageNet training except for the number of classes). The second configuration used a pretrained representation built from 100,000 unlabeled images from the aerial imagery dataset. These pretrained weights were used instead of the weights loaded from the pretrained ImageNet model. For both these configurations, the layers of the ResNet are frozen and the corresponding weights are not updated during training. This ensures that the model is forced to rely on only its pretrained representation as a feature extractor, while learning only very high level abstract concepts relating to the task at hand. A high learning rate of 30 was used with stochastic gradient descent since only a single linear layer was to be trained. The third configuration uses the pretrained representation learned in the second configuration, but uses it for end-to-end transfer learning. In this configuration, all the weights of the ResNet are updated during the training process, which is not the case in the other configurations. A learning rate of 0.001 was used with stochastic gradient descent in order to minimize changes to the pre-trained weights under this configuration. This low weight aims to minimize the destruction of pre-learned features in the model, by only performing small tweaks instead of big shifts in existing features. The neural network was trained for 200 epochs and the checkpoint with the best validation performance was used for reporting performance. The results are in Table 5 under Section 3.1.1.

As an initial evaluation of transfer learning from the self-supervised representation, an experiment was carried out to evaluate ResNet50 based on finetuning by transfer learning from the frozen MoCo representation. Results are in Table 6. The objective of this experiment was to better characterize performance of the two configurations built on the pretraining workflow. The complete dataset of training set, validation set and test set each representing 2 classes, contained 33,337 images. These aerial images were randomly selected from a large set of labeled road images sampled from areas known to have cycling infrastructure. Images containing cycling infrastructure were manually filtered such that 18,642 images contained cycling infrastructure, and 14,695 did not contain any cycling infrastructure. The ResNet50 architecture was used in all experimentation and the highest validation accuracy model was picked as the final model. For the \sayFrozen configuration, a learning rate of 30 and a batch size of 4 was used with stochastic gradient descent, while for the \sayTransfer configuration, a learning rate of 0.001 was used alongside a batch size of 16 with stochastic gradient descent. By testing different configurations of training and validation set sizes, the expectation was that a better understanding of model performance scaling with larger training set sizes could be obtained. This would in turn serve to confirm the results in Table 5 while indicating potential thresholds in terms of manual annotation requirements for solving tasks of this nature. The overall size of the dataset is kept fixed to more accurately mirror the actual situation of using a model to iteratively grow a dataset from a pool of unlabeled images: the size of the unlabeled image set would shrink as more images are moved out of the unlabeled dataset. The results are in Table 6 under Section 3.1.2.

To evaluate the suitability of semi-supervised learning, an experiment was carried out using the above configurations. These configurations were evaluated on a single task (cycling infrastructure categorization) on the same dataset of 33,337 images as in Section2.2.2. The results can be found in Table 7 under Section 3.2.1.

As a follow-up experiment, the consistency of continued semi-supervised learning was explored as a single-class fixed dataset experiment. A priority queue based implementation was used to track the top 500 highest and lowest cycle symbol confidence predictions from the test set to merge into the training set. The validation set was fixed at 1000 images each. Continuous evaluation of the bootstrapping approach using the transfer learning from the Frozen configuration was carried out, starting with 1000 training and validation images per class with a step size of 500. The results are in Table 8 under Section 3.2.2 .

While previous experiments were exclusively single class (looking at cycle symbol classification), this experiment aims to evaluate the methodology in a more generic manner. A practical limitation in this regard is the image annotation requirement for attempting many different tasks. To make the most of limited annotator time, a limit of 200 annotations per class per task was imposed, with 100 images each for the training and validation sets respectively. As before, the two classes correspond to a “Task” class vs a ”Background” class. The main reason for this experimental setup is that it is highly likely for multiple infrastructure classes to be present in the same image. Therefore, by creating a binary classification task, we are able to overlap annotations from multiple models on the same image in a similar fashion to object detectors, without needing to generate bounding boxes for the different tasks which would severely limit annotator time availability to explore multiple classes. Within this limitation of 100 training images, prior experiments (Section 2.2.1) indicate that the Frozen configuration performs best and that a validation set of 100 images should be sufficient in this respect. Evaluation of the Frozen configuration trained on 100 images of each class in training and validation is provided to compare the base performance of the methodology on each task. The Frozen configuration is used as it is useful for providing a baseline level of performance to compare against. Further, it has the added benefit of being trained very fast due to the high learning rate used. The percentages reported correspond to the precision of the class under investigation. No false negatives were detected during this set of experiments. Evaluation was carried out on a random sample of 100 images drawn uniformly from the top 1000 predictions at each location ordered by confidence. The results of this experiment can be found under Section 3.2.3.

This experiment explored the development of a workflow centered around using historic images at locations for improving model accuracy. In particular, the main objective was to build upon the results from Section 2.3.3 by using historical imagery as a data augmentation/semi-supervised learning strategy.

To this end, several key semantics of the task at hand are exploited. The key insight for this methodology is that infrastructure is static: if it is available at a location at present, it is likely to have been present at that location in the recent past. It is also reasonable to expect that the probability of finding that infrastructure would decrease if the image was taken at an earlier date than a later date, simply because the infrastructure might have been constructed at an intermediate date. By contrast for the background class: if a particular image does not contain some infrastructure it is highly unlikely to have been there in the past: effective planning schemes mean that cities and other infrastructure are usually planned well ahead of time, and drastic changes are unusual in the short term.

Hence, the following assumptions are made regarding historical images at a location:

• 1.

  The probability of finding the task class in historical images at a location correctly labeled as the background class is negligible.

• 2.

  The probability of finding the task class in historical images at a location correctly labeled as the task class is high, with the probability of finding the task class in more recent images being higher than in older images.

Considering the model as a \saytask class detector, a false positive confounder would be an image of the \saybackground class being incorrectly classified as belonging to the \saytask class (identical to a false positive). Let $\Phi$ be the class of all historical images at all background image locations from the training set. Then, consider the set $\Theta\subset\Phi$ which contains all the confounders from the model performing inference on images contained in $\Phi$. The set $\Theta$ is then a very informative dataset for the current model to learn from, as the model has been unable to classify them correctly, despite having seen a preceding image in the training set in the background class. Additionally, more recent confounders would be more useful than older ones, as the more recent images could be expected to look more structurally similar to the image at present and therefore contain more interesting features to include in the background class (as opposed to, for example, an unbuilt area from a long time ago which would likely not add much predictive value to the background class). Note that this logic is not necessarily commutative if the \saytask and \saybackground classes are swapped: the infrastructure under investigation might have been constructed/painted very recently and thus, may not necessarily be misclassified as being \saybackground class (since if the \saytask class is not present in the image, it belongs, by definition, to the \saybackground class). Conceptually, this is similar to boosting[47] in machine learning, as images misclassified by the model are assigned with an increased weight into the training set thus increasing their importance in terms of contribution to the decision boundary of the model.

Let $\Phi_{T}$ be the set of all historical images corresponding to training set locations and $\Phi_{B}$ be the set of all historical images corresponding to background locations, with $\Phi=\Phi_{T}\cup\Phi_{B}$. Note that the latest available images also count as historical images by definition and as such would be included in these sets. Due to being historical images of labeled locations, clearly the sets $\Phi_{T},\Phi_{B}$ contain images that the model could learn from, in a supervised manner. However, not all images would be equally useful or correct to learn from. Thus, assigning a weight to each individual historical image allows the training process to be controlled (when assigned a weight of zero, an image would essentially have no impact on the training process). Therefore, the problem at hand can be defined as follows:

Let $\Phi_{T}^{i}$ and $\Phi_{B}^{j}$ correspond to the above sets with $i,j$ corresponding to arbitrary orderings (indices). Then let the individual loss of each training sample be determined by the function L(x), which would apply the loss function used in the neural network to the corresponding output of x. Then, the overall loss function becomes:

Where $\alpha_{T}^{i}\in\mathbb{N}$ corresponds to individual historical task weights and $\alpha_{B}^{j}\in\mathbb{N}$ corresponds to individual historical background weights. Without loss of generality and for simplicity, let the first N elements of both orderings $\Phi_{T}^{i}$ and $\Phi_{B}^{j}$ be set to an arbitrary ordering of the initial human labeled training set of N images per class. As the model trains in a semi-supervised manner, the main difference in the data composition is tracked by the different values of $\alpha$ over the entire dataset. Note that images with $\alpha=0$ have no contribution to model training, and may be omitted during training.

The following operations are defined in order to modularise the workflow for the semi-supervised learning process for improving the performance of the models using archival imagery. It is important to note that confidence metrics are defined with respect to the \saytask class. The confidence metric corresponds to the probability of a particular image belonging to the \saytask class and is related to the probability of the image belonging to the background class as $P_{T}=1-P_{B}$ due to the presence of only 2 classes.

• 1.

  Train - Builds a classifier from the currently available training dataset as defined by Equation 1.

• 2.

  Predict - Uses the most recently built classifier to perform prediction on the historical datasets ($\Phi_{T}$ and $\Phi_{B}$ separately) and assigns confidence scores based on the task class (not the background class).

  The semi-supervised learning process relies only upon training several iterations of computer vision models which have access to different training sets. The train and predict operations provide interfaces for this functionality. As any modifications to the data/weights only affect the process once a model is trained and prediction is carried out on $\Phi_{T}$ and $\Phi_{B}$ (thus updating the confidence metrics), each step/iteration of the semi-supervised learning process begins with training the model and predicting on $\Phi_{T}$ and $\Phi_{B}$.

• 3.

  Update Task - Increments $\alpha_{T}^{i}$ corresponding to the $M_{T}$ highest confidence task detections in $\Phi_{T}$.

• 4.

  Update Background - Increments $\alpha_{B}^{j}$ corresponding to the $M_{B}$ lowest confidence task detections in $\Phi_{B}$.

• 5.

  Update Confounders - Increments $\alpha_{B}^{j}$ corresponding to the $M_{C}$ highest confidence task detections in $\Phi_{B}$ (hence matching the definition of confounder: high confidence, but assigned to the wrong class).

  The update operations are used for managing the datasets over iterations of the semi-supervised learning process. By updating the contribution of each individual image to the loss function, the decision boundary of the model is modified as well, with some images receiving a higher importance than others. It is important to note that order statistics (such as the $M_{T}$-th largest confidence value) need to be maintained independently for the two datasets $\Phi_{T}$ and $\Phi_{B}$ as the underlying semantics and class probability distributions for the two datasets are very different.

In combination, these operations define the behaviour of the semi-supervised technique. The temporal control of image availability over time is managed by gradually broadening the time frame over which the model is allowed to update values of $\alpha$: initially, only values corresponding to more recent images may be updated but in later iterations, $\alpha$ values corresponding to earlier images may be updated as well. This greatly decreases the probability that the model will incorrectly classify an image due to having less of a connection (structurally or otherwise) to the image’s present predecessor image. This behaviour is determined by the parameters $D_{T}$ and $D_{B}$ corresponding to the \saytask and \saybackground classes and denotes the maximum duration (in months) from the latest image in the dataset that an image would need to have been captured, in order for the $\alpha$ value to be updatable. In other words, $\alpha_{T}^{i}$ may be updated if and only if the image $\Phi_{T}^{i}$ was captured within $D_{T}$ months of the last image in $\Phi_{T}$, and similarly for $\alpha_{B}^{j},D_{B},\Phi_{T}^{i}$ and $\Phi_{T}$.

As some of the novelty of this work is concentrated in the update operations, we provide an algorithmic implementation of the base update functionality in Algorithm 1. Note that an implementation which uses a heap-based approach for tracking the top and bottom K-th confidence items in a given dataset with complexity O(N log K + K log K) for maintaining and iterating over such a list of items. An alternate implementation is to use the QuickSelect[48] algorithm to generate the K-th order statistics for a given unsorted dataset. This allows the creation of a list of the top or bottom K confidence items in O(N + K) time and is presented in Algorithm 1. Since the discussed applications are not time-critical and because K is generally much smaller than N in most situations, both implementations are expected to have similar performance and are listed here for the sake of completion. The update operations previously mentioned are implemented in Algorithm 2.

Algorithm 3 contains the overall procedure (corresponding to a single step from Table 4), building upon Algorithms 1 and 2. Additional subscripts are provided to indicate the recommended parameters for the function based on the previously defined data arrays.

After the steps corresponding to Algorithm 4, all $\alpha_{B}^{j}$ are incremented by one to guarantee inclusion in the training set (justified by the assumption that the class of infrastructure being detected was never present at background image locations). The final model used for querying the unlabeled non-archival dataset is then trained using transfer learning as in previous experiments. The results from this experiment are in Section 3.2.4.

A natural follow up to the experiment in Section 2.3.4 is to specialize the initial representation learned in a task-independent fashion. Additionally, the results from the ablation study in Section 2.2.1 indicate the benefits of specializing the representation in a more domain specific manner: ImageNet weights are often used as a generic task-independent representation, but by specializing the representation for aerial imagery, significant improvements are obtained.

The road images in this experiment were generated as the top 100,000 confidence predictions from the overall dataset by the road model from Table 11 which had a very high precision. This pre-training set was validated by sampling 10,000 images uniformly at random and performing manual annotation. The precision over this sample was 100%. Using this 100,000 image dataset, a further specialized representation was created by using an identical workflow to the initial task-independent representation. With this specialization, the representation is still task-independent, however, it is no longer as generic as the previous pre-trained representation (used in section 2.3.4) and better performance can be expected on tasks specifically associated with roads, and poorer performance otherwise. The results of this experiment are in Section 3.2.5.

The results of archival analysis are in Fig. 4. From an infrastructure analysis perspective, this allows analysis of the growth of infrastructure at the city level. Since infrastructure forms a key cornerstone of cities that affects all other aspects including transport and health, being able to analyse when specific classes of infrastructure were introduced is very useful. In particular, analysis across multiple classes of infrastructure is very valuable in understanding the relationship between such classes and other aspects of cities such as inhabitant behaviour and population health outcomes. This can also be used to identify the trajectory of cities with regards to how well they are supporting healthy habits through initiatives such as provision of safe, active transport infrastructure for citizens.

From a computer vision perspective, datasets such as these introduce new ways of utilizing geographical information spanning multiple time-steps. For example, in this work we have exploited the static nature of infrastructure. However, it is possible to take this even further by exploiting the fact that once infrastructure is introduced to a location, it is very likely to stay there. This \sayexpectation of the maintenance of infrastructure allows the introduction of additional analytical steps that can improve model performance. In this situation, for example, we can expect an example vector over multiple time steps at the same location to look like (False, False, False, True, True) with the infrastructure class not being present in the first three time-steps and being introduced sometime between the third and fourth time-steps. It is reasonable, then, to assume that it will also be present thereafter. Therefore, if we assume that for some time-step $t_{0}$, that the infrastructure was introduced to the location between $t_{0}$ and $t_{0}+1$, then the prediction for that location would take the form of a \saystep function, with confidence zero upto $t_{0}$ and confidence one beyond $t_{0}+1$. This could be used to optimize the infrastructure detection models either by introducing this as a consistency loss which penalizes how different the model’s characteristic function is from a step function, or by enforcing such behaviour by performing smoothing operations on the confidence scores across multiple time-steps at the same location.

The results (Table 5) for the ablation on using self supervision detailed in Section 2.2.1 indicate that the representation learned by MoCo is superior. Interestingly, allowing the model to modify the representation learned by MoCo (Configuration 3) leads to a drop in holdout accuracy from 72% (from Configuration 1) to 61% for the smaller training set size, which is indicative of the issue of overfitting to the data.

The results in Table 6 correspond to the validation accuracy obtained using the self-supervised representation as part of the experiment described in Section 2.2.2.

Evaluation results of the dataset after evaluating using the pretrained representation are presented in table 7, corresponding to the experiment described in Section 2.3.1. Step size refers to the number of the highest confidence predictions per class which are moved from the test set back into the training set. $P_{class}$ refers to the precision of each class $P_{class}=\frac{correct_{class}}{correct_{class}+incorrect_{class}}$ for the images that are to be moved into the training set for that class (which corresponds to the step size). For example, a $P_{non}=0.999$ with step size = 1000 would indicate that 1 image belonging to the \saynon class has been misclassified.

Using this data from Table 7, several conclusions can be drawn:

• 1.

  The accuracy of the models built using the Frozen configuration are not suitable for bootstrapping at this level

• 2.

  The size of the validation set only has a minor impact on test set accuracy (2%) based on the results from the two Frozen experiments with 100 training images. Therefore, 100 validation images may function only slightly worse than 1000, thus further reducing annotation requirements.

This section describes the results of the experiment described in Section 2.3.2. These results indicate that initializing semi-supervised analysis with around 1000 labeled images per class would result in the ability to consistently improve the accuracy of future iterations of models (based on the $P_{Class}$ results).

However, noticing the trend of test set accuracy introduces another issue: the accuracy increases and then starts decreasing despite more images being present in the training set. Due to the limited size of the testing set, the reduction in the overall size of the test set can be seen to affect evaluation in this case. This is because it becomes increasingly harder to correctly predict from a smaller test set of harder examples, as more confident predictions (i.e., samples that are ‘easier to predict’) are moved out of the test set and harder test-cases are left in. For example, in results in Table 8, the test set has shrunk by 3000 images (from 29337 to 26337) corresponding to over 10% of the test set. These results indicate that a larger test set would be beneficial to further analyse this methodology, additionally allowing exploration into more classes.

This section presents the results of the experiment described under Section 2.3.3. The results in Table 9 indicate that more specialized (hence rarer) classes are harder to detect. This is because the potential for misclassification with a dataset of this scale increases when the probability of occurrence of a class decreases. Importantly, this is because more common classes are easier to retrieve as the probability of misclassifying them is lower. As an extreme example, even for a model with perfect accuracy, it would be impossible to retrieve a class which does not exist in the imagery dataset, such as trying to retrieve ’desert’ in Antarctica. This experiment concluded that some classes are much harder to detect using self-supervised approaches in datasets of this scale. Therefore, further analysis was focused on improving the performance on such classes, which is discussed under Section 2.3.5. Since results are broadly consistent across different cities (performance is consistent across different columns in Table 9, indicating that the impact of location is minimal), further analysis was conducted on the entire dataset.

Accuracy results using the semi-supervised method described in Section 2.3.4 are given in Table 10. As precision (of the category under exploration) is a useful indicator of performance, precision results on the 60-million image dataset can be found under Table 11. The results are compared against a supervised model trained using the same original training set taken over 5 different runs.

The results in Table 12 are compared to the baseline results of this technique from Section 2.3.4 corresponding to the use of a more generic representation. The results indicate that specializing the representation using road imagery has improved performance on categories of infrastructure which coincide with roads, whereas some categories (such as water features) become harder to detect using the proposed methodology.

While the methods introduced in this work attempt fully automated analysis using semi-supervision, there are error rates associated with such analysis. It is possible to lower these error rates in between iterations by performing a manual labelling step on the results to prune out any anomalous detections. These erroneous detections can be quite helpful in directing the model away from such mistakes in future iterations by incorporating these samples into the negative class for the problem at hand. Additionally, even \sayfailed runs where the model cannot provide a high level of accuracy can still be quite useful if the precision is higher than the natural occurrence rate of the detection in the overall dataset. For example, consider a dataset of 100 million images with a detection which occurs in about 0.1% of images. The dataset would have about 10,000 images containing the class under investigation. If the precision of the generated model is at least 20% over the top 1000 confidence images, then by annotating those 1000 detections, at least 200 detections will be obtained and can be used to further expand the labeled training dataset. In contrast, it would require manual annotation of at least 200,000 images, on average, to do the same without using a model or some other method of filtering the data.