Adapting the re-ID challenge for static sensors

The GZCD dataset is comprised of 5,464 images sourced from Meru County, Kenya, taken over four days of GGR-16 and GGR-18 by 13 photographers (see Fig. 3). The photographers were trained to capture a consistent right-side viewpoint for Grévy’s zebra, and only images taken of the intended side were kept in the dataset by human reviewers. The dataset is highly curated: bounding boxes and labels (species, viewpoint, and quality) were set by human reviewers for all animals. The 7,372 right-view Grévy’s annotations were further filtered based on these quality labels, resulting in a set of 4,269 “quality baseline" right-view Grévy’s annotations. Note that the spatial subset for Meru County, Kenya is geographically isolated by mountains from neighboring conservation areas, giving the expectation that the population is largely self-contained. Thus, the GZCD dataset is an excellent testbed for our censusing pipeline.

We use images collected from a network of 70 camera traps distributed around the Mpala Research Centre in Kenya’s Laikipia Plateau. The network has collected 8.9 million images over the past two years of deployment. There are four types of camera trap placement schemes in the network: systematically in a grid, at “magnet" sites (e.g. salt licks), as well as expert-targeted & random placement along roads (see Fig. 2).

Prior to re-ID, the raw camera trap images were first passed through a YOLO v2 species detection model Parham et al. [2018a] to localize all zebra with a bounding box (both Grévy’s and plains zebra species), and crop the region of the image within each bounding box for downstream use. Localized bounding boxes crop out irrelevant and potentially distracting background information and yield distinct annotations of independent individuals from images that feature several animals. YOLO v2 has shown to be more accurate for animal detection than alternative object detection models, such as Faster R-CNN Parham and Stewart [2016]. Next, the cropped regions are classified to zebra species — Grévy’s vs. plains zebra — and viewpoint — left vs. right — by a DenseNet model. Only right, front-right, and back-right viewpoints are considered for identification; differing viewpoints cannot be matched with one another, as the right and left sides of Grevy’s zebra are distinct. Both the annotation localization and classification networks were trained on the WILD dataset Parham et al. [2018a].

Beyond assigning viewpoint and species to each annotation, we wish to ensure that these annotations — from both human-captured and camera trap images — would be universally comparable. In principle, when such an annotation fails to match the other annotations we should be able to safely conclude that it is the only annotation from that animal. Furthermore, focusing on these annotations (a) should make verification decisions easier both for an algorithm such as VAMP and for a human reviewer ("verifier"), (b) should allow recovery from mistakes such as incidental matching ¹¹1As examples, a matching algorithm may recognize animals seen in the background (i.e., a “photobomb” Crall [2017]); camera traps are acutely affected by a similar scenario by accidentally matching background textures, and (c) should avoid increased human effort to address matching ambiguities.²²2See time and effort studies of animal ID curation in  Parham [2021]. For Grévy’s zebras, this entails annotations that have both the distinctive hip and shoulder chevron (see Fig. 4, which we will call “census annotations" (CA) Parham [2021].

In order to train a census annotation network, we started by curating 10,229 Grévy’s zebra annotations from GGR-18. These were arranged in a series of grids and presented via a web interface to two reviewers, who selected the annotations that were Grévy’s CAs. Then, a DenseNet with a linear classification layer was trained on this CA dataset to decide whether an annotation was indeed a CA, producing an associated CA confidence score (see Fig. 5). Finally, a regression network was trained to narrow the annotation region to only surround the hip and shoulder chevron and minimize any distracting background information. Each of these new annotations was saved as a “Census Annotation Region”, or CA-R Parham [2021]. The CA and CA-R networks were then used to obtain the CA-Rs and corresponding CA scores for both GZCD and the camera trap dataset.

Mark-recapture statistical models are highly sensitive to inaccurate identifications, but are built with the assumption that not all individuals are seen. Thus, it is important to prioritize the precision of our re-identification pipeline over assigning an ID to each annotation. One significant factor in inaccurate identification, by both humans and algorithms, is image quality — blurry, or poorly lit annotations are difficult or impossible to reliably identify. Filtering annotations by quality is therefore a critical component of the re-ID pipeline, and automated quality filtering reduces the amount of human time and work required to rectify matching errors caused by undesirable annotations. Here, we devise an annotation filtering scheme suitable for re-ID from camera traps (Fig. 6).

As already described, we only consider census annotation regions (CA-Rs), those the show right-side viewpoints of Grévy’s zebra, including both the shoulder and hip chevron. Beyond this, we further filter by time of day: only annotations from images taken between 6:30 AM and 7:00 PM (sunrise and sunset at Mpala) were kept for re-ID. Due to the camera trap settings, images taken during the daytime are optical (RGB) and are of higher resolution (13 MP). Nighttime images are taken with an infrared flash at lower resolution (9 MP). Qualitatively, nighttime images are more difficult for human reviewers than daytime images; without modification, we believed the VAMP verification algorithm was likely to perform better on higher contrast and quality images taken during the day.

Next, the annotations were filtered by encounter. To define the encounters, we used an agglomerative clustering approach: for each camera, annotations from images taken within the same minute and in consecutive minutes were grouped together in the same encounter. Next, the annotation with the highest Census Annotation confidence score was selected from each encounter, and the rest were discarded. As the camera traps take images in bursts, the images in a encounter are nearly identical, and hence the corresponding annotations very likely must feature the same individuals. This step encourages matching across the best representatives from every encounter, reducing the amount of data required but preserving the relevant information. Experiments in Parham [2021] show that filtering in this manner, in particular filtering for CA-Rs, does not have a significant impact on the individual count, but dramatically reduces the human effort required to produce it (Fig. 7). While the GZCD used mobile photographers, the encounters it used were based on location and time, and often included many Census Annotations; the camera trap dataset used a shorter time-only decision to reduce trivial duplicates.

Lastly, the annotations were filtered for CA confidence score. Annotations above a 0.31 confidence score threshold were kept for re-ID. In previous work Parham [2021], a lower threshold was shown to add incomparable annotations to the re-ID database and encourage spurious matches, while a higher threshold was shown to eliminate relevant sightings without a considerable increase in matching speed and accuracy. The filtering pipeline yielded 685 right-view Grévy’s zebra annotations across the reserve, with each representing a single encounter (see Fig. 2).

The goal of the LCA decision management algorithm is to build an identification graph with the filtered annotations, using automated ranking & verification and delaying human intervention as much as possible. In the identification graph, annotations are represented as vertices, and relationships between annotations (whether positive or negative matches) are represented as edges. The LCA algorithm seeks to group annotations into clusters corresponding to individuals by maximizing positive edge weights within clusters and negative weights between clusters. This simplifies the identity labeling process significantly: instead of being asked to identify an annotation as one of up to 150 known individuals, reviewers are asked to compare pairs of annotations and determine whether they belong to the same individual or not. This contrastive task can be accomplished even by non-experts due to innate human capacity for pattern matching — it is similar to a game of spot the difference.

The ID graph is initialized with no edges, with every annotation constituting its own cluster. For every annotation, the Hotspotter ranking algorithm returns a list of its most confident matched annotations, and LCA forms edges between the annotation and its potential matches. Next, the VAMP verification algorithm evaluates every pair of vertices connected by an edge and assigns an edge weight (positive or negative) based on its confidence in the match. The LCA algorithm has two main phases. In the scoring phase, it keeps the initial edges and weights intact. LCA iterates through every local clustering (a single cluster or a pair of clusters) and checks to see if there exists an alternative clustering that increases a score measure defined as the sum of intra-cluster edge weights minus the sum of inter-cluster edge weights. If such an alternative clustering exists, the new arrangement is adopted. Once there are no longer any better local alternative clusterings, LCA proceeds to the stability phase. In this phase, LCA considers local clusterings for which the difference between it and its next best alternative are sufficiently small that changes in edge weights can potentially introduce a superior alternative clustering. For such local clusterings, LCA requests additional reviews to VAMP and the human (if VAMP cannot litigate the review on its own) to determine if certain edge weights must change. Once all local clusterings are significantly better than their next best alternative, the algorithm is considered to have converged.

As discussed above, there were 4,269 “Quality Baseline” right-view Grévy’s annotations selected for identification from GZCD. Additionally, from the initial set of 7,372 right-view Grévy’s annotations in the GZCD, 4,142 were classified as CAs by our CA model using a score quality threshold of 0.31. Furthermore, each of these CAs were manually annotated to yield 4,142 corresponding CA-Rs. It should be noted that the CAs are not a strict subset with the Quality Baseline set; the two sets overlap significantly for trivially good sightings but each have annotations that are not present in the other. These three annotations sets — quality baseline, CAs, and CA-Rs — were each passed into the LCA algorithm for identification.

As the GZCD is composed of images taken during GGR-16 and GGR-18 in Meru County, we are able to compare the ID curation results with ground-truth estimates obtained from the two rallies. The Lincoln-Petersen index Petersen [1896] for the “Quality Baseline” is $360\pm 27$ in 2016 and $399\pm 29$ for 2018. The population estimate based on only CA was $366\pm 27$ for GGR 2016 and $373\pm 29$ for GGR 2018, whereas the CA-R estimate was $366\pm 27$ zebra in 2016 and $373\pm 29$ animals in 2018 Parham et al. [2018b].

With the Quality Baseline set, the LCA algorithm requested 22,972 decisions in total, with 420 from the human reviewer (yielding an automation rate of 98.2%). However, LCA converged faster with the CA set, with 18,607 total number of reviews, 352 of which were performed by the human reviewer (yielding an automation rate of 98.1%). We observe that LCA converged with 4.4 decisions per annotation with the CA set compared to 5.3 with the quality baseline set, indicating that the census annotations were more discriminative and required fewer alternative clusterings for LCA to consider. Finally, LCA converged the fastest with the CA-R set, with 13,427 reviews requested in total and only 120 performed by the human reviewer (yielding an automation rate of 99.1%). The results with the CA-R set were accurate within 4.6% on GGR-16 data (predicting 349$\pm$26 animals against a ground-truth value of 366$\pm$27) and within 0.5% (371$\pm$30 animals predicted against 373$\pm$29 ground-truth). In comparison, the quality baseline LCA results underestimated the ground-truth counts by only 12 individuals for GGR-16 and 1 individual for GGR-18, but required nearly four times the number of human reviews – therefore, considering CA-Rs relative to the quality baseline results in a 71.4% reduction in human work, while the resulting estimates still remain within the confidence interval of the established baseline count. These results are summarized in Table 4. While the CA-R result was obtained with a major reduction in human effort, Figure 7 also demonstrates that the amount of time spent on each CA-R review is substantially decreased (and more accurate).

8.9 million camera trap images from the initial dataset were passed through the species ID pipeline, yielding 84,383 zebra images (including both plains and Grévy’s zebra). Following species and viewpoint filtering, 23,512 right-view Grévy’s zebra annotations remained across 3,338 distinct encounters. After excluding nighttime encounters, 1,138 daytime encounters remained. We then sampled the single highest-quality annotation from each of the 1,138 encounters, as determined by our CA model, to avoid matching on near-identical images from the same burst. We further used a score quality threshold of 0.31 on census annotations that reduced the number of annotations to be identified to 734. As a last step, we used a blurriness filter to remove any additional challenging annotations, resulting in a final set of 685 CA-Rs.

In order to associate these 685 annotations to clusters of individuals, the LCA algorithm requested 5,403 automated reviews by VAMP and 331 additional human reviews; pairwise comparisons of annotations were performed largely automatically using VAMP, with a automation rate of 93.9%. The converged ID graph had 173 clusters, each corresponding to an individual zebra ID within the 685 annotations. On average, each individual was sighted across 1.9 static camera traps and 3.96 encounters, indicating significant matching across space and time. See Supplementary Fig. 8 for the distribution of encounters for each individual, and Supplementary Fig. 9 for the distribution of static camera traps that sighted each individual. The number of individuals sighted by each camera also differed based on its placement strategy (random grid, known Grévy’s territories along roads, randomly along roads, and both timelapse & motion-trigger traps at magnet sites). On average, camera traps at a magnet site (e.g. salt lick, dam, etc.) sighted the most individuals on average (8.5 per trap) and also produced the most first individual sightings on average (5.85 per trap). See Table 4 and Supplementary Fig. 10 for the distribution of total individuals sighted by cameras of each placement strategy. Particularly, the encounters for large individual clusters were spread across many months, with some spanning more than a year (see Supplementary Fig. 11). Based on the distinct encounters and cameras associated with each individual, we can generate spatial maps of encounters for any individual across the reserve: see Fig. 1 for a map of encounters for a particular zebra (Individual 32).