Detecting and recognizing characters in Greek papyri with YOLOv8, DeiT and SimCLR

The training dataset provided for the competition consisted of 153 images from 108 Greek papyrus manuscripts which preserve text from the Iliad of Homer. Bounding boxes were added and annotated with the Greek letter for each character present. Apostrophes and periods were also annotated but ignored in evaluating the results. We divided the training dataset into five cross-validation partitions. Multiple images present in the same source manuscript were included in the same partition. The test dataset included 34 images from 31 manuscripts. Each submission required a JSON file in the COCO format to be uploaded to the competition site on CodaLab [15].

To supplement the images in the training dataset, we included 4,533 images from volumes XV–LXXXII of the Oxyrhynchus Papyri (excluding volume XXVII). Images from these volumes included literary texts like in the competition dataset but it also included documentary texts. To help train models specifically for recognition, we used the second version of the AL-PUB dataset which contains 205,797 cropped images of individual characters taken from images in the Oxyrhynchus Papyri collection as part of the ‘Ancient Lives Project’ [21]. These images were labeled with the Greek letter from each image.

For character detection and preliminary recognition, we used YOLO (You Only Look Once) [18]. The YOLO neural network model uses a series of convolutional layers to predict bounding boxes and probabilities for each categorical class simultaneously. Initial detections are made on coarse cells produced from the convolutions and, if the same object is predicted in multiple cells, they are corrected using non-maximal suppression. The release of YOLO9000 [16] and YOLOv3 [17] brought about multiple enhancements. In 2020, Glen Jocher released a PyTorch implementation of YOLOv3 which he named YOLOv5, published by Ultralytics [7]. Our models used Jocher’s later version YOLOv8[8]. We used the ‘x’ sized pretrained model which is reported to have achieved a mAP^val of 68.2 on the COCO val2017 dataset. We trained the model using the test dataset at three image resolutions: small (1280$\times$1280), medium (1600$\times$1600) and large (2048$\times$2048). Each YOLOv8 model was trained for 200 epochs and the weights from the epoch with the best result on the validation set were saved. The trained model was used to make predictions on the unannotated Oxyrhynchus Papyri images which were used as pseudo-labels [10] and concatenated with the competition training set. The YOLOv8 models were then trained using the combined input from the competition dataset and the pseudo-labels.

To enhance the recognition performance of the YOLOv8 results, we trained additional specialized recognition models. This allowed us to incorporate additional labeled and unlabeled data, and to leverage self-supervised learning approaches.

In the first instance, we trained a self-supervised SimCLR model [2] on all of the available data containing cropped Greek characters. This included the competition dataset, the AL-PUB dataset, and the characters detected by the YOLOv8 models from the Oxyrhynchus Papyri.

The SimCLR approach takes a mini-batch of images as usual, but augments each image to produce two views and employs a contrastive loss to teach the network to match each augmented view in the batch with its positive pair. On datasets such as ImageNet and CIFAR-10, augmentations including cropping, horizontal flipping, color-jitter and Gaussian blur are used to get best results [2].

Selecting appropriate augmentations for the problem is important for models trained using SimCLR to perform well because these influence the image features leveraged by the model to distinguish different images within the training process [4]. In the context of Greek character detection, augmentations such as cropping and horizontal flipping may change the character sufficiently to make it a different class. Accordingly we reduced the degree of cropping, removed the horizontal flips and did stronger color-jitter and blur.

After pretraining a ResNet-50 convolutional neural network on the data, we fine-tuned the model [3] using the labels available in the competition and AL-PUB datasets using fivefold cross-validation.

In the second instance, we trained DeiT transformer models using transfer learning [22]. This was an entirely supervised process so we used the same labeled data as for the ResNet-50 SimCLR models with the same cross-validation splits. For this approach we also used less aggressive cropping during training, so that a significant portion of the character of interest would always be present during the training process.

For our final submission, we ensembled the results from our trained models. For character detection, we ensembled the results for the YOLOv8 trained models using Weighted Boxes Fusion [20]. We then cropped the characters resulting from this procedure and used our recognition models to make predictions for each. We first ensembled the predictions for the ResNet-50 SimCLR and DeiT-small models separately, using a hard majority voting approach. We then ensembled the YOLOv8 classification with the ResNet-50 SimCLR and DeiT-small overall votes, by adjusting the bounding box confidence by the proportion of votes for each method for a particular character.

Fig. 1a shows the mean average precision results of cross-validation on the training dataset. Increasing the resolution yielded a small improvement for both detection and recognition. But adding pseudo-labels (fig. 1b) from predictions of the model on the Oxyrhynchus Papyri images resulted in a 6–8% increase for detection and a 8–11% increase in recognition.

Fig. 1c shows results on the test set for YOLOv8 models trained on the five cross-validation partitions and at the three image resolutions. We achieved better results with higher resolutions but with diminishing returns. The average mAP score for detection across the five cross-validation partitions at the small resolution was 48.7%. This increased to 49.5% at the medium resolution but a further increase to the large resolution only raised the average mAP to 49.6%. In terms of letter recognition, the lower resolution gave a mAP of 35.8% while the medium and large resolutions gave 37.3% and 37.6% respectively. Ensembling the five models resulted in a distinct improvement for both detection and recognition. The final ensemble combining the YOLOv8 models at all resolutions produced the best results with a mAP score of 51.8% for detection and 41.4% for recognition.

The results of the models trained specifically for recognition are shown in fig. 2. The YOLOv8 ensemble outperformed both the DeiT and the SimCLR models. But when they were ensembled, we achieved the highest recognition mAP score of 42.16%. This was our ultimate submission for the competition. The ensembling process adds new bounding boxes and modifies their confidence scores to account for each of the characters predicted to be present, which results in the slightly lower detection mAP of 51.42% for a larger gain in the recognition mAP.

We found that the SimCLR ResNet-50 and DeiT-Small models performed similarly in identifying the Greek character present in an image. Averaging the cross-validation results, we obtained 88.4% accuracy for the former and 87.4% accuracy for the latter (fig. 2a). The combination of these models achieved 89.1% accuracy.

The supplementary material contains a confusion matrix for our ensembled predictions, which shows that we obtained reasonable recall for every character, even those poorly represented in the dataset. The most common mistakes are between characters having a similar appearance, such as A and $\Lambda$ or Z and $\Xi$.

The results of our submission in the detection task is shown in fig. 3a. In the primary metric (and at an IoU of 0.75) our submission was behind that of Vu & Aimar, giving us the second place for this task. At the lower IoU value of 0.5, we achieved a much higher mAP at 93.2%. This was the highest of all submissions. This indicates that while our bounding boxes were not as localized as the winning submission, our method was still able to accurately identify the vast majority of letters. In common with the other submissions, the recall scores were substantially better than the precision scores. We achieved an average recall of 98.6% at an IoU of 0.5 and 81.2% at an IoU of 0.75, which were the best of all submissions.

The results of our submission in the recognition task is shown in fig. 3b. In the primary metric we achieved a mAP of 42.2%, which was the winning submission for this task. As with detection, the result for an IoU of 0.75 was not as good as the submission by Vu & Aimar but at an IoU of 0.5 our result was 74.7%, more than 2% higher than the next closest submission. Our submission also ranked highest on average recall for both an IoU at 0.5 and 0.75.