DeepScribe: Localization and Classification of Elamite Cuneiform Signs Via Deep Learning

Tools to partially automate or aid human annotation would enable the rapid creation of richly annotated digital archives. The success of machine learning algorithms in image analysis tasks both inside and outside of document recognition (Krizhevsky et al., 2012) (Zhou et al., 2021) (Lin et al., 2017) (Ren et al., 2015) (Chung and Delteil, 2019) (Hodel et al., 2021) and the proliferation of open-source software tools for developing machine learning products (Buitinck et al., 2013)(Wu et al., 2019)(Paszke et al., 2019) present the possibility of using these tools for the analysis of historical documents.

Extensive research on computer vision methods for document recognition has been conducted over the past 30 years, resulting in software packages that are widely used in both research and industry (Smith, 2007) (Lee and Smith, 2012). Such methods rely on software pipelines that identify lines of text, segment lines into words or characters, and identify characters. Handwritten text recognition, due to the relative irregularity of handwritten characters, typically requires dedicated processing pipelines (Chung and Delteil, 2019) (Renton et al., 2018) (Graves et al., 2007) (Hodel et al., 2021) but can be performed effectively by an automated system, with character error rates under 8% (Bluche, 2016). Full-page handwriting recognition methods typically segment an image into text lines via object detection methods that are then decoded using an alignment-based loss function (Wigington et al., 2018) (Chung and Delteil, 2019), although segmentation-free methods exist and are known to be effective (Bluche, 2016). We apply a similar factorization, first using an object detector to identify cuneiform signs and then to classify them, although our access to per-character bounding boxes allows us to avoid the problem of line segmentation. In the context of the PFA, text lines are often curved, which presents a problem for standard line segmentation techniques. Methods for detecting curved text lines exist (Zhou et al., 2019), but the PFA dataset we utilize does not possess line-level annotations. Training modularized subcomponents of OCR systems has precedent in the transliteration of modern texts (Chung and Delteil, 2019), but the primary purpose it serves here is to improve our understanding of the novel PFA dataset from the perspective of computer vision. Modeling OCR as a combination of object recognition and transliteration tasks is fairly common in the field (Wigington et al., 2018) (Chung and Delteil, 2019), but it is relatively rare to perform OCR on the word or character level for Latin scripts. However, prior work transliterating Chinese characters (Yuan et al., 2018) has adopted a per-character object recognition approach, showing that different linguistic contexts often require customized pipelines.

The success of these methods has inspired their application to historical document analysis, developing tools to automatically extract useful semantic information from documents that are typically only readable by specialized scholars. For instance, (Colutto et al., 2019) describes a text recognition system intended for use on handwritten historical documents, as well as a publicly available software tool that allows for fine-tuning on user-provided corpora.

(Bogacz and Mara, 2022) provides an excellent review of projects applying techniques from computer vision to cuneiform tablet images, as well as a typology of the diverse methodologies that have been adopted to analyze these data sets. While the relatively small datasets of 3D-scanned or photographed cuneiform script typically limit the utility of the large deep learning models that are often successful in addressing computer vision problems, digital humanists and computer scientists have spent decades building automated systems that can leverage what data exists in archives such as the Cuneiform Digital Library Initiative and ORACC. (Bogacz and Mara, 2018) uses a keypoint-based method to identify cuneiform signs on vector graphics representations of cuneiform text, but this requires the manual creation of a vectorized transcript before any analysis can be performed.

The recent work of (Dencker et al., 2020) uses a weakly-supervised algorithm to learn to detect cuneiform signs from a 2D image dataset of Neo-Assyrian tablets found on ORACC (Tinney and Robson, 2019), which are linguistically distinct from our Elamite tablet set. Due to the lack of annotated sign localizations in the ORACC dataset, they develop an iterative learning procedure using a Single-Shot Detector (SSD) object detector algorithm (Liu et al., 2016) along with line alignment techniques to iteratively propose, filter, and retrain on automatically generated sign localizations. Additionally, this work includes a Conditional Random Field (CRF) (Lafferty et al., 2001) sequence-modeling step as the final stage of the annotation-alignment pipeline, but this uses geometric and spatial information to match predicted and classified regions to transliteration characters, rather than using contextual information to refine the raw predictions of a sign detector. They also find a boost in accuracy when their models are provided with a small set (¡ 60) of fully annotated tablet images. Our approach distinguishes itself from this effort firstly by utilizing a novel corpus of Elamite tablets, and secondly by directly training on localized sign annotations rather than leveraging weak supervision. However, we see several promising avenues for future works in the combination of these two approaches, such as the leveraging of a model trained on a large dataset of annotated signs to initialize weakly-supervised learning methods on a low-resource dataset, although there are many questions to be answered on the efficacy of cross-linguistic transfer. Given the large linguistic differences in the two corpora (Neo-Assyrian Akkadian and Elamite), metrics are not directly comparable between the two methods.

An orthogonal approach that that may be better suited to text written on curved tablets uses 3D scanning information as input to cuneiform tablet analysis. However, acquiring 3D-scanned data requires far more time, labor, and cost than 2D image data. In addition, acquiring permission to 3D scan tablets from museums is often difficult. (Bogacz et al., 2015) uses 3D models of cuneiform tablets to extract vector drawings and retrieve the corresponding cuneiform characters. (Bogacz and Mara, 2020) also uses 3D data, but predicts the time period of a cuneiform tablet instead by training a Convolutional Neural Network (CNN) on a partially transliterated cuneiform tablet 3D surface mesh dataset. (Kriege et al., 2018) addresses the geometric nature of cuneiform script by classifying signs based on graph model data of cuneiform signs extracted from 3D scans using the method of (Fisseler et al., 2013), but still must rely on a small set of annotated data. (Rusakov et al., 2019) attempts to resolve this by generating artificial 2D projections of 3D data using a Generative Adversarial Network (GAN) (Goodfellow et al., 2014), although annotations still must be provided manually for downstream classification tasks. We instead focus on the problem of recognizing 2D signs from flat images of a tablet, due to the nature of the annotations present in our dataset.

In 1933, the University of Chicago’s Institute for the Study of Ancient Cultures (ISAC, formerly Oriental Institute) discovered a large cache of cuneiform tablets at the site of Persepolis. The tablets were discovered in two rooms inside a fortification wall and were dubbed the Persepolis Fortification Archive (PFA) (Stolper, 2007b). Scholars at the ISAC and world-wide have been studying and publishing editions of these tablets since 1937 (Hallock, 1969). The tablets provide valuable insight into the administration and economic history of the Achaemenid Empire, and their digitization enables the rapid search and indexing of this large corpus. The majority of the PFA texts record the distribution of commodities and allocation of resources, which has led to a greater understanding of the internal administrative details structure of Persia (Henkelman, 2013). In 2006, a renewed effort to document and publish the texts led to the creation of the PFA dataset as it exists today. University of Chicago professor Matthew W. Stolper assembled an international editorial and technology team to work on the PFA project. The team included specialists in Elamite and Aramaic language, seal iconography, digital imaging, and data representation. The editorial team worked on new editions of the texts. The seal iconography team faced the massive task of identifying, classifying, and studying the thousands of unique seal impressions recorded on the tablets (Garrison and Root, 2001). The photography team instantiated three separate digital photography modalities: conventional digital photography, high-resolution scanning under filtered light, and Polynomial Texture Mapping (also known as Reflectance Transformation Imaging) (Stolper, 2007a). The database team consulted with the rest of the PFA team to record all of this information in an integrated system called the OCHRE database platform.

The PFA dataset was annotated with character bounding-box level annotations using the Online Cultural and Historical Research Environment (OCHRE, ochre.uchicago.edu), a research database platform designed for humanities and social science data. Each clay tablet, each digital photograph, and even each cuneiform sign in each text is a discrete database object, described by metadata and associated with hundreds or thousands of other database items.

In OCHRE, the PFA editorial team has created text editions in which each cuneiform sign was recorded as a discrete database item. Each sign reading represents an interpretation by a PFA editor, reached through careful examination of the clay tablet under magnification which leads to a reading of the signs and translation. As shown in Figure 2, PFA team members use OCHRE to associate manually-created tablet image hotspots with transliterations of the text in question. Linguistic information is therefore incorporated at the sign annotation level even if individual signs may be difficult for humans to read in isolation - i.e., if a sign is difficult to read but its value is inferred via linguistic context, its bounding box will still be present. To account for the polyvalent nature of the logographic and syllabic cuneiform writing system, each sign is linked with both its sign identity and its syllabic value. For example, the sign GIŠ can communicate syllabic values is/ez, giš and bil as well as the logographic usage meaning ”wood” and the determinative usage which refers to wood and wood-based products. The identification of the value of a sign must be determined by context. In this stage of the DeepScribe project, we set aside the sign–to–value mapping problem and focus entirely on sign identity prediction. In future work, we plan to take advantage of the extensive PFA glossary of Elamite words in OCHRE to suggest sign values and word identifications. The original intent of the PFA annotation effort was to create a pedagogical tool to help teach the reading of Elamite. Instead of providing students with a hand-drawn version of a cuneiform text with highly regularized cuneiform signs—as it is often the case in the classroom—these annotated images would allow the student to engage with the cuneiform text as recorded by the scribe.

The sign-bounding-box level annotations provide a dataset that is well suited for the application of off-the-shelf computer vision algorithms. Unlike other labeled cuneiform datasets which typically consist of whole-tablet images paired with transliterations (Tinney and Robson, 2019), the PFA tablet images in OCHRE are labeled much like object recognition datasets such as COCO (Common Objects in Context) (Lin et al., 2014). Each image of a PFA tablet is paired with a set of labeled bounding boxes, which can either be used out-of-context to classify cuneiform signs, or in-context for sign bounding box detection. We explore both in this work as part of a modular recognition pipeline. The dataset is composed of images of tablets themselves, not line drawings or artificial constructs - the dataset even contains natural handwriting variation among scribes. This allows computational models to learn directly from cuneiform text as it was written by scribes. We anticipate that this will produce models capable of recognizing Elamite cuneiform text in situ, without requiring human scholars to perform tedious pre-processing before annotations can be generated.

The full dataset used in this work consists of 5007 images of 1360 unique tablets, where each image contains a set of hotspot region annotations. We find that the relative frequencies of the 141 unique Elamite signs contained within the dataset are highly unbalanced, with the vast majority of signs being fairly poorly attested. As expected from a natural language dataset, the dataset displays an approximately Zipfian distribution (Piantadosi, 2014). High performance on high-frequency signs can easily “wash out” low performance on low-frequency signs on aggregate metrics. As would be expected from a power-law distributed data set, the top 50 most frequent signs cover 86.1% of the dataset, indicating that the remaining majority of sign classes are poorly represented. As such, there may be a trade-off between a system focused on low-resource signs and a system capable of automatically annotating “easy” or high-resource signs then flagging others for human intervention. We explore the relationship between class frequency and performance in Section 5.2.1. We perform basic processing to consolidate redundant class labels and automatically remove unannotated image regions, described in more detail in Appendix A.2.

A public subset of the dataset, consisting of 1239 tablet images, is available in JSON format ²²2See https://pi.lib.uchicago.edu/1001/org/ochre/ad64d9db-de4a-426c-81d5-10bf292ffd0e for links to the project code and data.. The “full” dataset used in this work contains unpublished tablets courtesy of the PFA project. These tablets can only be released after the primary publication of the cuneiform texts. We provide code examples of training and evaluating models on public data using Google Colaboratory (goo, [n.d.]). We find that models trained only on the public subset achieve somewhat reduced performance on held-out test data, consistent with the scaling explored in Sections 5.1.1 and 5.2.1.

We use a series of metrics designed to evaluate model performance on subtasks within our dataset, in addition to the general task of end-to-end cuneiform sign localization.

The structure of our modular computer vision pipeline is designed to explore two sets of questions: both to understand performance on the individual subtasks presented by our dataset, as described in section 3.5, as well as to combine individually trained components to produce an end-to-end OCR system. While it is certainly possible to model this dataset using end-to-end object recognition systems (possibly with an additional linguistic refinement component), the modular structure of this workflow arose from our exploration of individual tasks that could be accomplished within the dataset. Our vision pipeline consists of three stages: sign detection, sign classification, and line detection (as shown in Figure 1). Rather than jointly training a model to produce hotspots and classify signs, we decided to split the problem into two independent modeling steps, inspired by (Chung and Delteil, 2019). As such, each stage in the pipeline is trained separately using ground-truth annotations. Potential users of this modeling pipeline identified being able to manually adjust suggested hotspot regions as a key feature, which was facilitated most easily by suggesting hotspots first and then performing inference using a different network. We performed experiments with an end-to-end single-stage RetinaNet (Appendix F), but found similar performance at the cost of modularity and a reduction in interpretability. We leave exploration of additional pipeline variants to future work. We also perform separate analyses of learning tasks within the PFA dataset. These serve to characterize the “difficulty” of each learning task (i.e., hotspot localization, sign-classification) given the novelty of this dataset for machine learning training. There may in fact be synergies that an end-to-end system could exploit, such as using contextual information to refine classification prediction, although these could also be achieved using a final, modularized language modeling step.

We define cuneiform sign localization as an object detection problem - i.e., selecting a single rectangular bounding box for each cuneiform character. All boxes are assigned to an identical class and an object detection algorithm is trained to distinguish the class against background regions. We use a RetinaNet (Lin et al., 2017) as implemented in the Detectron2 library (Wu et al., 2019) to perform sign localization. The RetinaNet is trained using the Adam optimizer (Kingma and Ba, 2015), with an initial learning rate of 1e-4 and a learning rate scheduler that decreased learning rate when performance on validation loss plateaued. Before training, images are rescaled using preset per-channel mean and standard deviation values originally defined on ImageNet. Images are augmented using a mixture of image rescaling, random cropping, and random horizontal flipping and fed into the network in batches of 10. We note that flipping transforms, especially vertical flipping, are problematic in the case of cuneiform character classification. Obviously, many cuneiform characters are not invariant to rotations or reflections. However, at the detector stage, the model is not concerned with the specific identity of a character and is designed to recognize any character regardless of value. The network is limited to a maximum of 40000 training iterations, with early stopping performed once the model’s validation performance ceased improving. Final performance metrics on validation folds are computed using the best iteration as judged by AP@50.

We use a ResNet (He et al., 2016) image classifier to map tablet image regions containing a single sign to a sign class. The classifier maps an image to a 141-dimensional vector of logits, which we use to rank predictions and compute top-k results. Image regions are resized and padded to 50 x 50 pixels before input into the network. During initial experiments, we found that data augmentation (RandomAffine, ColorJitter, and RandomPerspective as implemented in TorchVision (maintainers and contributors, 2016)) as well as standard BatchNorm layers are effective at preventing overfitting, and thus do not apply any kind of weight decay. Early experiments indicated that additional image processing such as Gaussian smoothing and Otsu thresholding (Kurita et al., 1992) provided no benefit during training - likely due to such transforms being well-captured by a ResNet’s convolutional filters. The classifier network is trained using the Adam optimizer, with initial learning rate 1e-3 for a maximum 500 epochs across all experiments. A learning rate schedule with a patience of 5 epochs is used to reduce learning rate by a factor of 10 when validation loss plateaus. After 10 epochs of no improvement, learning is terminated. The classifier was implemented in PyTorch (Paszke et al., 2019) using the TIMM library (Wightman, 2021) of predefined model architectures. Statistics were computed at the final epoch of training. Interestingly, we find that using pre-trained models (obviously trained on a very different domain) improve convergence time but not validation performance. To remove unexpected confounding variables, we perform all analyses in this paper on randomly initialized models. We train all classification models on ground-truth, human-annotated hotspot regions and perform tests on both held-out ground-truth hotspots and hotspots predicted by the object detector described in Section 4.2.

We experiment with several methods to address severe class imbalance during classifier training, including reweighting per-class loss using inverse frequencies (King and Zeng, 2001), reweighting with inverse log-frequencies, and using the focal loss (Lin et al., 2017) that was also used to train the RetinaNet sign detector. However, results are inconclusive (see Appendix C) and we perform end-to-end inference using a classifier trained using the unweighted cross-entropy loss.

To sort detected hotspots into discrete “lines” facilitating the left-to-right reading of Elamite texts, we use a variant of the Sequential RANSAC algorithm (Vincent and Laganiere, 2001) to iteratively assign elements of the set of hotspot centroids on a tablet to a set of linear models. The algorithm fits a series of L2-regularized RANSAC linear models to the data. Regularization serves to produce lines as flat as possible, and we force each linear model to have a slope less than 0.3 as a further constraint. These parameters, tuned manually, are found to encode our prior belief that lines in this corpus are generally flat or very slightly slanted and rarely intersect. Outliers from the initial RANSAC procedure are used to fit additional RANSAC linear models until there are one or fewer points remaining. Final outliers are assigned to their nearest text line by Euclidean distance. The set of text lines are sorted by their y-intercepts to provide a full left-to-right reading order across multiple horizontal lines. Note that this method of sorting could be problematic for highly slanted lines, which are relatively rare in cuneiform texts, but in practice the majority of the text lines in the PFA are flat enough to avoid significant issues.

Cross-validation results of the RetinaNet’s single-class object detection performance can be found in Table 1. We observe relatively consistent behavior across folds, indicating that each fold provides sufficient coverage of the sign dataset for the purposes of this training task. We find that increasing the backbone size from 18 ResNet layers to 50 ResNet layers provides no statistically significant improvement in aggregate detection performance, and we also find minor underperformance with 101-layer backbones. This suggests that, on this dataset, it is unlikely that further increases in network depth with the same architecture and loss will provide improvements in performance. As a result,we experiment with using the ResNet18 and ResNet50 backbones for end-to-end experiments in Section 5.2.4.

We report sign classification results on ground-truth annotated hotspots in Table 2, and observe that the top-1, top-3, and top-5 accuracies of ResNet-based classifier models are relatively consistent across folds. We see an increase in classification accuracy as the depth of the ResNet model increases, although the change between an 18-layer ResNet and a 50-layer ResNet is much smaller than that between a 50- and 101-layer ResNet. We also investigate the relationship between a sign’s frequency in the training set and its per-class test performance in Table 3. We find that the mean recall, sometimes referred to as the balanced accuracy (Mosley, 2013), increases with network depth, although in a diminishing manner. However, we find that the rank-order correlation between the per-class test recall and the class’s train frequency is very high. Similarly, per-class test precision is correlated with the class’s frequency in training data. This is unsurprising given the extreme imbalance in class frequencies, and provides a limit to the utility of the models—they are likely to be be more useful in automating the annotation of well-attested signs than predicting results for rarer signs. We provide a more detailed exploration of learned sign class representations in Section 5.2.3. We also hypothesize that providing sign ranking information could improve the performance of a language model attempting to predict the value of a given sign in context, but we leave this exploration for future work.

Previous sections evaluate classification performance with respect to ground-truth bounding boxes, but ultimately the utility of this computer system will be evaluated by its performance on completely unannotated tablets — i.e., where no ground-truth annotations for intermediate pipeline stages are even present. As such, the estimates of classification performance presented earlier are likely upper bounds. We approach this final evaluation in stages: first, by evaluating the performance of the sign classifier on bounding boxes predicted by the sign detector, where ground-truth labels have been imputed by alignment to ground-truth hotspots, and finally by evaluating the end-to-end character error rate (CER) of transcribed tablets. Under the existing pipeline, the results provided to the user will consist of predicted bounding box locations for each sign along with a top-5 set of predicted sign identities for each bounding box. Additionally, the pipeline returns a putative horizontal line assignment for each predicted bounding box, which can be used to construct an end-to-end transliteration. This allows users to incorporate the ranked list of sign predictions into their annotation workflow without being overly constrained by the model’s top-1 prediction.

Table 6 shows the classification performance of the sign classifier trained on ground-truth bounding boxes, as described in Section 4.3, when performing inference on bounding boxes predicted by the sign detector described in Section 4.2. We observe a degradation in performance when evaluating on predicted signs, likely due to some of the issues that we observe with the sign detector—erroneous joining or splitting of signs, as well as alignment errors. An end-to-end hotspot detection and labeling training process may provide improved performance here by learning to compensate for shifts in input data distribution induced by the hotspot detection algorithm. To evaluate the CER of transcribed tablets with the current pipeline, we use predicted hotspot locations and top-1 predicted sign labels to predict text ordering using the Sequential RANSAC line detection algorithm described in Section 4.4. The line detection step groups detected signs into lines, which are ordered top-to-bottom using their Y-intercept. This produces a left-to-right and top-to-bottom reading order sequence of the entire tablet which can be compared to the original tablet’s transliteration. We observe a relatively high CER, which indicates that the current pipeline is unable to reconstruct text sequences. A top-1 accuracy of approximately 0.56, along with probable alignment errors, is a promising step towards automatic tablet transcription, although insufficient.

The combination of each model’s pathologies, particularly those of the sign classifier, render the combined pipeline unable to accurately transcribe tablets. This result primarily highlights the need for explicit linguistic supervision and the limits of a vision-only approach to tablet transliteration. The noisy outputs of the sign classifier, while useful for providing suggestions to cuneiformists, most likely require effective denoising using linguistic and contextual information.

We have demonstrated that the large and richly annotated Persepolis Fortification Archive enables direct training of computer vision models to recognize and classify Elamite cuneiform signs. A trained object detector can localize cuneiform signs with high precision and qualitatively useful performance. A separately-trained classifier is able to produce high-quality predictions of sign identity. These two systems on their own are sufficient to aid cuneiformists in annotating and identifying signs in Elamite tablet images, although the current iteration of our end-to-end pipeline has insufficient top-1 sign classification accuracy to produce complete transcriptions automatically without any cuneiformist intervention. To our knowledge, this work presents the first computer vision-based analysis pipeline on Elamite cuneiform, demonstrating practically useful performance using a novel dataset. The pipeline presented in this work will serve as a basis for future work on Elamite cuneiform tablet transliteration. We have released a processed subset of the PFA dataset as well as the source code and trained model parameters for our pipeline.

While our pipeline can effectively localize cuneiform signs and provide a list of suggested readings, the current system was not built to provide complete transliterations. Our future work will seek to rectify this by incorporating linguistic supervision to improve the quality of sign predictions and to provide sign values. This could be achieved in a modular fashion by using a language modeling layer on top of the existing pipeline, or incorporated into the detection phase via a context-aware object detection method such as (Chen et al., 2018) and (Carion et al., 2020). The PFA in OCHRE contains sign to value mappings for each hotspot annotation, so we anticipate this to be feasible on the dataset. However, the latter approach would likely require adopting an end-to-end sign detection and identification scheme rather than the modular scheme we currently adopt.

A second avenue of future work consists of applying the hotspot detector to non-Elamite tablets to determine whether or not a hotspot detector ostensibly trained without explicit linguistic information can identify signs in other cuneiform corpora. While it is likely that there is some bias towards the specificities of Elamite cuneiform, preliminary experiments (Appendix G) have indicated that the detector component of our pipeline can localize signs (without identifying their value) on Ur III Sumerian tablets that predate the tablets in the PFA dataset by over a millennium and a half. However, we suspect that the vast linguistic differences between the corpora will limit the performance of a vision-only or single-language model. The method of (Dencker et al., 2020) was shown to perform reasonably well on a different cuneiform corpus, indicating that there is some transferability between corpora given a trained detector. A combination of our methods and the semi-supervised methods described by (Dencker et al., 2020) may be useful here—for example, using our models (pre-trained on a large annotated dataset) to initialize semi-supervised learning on another cuneiform corpus.