Temporal Sequencing of Documents

The accurate dating of historical and heritage texts is of paramount importance to historians. On the basis of such correctly sequenced texts, historians can examine, judge, and analyze events within the context of a specific time period. Often, only the undated textual contents of the historical documents are available to historians, on the basis of which they must infer the dates of composition (Gervers, 2002). English property-transfer documents (charters or deeds) were selected as one component of this study because of their particular nature. That is, while the earliest surviving examples from the Anglo-Saxon period (c. A.D. 670 to 1066) are invariably dated, only 300 out of a total of approximately 1,600 can be considered originals. Experts have noted that many supposed Anglo-Saxon documents are actually later forgeries, but nonetheless are difficult to distinguish from genuine charters; “Thus in some cases, the date given is either demonstrably fictional or suspect” (Robert Getz, personal communications, April 25, 2019) or, in many cases, the charters survived only in later copies made centuries after the date of issue, resulting in genuine errors arising from misreading or miscopying by scribes. Common instances include miscopying of Roman numerals, confusion about a witness’s identity, or the names of reigning monarchs (Whitelock, 1979; Cubitt, 1999). For examples of documents allegedly forged or fabricated to justify social realities and for political motivations, see Hiatt (2004).

When the Anglo-Saxon political and judicial system was largely replaced by that of the Normans following their conquest of England in 1066, an entirely new phenomenon was introduced: the undated charter. From 1066 to circa 1307 (the start of the reign of King Edward II) only about 3% of the million or more known charters issued bear internal dates. Dating was reintroduced to the royal chancery in 1189 under King Richard the Lionhearted, but the example was not followed by the nobility and commoners for five score years more. Compared to Continental charters, which with few exceptions were regularly dated internally for the duration, the first 600 years of the English charter record has always floated on a sea of incertitude.

In historical research, the one essential principle is to identify the correct order of events. As is evident from the foregoing, that is one of the most difficult tasks in the profession. It is of far greater concern, however, than for historians alone; in fact, it is common to virtually all avenues of scholarship, not to mention of the many institutions upon which literate society depends. Undated documents are everywhere, leaving lawyers, police and spy agencies, forensic linguists, code breakers, artists and art historians, businesses, real estate agents, medical practitioners, military analysts, philosophers (the list is endless), with the responsibility of determining what event preceded or succeeded another. This study sets the stage for anyone with a series of undated, digitized texts, or even lists, to determine a chronological order thereof without having to undertake the arduous task of examining each document for contextual clues and references to specific events, and of identifying periodization through content analysis, handwriting and/or watermarks. All of these aspects can be accomplished automatically through the temporal sequencing methodology outlined below.

By way of examples in which correct temporal ordering was essential, we note that in the financial fraud investigations of Enron Corporation, forensic linguistics was used to analyze emails, memos and internal communications to re-construct the timeline of fraudulent activities even when the timestamps of these evidential materials were not always available (McLean and Elkind, 2003).

The Library of Congress contains many written documents from former presidents of the United States. For example, The Papers of Abraham Lincoln¹¹1https://www.loc.gov/collections/abraham-lincoln-papers/ is a digitized corpus of over 40K documents consisting of the correspondence, notes, memos, personal letters, drafts of speeches of Abraham Lincoln from his time as a lawyer, congressman and then as the 16^th president of the United States. Chronological gaps in The Papers remain, as not all of the original letters and documents were meticulously dated or preserved. A proper chronological order would give insight into the President’s evolving thoughts and ideas through a tumultuous period of American History. Another similar example is The Papers of Thomas Jefferson²²2https://www.loc.gov/collections/thomas-jefferson-papers/, a digitized corpus of 25K items consisting of the correspondence of Thomas Jefferson, who was a diplomat, architect, lawyer and the third president of the United States. Besides correspondence, the collection also includes his drafts of The Declaration of Independence, drafts of laws, fragments of his autobiography, his personal notes, including his records of spending and even recipes! Establishing an accurate chronological order of The Papers is crucial in understanding the personal worldview and the evolving visions about the early Republic by one of the prominent Founding Fathers.

The medieval Exeter Book³³3https://www.exeter-cathedral.org.uk/learning-collections/explore-the-collections/the-exeter-book/ is another example. It is an anthology of Old English poetry and riddles from the 10^th century, but the chronological order of none of the texts is known. Establishing a chronological order of the texts would give us a deeper understanding of the evolution of Old English and the literary culture of the Anglo-Saxon people.

Previous efforts in document sorting have been directed towards the development of historical language models (Feuerverger et al., 2005, 2008; Tilahun et al., 2016; Gervers et al., 2018). Within the broader field of information retrieval, investigators employ statistical models incorporating temporal aspects of term usage (Swan and Jensen, 2000), study the relationship between time and the retrieval of relevant documents (Li and Croft, 2003), and classify document dates according to time partitions of predefined granularity (De Jong et al., 2005). In preparing web pages that present document lists the accuracy of time-stamping is of paramount importance. In this regard, Kanhabua and Nørvåg (2008) and Kanhabua and Nørvåg (2009) extended De Jong et al. (2005)’s work by integrating semantic-based techniques into the document pre-processing pipeline, the aim being to improve the temporal precision of web pages and web document searches where the trustworthiness of document time-stamps is often questionable. Lebanon and Zhao (2008) modelled temporal text streams, and Mani and Wilson (2000) extracted temporal expressions such as now, today, and tomorrow from print and broadcast news documents to resolve uncertain temporal expressions. Chambers (2012) presented models that incorporate rich linguistic features about time, whereas Vashishth et al. (2019) employed deep neural network-based methods that also exploit linguistic features about time in order to date the documents. However, as pointed out by Kotsakos et al. (2014), relying upon temporal terms for dating suffers from the drawback that terms “can be very sparse as well as ambiguous, referring to irrelevant timeframes”. These authors proposed using statistical approaches based on lexical similarity and burstiness — the sudden increase in the frequency of terms within a time-frame.

In the current work, we propose TempSeq⁴⁴4The TempSeq pseudo and source codes used in this paper are available at https://github.com/gitgelila/TempSeq , an unsupervised method for the temporal sequencing or ranking of documents. This approach is designed to be applicable when the only available data are the undated documents that are to be temporally sequenced. TempSeq relies on a ‘bag-of-words’ approach, and does not make use of linguistic features about time, nor does it use a training set data with time-tag. In addition, our approach does not make use of specific language rules, word representations, or any other metadata information, thus presenting a potentially significant advantage in the task of document temporal ordering. TempSeq relies on measuring word usage drift under the assumption that word usage changes gradually over time, which means that the temporal variability in word usage is low. We model word usage drift via the non-parametric Generalized Linear Models regression (Fan et al., 1995), and we estimate the correct temporal sequencing of the documents to be the one that minimizes, on average, the associated kernel bandwidths (a direct measure of the temporal variability in word usage). To our knowledge, using the variability of word usage drift to ascribe a temporal sequencing for a set of documents is entirely new.

The necessity for temporal sequencing of documents arises not only in the field of information retrieval, but also in studies of heritage texts, which frequently lack timestamps, are intentionally ambiguous with respect to time of issue, or can even be outright forgeries. Often, only the textual contents of heritage documents are available, from which one must infer the dates of issue (Gervers, 2002). Furthermore, heritage texts that are available as a training data set can be limited in number, as proportionately few documents have survived across the centuries to the present time, thus necessitating an unsupervised method for inferring document dates. The task at hand is not only to infer the temporal ranking of a collection of documents or corpus, but also to identify the terms that contribute most towards the task of identifying the correct temporal ranking/ordering. We believe that such terms are most likely the temporal signatures for identifying the characteristics of intentionally or inadvertently temporally mislabelled documents, or documents with missing or corrupted timestamps.

Past and present research about problems arising in document sequencing involve criteria for document classification, for example, by topic, Blei et al. (2003); Blei and Lafferty (2006); McAuliffe and Blei (2008); Taddy (2013, 2015), document indexing Roberts et al. (2016), and document ranking using latent semantic analysis Deerwester et al. (1990); Hofmann (1999). Cohen et al. (1998) consider the problem of machine learning to order instances, as opposed to classifying them, when an algorithm’s output receives feedback in the form of preference judgments, i.e., “statements indicating which instances should be ranked ahead of the others” (Cohen et al., 1998). However, such lines of research have not directly addressed the problem of document ordering per se. Perhaps the closest approach to dealing with this problem is that of Thinninyam in her dissertation (Thinniyam, 2014). Her approach is based on the notion of similarity (or distance) between two documents, namely the supposition that similar documents are more likely to discuss similar topics, and therefore should have closer underlying temporal footprints. She proposed a linear regression-based approach, with regression of observed document distance measures of spacings between consecutive documents. In a separate approach, Thinnayam also framed the document ordering problem in a Bayesian framework, where the distribution of pairwise distances between documents was modelled conditionally on a timeline vector, where the $i^{th}$ coordinate value of the vector represents the time interval between the $i^{th}$ document and a reference document. Herein, a Markov Chain Monte Carlo (MCMC) method was employed to sample from the posterior distribution of the timeline. Thinniyam’s ordering methods fundamentally require pairwise distance measures of documents (i.e., a quantifiable measure of dissimilarity between two documents) to estimate the temporal order of a set of documents within a corpus. Other studies have shown that such measures are prone to yield spuriously high values of similarity due to an abundance of uninformative terms within the documents, including, but not limited only to stop words (Tilahun, 2011); it is not always a straightforward matter to identify these uninformative terms, and the requisite degree of filtration. Moreover, using measures of document distance does not allow identification of the particular words or terms that are essential for determining the predicted temporal rank orders. In addition, the degree to which two documents are similar/dissimilar is highly dependent on the type of distance measures that are used Broder (1997).

In contrast to Thinnyam’s previous approach, the present TempSeq method temporally ranks a set of documents even when a reliably dated training dataset is not available, and/or when there is a very limited number of documents in the set. The TempSeq method relies fundamentally on modelling the probability of occurrence of words in a given date range, thereby avoiding the need for a document distance measure. By design, TempSeq also filters out words according to their degree of uninformativeness, thereby allowing us to gain insights into the history underlying the documents by identifying words that are most putatively useful for determining the correct temporal ordering of documents. We test the TempSeq method on two corpora of heritage texts; one written in American English and the other in Latin. When a set of training data is available, TempSeq can optimize the smoothing parameter for temporal sequencing. More importantly, we show for both corpora that the TempSeq temporal sequencing method performed significantly better as compared to random sequencing, in the absence of training data.

We evaluated our temporal sequencing methods on two different sets of corpora with time-tags. The first corpus consisted of 240 transcripts of the American State of the Union Address (SOTU), from the years 1790 to 2020. Each transcript had a median average length of 6400 words. This corpus is available from the R package, Arnold (2022). The second corpus is from the Documents of Early England Data Set (DEEDS)⁵⁵5https://deeds.library.utoronto.ca. From within this corpus, we focused on a set of 11,463 English property conveyance records issued in the years 1120 to 1300. All the records are written in Latin, and have been inspected for content by subject expert historians to accurately verify the date of issue. The Latin documents have a median length of 175 words. We chose to evaluate the TempSeq temporal document sequencing method on this corpus as it consists of documents similar to corresponding DEEDS documents from the Anglo-Saxon period, which, as mentioned in section 1, have generally unreliable dates. In this project, we considered the DEEDS corpus in two different forms. In the first form, we conflate all the documents written in a given year into single texts, thus yielding 181 conflated DEEDS records, of mean average length of approximately 11,000 words. We denote the conflated collection as “DEEDS-conflated”. In the second form, we denote the entire set of 11,463 unconflated records as “DEEDS-single”.

When a set of training documents with known dates is available, Tilahun et al. (2012) have proposed the “Maximum Prevalence” method for their dating. This approach is based on modelling (on the basis of the training data) a curve that describes the temporal pattern of the probability of occurrence of each word from the undated documents. For example, in the DEEDS corpus, the proposed dating method achieves very reliable date estimates, giving a test set median dating error of $\pm$5 years within the 230 year span  (Gervers et al., 2018). High accuracy validates an underlying feature of the model, namely useful words for dating a document are those with a non-uniform probability of occurrence across a date range, and showing a gradual change in the variability of their usage changes. Words such as et, de, huic (in Latin), the, to, that, on (in English), and stop words, which appear in consistent proportion at all times, that is to say uninformative words, do not contribute to the date estimation of an undated document.

In section 4, we discuss our modelling approach to estimate the curves best describing the temporal pattern of the probability of occurrence of a given word/phrase, and present examples of such curves. We also examine the properties of the curve estimates (in particular, that of a smoothing parameter) in relation to the bias-variance trade-off, using the form of the bias-variance trade-off to select the optimal curve. This trade-off is at the heart of any statistical learning process. We could perfectly fit training data (zero bias) to a model by including excessive amounts of parameters (excessive under-smoothing in the case of our model). This situation, referred to as over-parametrization, risks overfitting the data because the model learns not only the pattern in the data but also the random noise and fluctuations that are present. When an overfitted model is applied to a test data, the performance is often poor. On the other extreme, when a model is under-fitted (over-smoothed in the case of our model) the effect from random noise is eliminated but at the expense of failing to learn the pattern from the data. The right amount of parameterization is one that balances bias and variance (large bias and low variance). We seek an optimization process which can balance this trade-off. This optimization seeks on the one hand to minimize bias, thereby increasing curve fluctuation to accurately track the empirical values of the probability of word occurrences. At the same time, the optimization minimizes variance, thereby decreasing the amount of curve fluctuation to obtain a smooth curve. The optimally smoothed curve for balancing the trade-off between these demands is a quantifiable parameter value that can be estimated using a “rule-of-thumb” smoothing parameter estimate (Fan et al., 1995). In section 5, we address the problem of temporally ordering a set of documents in the absence of a set of dated training data. To this end, we compute the average value of the optimal smoothing parameters for estimating the probability of occurrence of each word in the documents. Here, we find a close estimate of the correct temporal order of well-spaced subsets of ($m=10$) documents by searching among all possible temporal orderings to identify the highest average optimal smoothing parameter. We carried out this search using combinatorial optimization via the Simulated Annealing algorithm. In section 6, we evaluate the TempSeq method and present its results for the two distinct corpora — the DEEDS corpus and the SOTU corpus. In addition, we identify the informative words that enabled TempSeq to establish the correct temporal order for the selected subset of documents. In section 7, we provide error analysis, and present our general conclusions in section 8. Theoretical background and operational equations are presented in Annex A and Annex B.

Our fundamental assumption is that word usage changes gradually. We model the probability of word usage as a function of time using the local polynomial kernel regression for the generalized linear model, Fan et al. (1995). For further details of this model, see Annex A, section A.1.

Suppose $(D_{i},t_{D_{i}}),i=1,\ldots,n$ represents a sequence of data pairs, where $t_{D_{i}}$ represents the date of the ${D_{i}}^{{th}}$ document and $n$ denotes the size of the data set, that is to say, the total number of documents in the collection. Let $n_{w}(D_{i})$ denote the number of occurrences of the word (or term) $w$ in document $D_{i}$. Finally, let $N(D_{i})$ denote the total number of words (or terms) in document $D_{i}$. We are interested in estimating the probability of occurrence of the term w at time $t$, which is given by:

We define the weight term $K_{h}$ to be $K_{h}(u)\equiv h^{-1}K(u/h)$, where $K$ is called a kernel function. Typically, $K$ is a bell-shaped, non-negative function, with an area under the graph equalling unity. The function decays fast enough to eliminate the contributions of remote data points. (See equation 9, in Annex A, section A.1). For the present study, we used the t-distribution function with a low degree of freedom value (equal to $5$) to adequately weigh distant points.

As a weight term, $K_{h}$ fits a polynomial regression curve around the data in the neighbourhood of $t$, where $h$, called the bandwidth parameter, is the size of the local neighbourhood, such that data points distal from the neighbourhood are down-weighed (for this reason, $h$ is also referred to as the curve smoothing parameter). In simple terms, if $h$ is very large (highly smoothed), then $\hat{\pi}_{w,h}(t)\approx\sum_{i=1}^{n}n_{w}(D_{i})/\sum_{i=1}^{n}N(D_{i})$, thus representing an overall proportional outcome of word $w$ which does not change with $t$. On the other extreme, if $h$ is very small, then, $\hat{\pi}_{w,h}(t)$ evaluated at, say, $t_{D_{j}}$, has the value $\hat{\pi}_{w,h}(t_{D_{j}})\approx n_{w}(D_{j})/N(D_{j})$, which is the proportional outcome of word $w$ in the document written at time $t_{D_{j}}$. In this case, information on the frequency of occurrence of word $w$ in documents written at dates near to $t_{D_{j}}$ has been completely ignored in determining the value of $\hat{\pi}_{w,h}(t_{D_{j}})$. When $h$ is very small, the curve $\hat{\pi}_{w,h}(t)$ overfits, thus fluctuating rapidly to attain the values $n_{w}(D_{i})/N(D_{i})$ for each time point $t_{D_{i}},i=1,\ldots,n$. Although it is possible to draw a curve that perfectly describes the empirical probability of occurrence of a word across a date range (i.e., a bias with a value of zero), the consequent high variance of the curve means that, when applied to a test data set, the curve would overfit thus depicting a very inaccurate description of the probability of occurrence of the word of interest. In the field of kernel regression, there has been extensive research on how to select the appropriate bandwidth parameter $h$. However, in implementing the TempSeq method, we have relied on using a rule-of-thumb selection approach (Fan et al. (1995)), which we describe in Annex B, section B.1. The theoretical details for the derivation of equation 1, which falls under the locally constant case ($p=0$), can be found in Annex A, section A.1.

Relying on our assumption that change in word usage is gradual, we now focus on the role played by the bandwidth parameter $h$ in setting the bias-variance trade-off of the estimator $\hat{\pi}_{w,h}(t)$. Figure 1 illustrates the probability of occurrence of the words Drug(s) in the SOTU corpus at different dates and for variable bandwidth settings. The $x$-axis illustrates the calendar year (the time t), ranging from 1790 to 2020. The $y$-axis illustrates the values of $\hat{\pi}_{w,h}(t)$. The asterisks show the proportion of occurrences of the words “Drug(s)” in the years for which dated SOTU documents are available. The proportion is greater than zero in a few of the years, but is zero for a few years preceding or following that time. In that circumstance, when the bandwidth value h is small, the resulting estimate, $\hat{\pi}_{w,h}(t)$, is highly variable (the closer h approaches zero, the closer its values match the recorded proportion of occurrence of the term in the training data). This behaviour is illustrated by the dashed-line curve of $\hat{\pi}_{w,h}(t)$ in the figure. When the value of $h$ is larger (smoother), then the resultant probability curve $\hat{\pi}_{w,h}(t)$, overlaid in solid, has less variability. Conversely, when the value of $h$ is very large, the closer the values of $\hat{\pi}_{w,h}(t)$, for all $t$, approach the proportion of the real occurrence of the term $w$ across all the dates in the document set (illustrated in dotted lines). The bandwidth, which controls both the bias and the variance, is therefore a crucial parameter of the estimator $\hat{\pi}_{w,h}(t)$.

The optimal amount of smoothing of the data in figure 1 results in the solid curve, which illuminates a clear pattern in the data. The first peak in the figure (coinciding with the presidency of Richard M. Nixon (1970-1974)) refers to the emergence of the so-called War on Drugs, a US-led global policy aimed at the production, distribution, and use of psychoactive drugs, which was presented in Nixon’s State of the Union address of 1972. That address contains the phrases ‘… strong new drug treatment programs …’, ‘… by continuing our intensified war on drug abuse …’, ‘… Special Action Office for Drug Abuse Prevention …’, ‘… collective effort by nations throughout the world to eliminate drugs at their source …’, ‘… to drive drug traffickers and pushers off the streets of America …’, ‘… to curb illicit drug traffic at our borders and within our country …’. His 1974 address contains the phrases ‘… the spiraling rise in drug addiction … ’, ‘… The Psychotropic Convention … treaty regulating manufactured drugs worldwide …’ and ‘… the drug battle is far from over …’. The first peak extends to President Gerald Ford’s 1976 address, which contains phrases such as, ‘The sale of hard drugs is tragically on the increase again …’ and ‘… shipment of hard drugs …’. The second peak in figure 1 occurs decades later, around the year 2000. Then President William Clinton’s 1998 address contains phrases such as, ‘… to crack down on gangs and guns and drugs …’ and ‘… the largest anti-drug budget in history …’; his 1999 address has phrases such as ‘… if you stay on drugs, you have to stay behind bars …’ and ‘… to strengthen the Safe and Drug-Free School Act …’. In his 2000 address, Clinton announced ‘… new legislation to go after what these drug barons value the most, their money.’. Clinton also invokes the words drug(s), in the positive sense of insurance coverage for affordable prescription drugs. In that context, he used phrases such as ‘… seniors now lack dependable drug coverage …’ and ‘Lifesaving drugs are an indispensable part of modern medicine …’. In the following years, under President George W. Bush, policy regarding affordable drug coverage becomes a major issue domestically and globally; ‘… some form of prescription drug coverage …’ (in the 2001 address), ‘… new drugs that are transforming health care in America …’ (in the 2003 address) and ‘More than 4 million require immediate drug treatment’ (in the 2003 address regarding the lack of antiretroviral drugs in Africa). Thus, the first peak in figure 1 is exclusively related to illicit drug issues, whereas the second peak, some 25 years later, is primarily related to the affordability of prescription drugs.

Figure 2 illustrates the probability of occurrence of the words Anglicis and Anglis in the DEEDS corpus. These words are often found within the form of address ‘Franc[igen]is quam(et) Angl[ic]is’ (French and English), such as ‘… tam presentibus quam futuris tam Francigen[is] quam Anglicis salutem Sciatis me intuitu dei assensu …’, (… both present and future, both French and English, greeting. Know that with God’s consent I have [granted] …) and ‘… omnibus ministris et fidelibus suis Francis et Anglis de Oxenfordscira …’, (… to all his French and English ministers and servants of Oxfordshire …). The above form of address was commonly used by French and the English barons of the time to address their subjects. However, after the province of Normandy was conquered by the French in 1204, this form of address virtually fell out of use.

Figure 3 illustrates the probability of occurrence of a common stop word ‘de’ (of) in the DEEDS corpus. The asterisks in the figure show the proportion of occurrences of the word across time (1120 to 1300), and the line curve is found by smoothing the proportion of occurrences of the word across those same years. When comparing the smoothed black curves in figure 1 to figure 3, we see that the latter curve is more uniform across time (except for the years prior to 1125, when fewer documents were available). This behaviour, which we call the principle of temporal uniformity (non-uniformity) of uninformative (informative) words, is shown in the figure where the probability of occurrence of de has no defining temporal feature, and is uniform across the date range. For the purpose of temporally ordering a set of documents, no matter what combination of temporal ordering is evaluated in the TempSeq process, the contribution of uninformative words is immaterial.

In all of these figures 1 to 3, we see that the solid black curves have relatively the most optimal smoothing, as opposed, for example, to the highly variable dashed-line curve in figure 1. The optimal smoothing for a given curve represents the trade-off between small bias and small variance for the curve estimator. If the data analyst then randomizes the true temporal ordering of word usage, applying the optimal smoothing parameter will now result in a curve estimate that rapidly oscillates (high variance) due to seeking a minimum bias.

For a task of ordering temporally a set of $m$ number of documents ${\{D_{1},D_{2},\ldots,D_{m}\}}$, as in section 5, let the true temporal rank order of the documents be ${l=(1,2,\ldots,m)}$. Let $\sigma(l)$ represent a permutation of the ranks and let $\sigma_{0}$ represent the permutation identity ($\sigma_{0}(l)=l$). We measure the extent to which two permutations are in close proximity to one another using the Spearman’s ($\rho$) rank correlation. If $\hat{\sigma}(l)$ is the predicted temporal rank order for the set of documents ${\{D_{1},D_{2},\ldots,D_{m}\}}$, then the closer ${|\rho(\hat{\sigma}(l),\sigma_{0}(l))|}$ is to unity, the more accurately the predicted order matches the true order (we only consider the absolute value of the correlation since forward and reverse orders are equated, as noted above).

We randomly selected sets of 10 documents, dated approximately 24 years apart for the SOTU, and 18 years apart for the DEEDS-conflate corpora. For the random selection, we used systematic sampling, as follows: First, we randomly selected the start date document, and then every 24^th year in succession for the annual SOTU, and then for ever 18^th year for the annual conflation of DEEDS documents (DEEDS-conflated), or the corresponding randomly selected single DEEDS documents (DEEDS-single). If the subsequently selected year exceeded the range of dates, we cycled through from the start date. Then, we labelled the temporal rank of the resulting documents from 1 to 10. Starting from an initial random temporal order of the ten documents, we estimated their true temporal rank order as described above, and calculated the Spearman correlation for the true and shuffled ordering. In all cases, our analysis was based on words that occurred at least once in two separate documents (otherwise, no information regarding temporal ordering can be inferred). For 100 replications of the above procedure, the median of the absolute value of the correlations between the estimated and the true rank orders was 0.66 for the SOTU corpus and 0.78 for the annually conflated DEEDS corpus. As a baseline comparison, we note that the median of the absolute value of the correlations between the true temporal rank orders and their 100 random permutations was 0.24. In the computation, we only considered words that occurred more than once in the set of 10 documents. As shown in the box plots of figure 6, the TempSeq method performed significantly better than the baseline correlation. The Wilcox rank sum test and the t-test showed a statistically significant difference between the baseline correlation and the correlations associated with each of the SOTU and the DEEDS-conflated corpora. In both cases, $\mbox{p-value}<2.2\times 10^{-16}$.

Regarding the DEEDS-single corpus, the TempSeq did not perform as well as for the temporally DEEDS-conflated collection. Although statistically significantly better than the baseline correlations (for tests based on Wilcox rank sum and the t-test, $\mbox{p-value}<5.5\times 10^{-7}$), the median correlation coefficient between the estimated and the true rank orders was only 0.45 (see figure 7). This raises the question of why TempSeq under-performed on the DEEDS-single collection as compared to the DEEDS-conflated collection. These two collections differ in that the latter was created by merging all DEEDS documents for a given year, such that the DEEDS-conflated documents had a mean of 11,000 words as compared to only 175 words for the DEEDS-single collection. As such, the temporal sequencing for the DEEDS-single collection according to equation 4 is based on a very small sample of words. Given the requirement that each word should occur at least once in two separate documents within a set, there are correspondingly fewer words informing the estimation of temporal order.

As stated in section 1, the TempSeq approach for document sequencing allows the identification of words that are most informative for determining the correct temporal order. The process that we used to identify and analyze the informative words is as follows: As an example, we first considered a set of 10 documents from the DEEDS-conflated collection for which the TempSeq prediction of ordering, $\hat{\sigma}(l)$, resulted in a Spearman correlation coefficient of 0.92. Although the total number of unique words across these 10 documents was 15,757, when we considered the number of words that occurred at least once in two documents, the number of the relevant unique words declined to 5,988. Of these 5,988 words, we selected those with frequency values in the top 50^th percentile. For each of these selected words, we computed the rule-of-thumb optimal bandwidth values under the predicted TempSeq order, $h_{amise,w,\hat{\sigma}(l)}$. On the basis of these optimal bandwidths, we extracted words that were in the top 88^th percentile for their maximum probability of occurrence score over the temporal rank domain in equation 1. The first condition ensured that the words under consideration were those that occurred with sufficient frequency (exceeding the median). The second condition ensured that when a set of documents was listed in its correct temporal order, the informative words (i.e., the words that were most useful for attaining the correct temporal ordering using TempSeq method) were those whose occurrence clustered in near-by time periods. In turn, these clusterings resulted in higher word probability occurrence around those same time periods (refer to equation 1) as compared to the other time periods. Using the above word filtering procedure, we obtained a final list of around 360 relevant words.

To interpret meaningfully the 360 relevant words in the list, we compared them with the representative topic terms from an LDA (Latent Dirichlet Allocation,  Blei et al. (2003)) topic modelling run on eight topics on the entire DEEDS-single documents (Gervers and Tilahun, 2023). In the pre-processing stage (prior to running the LDA algorithm), each document was split into tri-gram words (sequences of three consecutive words). The topic proportions for each of the documents were aggregated in accordance with their date of issue (see Figure 6.1 in  Gervers and Tilahun (2023)). When we examined the dominant topics corresponding to the dates from the set of the 10 documents, we found that the vast majority of the informative words extracted using the procedure outlined above matched the words in the tri-gram topic terms. Some examples of the informative words and their contexts include finalis facta concordia (indicating that the document is a ’final concordance made’), anno regni regis (’in the year of the king’s reign’), scripto sigillum meum (’marking the document (with) my seal’, indicating the sealing of a grant), pro omni seruitio (’for all service’, indicating that a transfer was not a simple donation), and perpetuam elemosinam (’in perpetual alms’, indicating that the transfer was a donation).

For the SOTU corpus, we similarly selected a set of 10 documents, each one separated by 23 years during the interval from 1810 to 2019. The Spearman correlation coefficient obtained between the TempSeq ordering method and the true orders for this set of documents was 0.95. The informative words were extracted and examined in the same way as described for the DEEDS-conflated corpus. Of a total of 29,537 words, there were 3,957 words that occurred at least once in at least two documents, of which the top 20^th percentile of the maximum probability of occurrence score over the temporal rank domain in equation 1 yielded 435 words. For the purposes of illustration, we present three words from the short list of relevant words: Britain, Families, and Court. A bar graph of the frequencies of these words, counted from the selected set of 10 presidential speeches, is illustrated in figure 8 .

We then ran an LDA topic modelling with five topics, where in the pre-processing stages, documents were split into bi-grams (sequences of two consecutive words), over the entire SOTU corpus. To enable an interpretation of the words that drove the temporal ordering, we examined the high ranking (top 20) bi-gram words of a topic associated with fiscal and commercial interests of the US. Within this topic, Great Britain was one of the high ranking words. One of the uni-gram forms of the term, Britain, turned up in the list of the top relevant words. From the 1810 SOTU address by James Madison, the term Britain was invoked in the context of the naval blockade suffered by the US when the Napoleonic Wars had spilled into the Atlantic. Twenty-three years later, Britain was discussed in the context of final settlement on the US North-East boundary and navigational safety concerns; another twenty-three years later, in 1856, Franklin Pierce’s address discussed Britain in the contexts of her desire to dominate the Panama routes (and US refusal thereof), rights to fisheries, increasing trade between the US and British Provinces in North America, and maritime rights regarding immunity from seizure: ‘… the private property of subjects and citizens of a belligerent on the high seas … by the public armed vessels of the other belligerent, except it be contraband.’ Britain was a subject in Ruthford Hayes’ 1879 address regarding the settlement of a dispute over rights to fisheries in Canadian waters. In two of the subsequent presidential addresses from the 10 documents, Britain was mentioned once in each, and not mentioned thereafter. Thus, the correct temporal order (and the TempSeq ordering method) of the 10 selected documents optimized a gradual pattern of change in the usage of the word Britain.

Another informative word to TempSeq analysis is Families, although it was not ranked highly from LDA, either as a uni-gram or as a portion of a word in a bi-gram. In the set of 10 SOTU documents, the word Families was barely mentioned prior to the address by Harry Truman in 1948. The frequent usage of that word appears in the later time periods. For example, in the 1948 Address, Harry Truman mentioned Families in the contexts of a social safety net and policies aiming to raise the standard of living for ordinary Americans. For example, we note ‘public housing for low-income families’, the provision of price support for farm commodities to enable ‘farm families … to catch up with the standards of living enjoyed in the cities’ and anti-inflation measures to fight the ‘undermining [of] the living standards of millions of families’. In Lyndon B. Johnson’s 1968 address, the word Families was invoked to boast about the increase in the wealth accumulation of ‘most American families’ as ‘more and more families own their own homes … television sets’. He urged congress to authorize more money to allow ‘new housing units for low and middle-income families’ to be built in order for ‘… thousands of families to become homeowners, not rent-payers’. In William Clinton’s 1996 Address, the word Families is invoked in the context of sheltering ‘working families’ from the effects of government cuts. In spite of cuts and a shrinking government, Clinton states his belief in the possibilities of cultivating ‘stronger families’. Further, he speaks of the challenge to ‘strengthen America’s families’ and thanks his wife for having taught him ‘the importance of families and children’. Clinton also challenges ‘America’s families to work harder to stay together’ because ‘families who stay together not only do better economically, their children do better as well’. Furthermore, the word Families is invoked in the context of health insurance policies –‘… over one million Americans in working families have lost their health insurance’. The 2019 Address by Donald Trump invokes the word Families in his statement ‘We passed a massive tax cut for working families’. The word Families is also mentioned in the context of victims of criminal violence whom Trump had met – ‘I have gotten to know many wonderful angel moms, dads and families’.

Among the 10 selected presidential speeches, Court was found to be an informative word, despite not being ranked highly by LDA. Examining the bar graph in figure 8, there is an abundant usage of the word (34 times) in the 1925 address by Calvin Coolidge. Although occurring a few times in the other prior presidential speeches (except for that of James Madison), the word Court did not occur after this date. Emerging as a great power after the end of World War I, the United States sought a foreign policy with global influence. Calvin Coolidge invoked the word Court primarily in discussing his administration’s support in joining the Permanent Court of International Justice, which had been set-up in 1922. In his speech, Coolidge encouraged the senate to support adherence to the Court by arguing that the United States’ interests would not be negatively affected, for example in stating ‘… by supporting the court we do not assume any obligations under the league …’; ‘… the statute creating the court shall not be amended without consent …’; ‘No provision of the statute … give[s] [the] court any authority to be a political rather than a judicial court’, and ‘If we support the court, we can never be obliged to submit any case which involves our interests for its decision’.

The word Court was also invoked in the 1879 address by Rutherford B. Hayes, although to a lesser extent than compared to that of Calvin Coolidge. Here, the primary contexts of usage involved criminal offenses and court administration. In the context of criminal offences, there is the example of the practice of polygamy in Utah, which would no longer be defended under the constitutional guarantee of religious freedom, ‘The Supreme Court of the United States has decided the law to be within the legislative power of Congress’. Another instance concerns the urgency to introduce a justice system to prosecute criminals in the newly acquired territory of Alaska: ‘bill authorizing … detention of persons charged with criminal offenses, and providing for an appeal to United States courts …’. In the context of court administration, we find the phrases, ‘The business of the Supreme Court is at present largely in arrears’; ‘… magistrates who compose the court can accomplish more than is now done’, and ‘… additional circuit judges and the creation of an intermediate court of errors and appeals, which shall relieve the Supreme Court of a part of its jurisdiction … ’.

We conducted an error analysis on the subset of the 100 replication sets of 10 randomly selected documents for which TempSeq under-performed, with a cut-off of correlation coefficients falling below the 10^th percentile. For the SOTU corpus, these were the sets of documents for which correlation coefficient between estimated temporal ordering via the TempSeq method and their true temporal ordering was less than 0.27. For the richer DEEDS-conflated corpus, the corresponding threshold correlation was 0.62. When comparing the average bandwidth values, equation (4), of the estimated temporal orderings for such sets of documents to that of the average bandwidth value under their correct temporal orderings, i.e., $H_{\sigma_{0}(l)}$, the values of $H_{\sigma_{0}(l)}$ were generally larger for the latter case, as shown in figures 9 and 10. This reflects the lesser variability of word usage, and the gradual change in word frequency with time. The under-performance of TempSeq on the sets of documents under discussion is therefore explicable by the inadequate search runs of the Simulating Annealing algorithm that searches for the optimal temporal ordering in equation (4).

A natural question that arises at this point is whether employing the models used in Large Language Models (LLM), which have shown unprecedented capabilities in understanding and generating natural language, can be used to solve the temporal sequential ordering problem described above. The fundamental problem with the possible application of LLM for the research question posed in this paper is that the amount of textual data required to train an LLM is typically massive (for example, the first BERT model was trained on over 3.3B words, Devlin (2018)). By comparison, the SOTU corpus has only a total of around 1.5M words, and the DEEDS corpus has only a total of around 2M words, with the dates of documents in each corpus spread across two and half centuries. Even to employ static word embeddings, which are ways to represent words as vectors in a multidimensional vector space, the smallest training sets required for reliable word representation using Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) were 24M words (a subset of Google News corpus) and 1B tokens (2010 Wikipedia), respectively. If pre-trained LLM models were to be leveraged for the task at hand, those models would need to have been trained on the right kinds of corpora, in the sense of topics, genre, time periods and language. Moreover, assuming we can obtain effective word representations using LLM, it is not clear how we could use them for the task of temporal ordering of a set of documents. More pertinently, the research question posed in this paper seeks to temporally order a set of documents (for example, sets of ten documents following the TempSeq experiments in sections 5 and 6) when a training corpus is not necessarily available, as is the case for many heritage texts. Far below the size of data required for LLM, a set of ten documents from SOTU has a total average count of 64K words (an average of 6400 words per Presidential Address), and a set of ten documents from DEEDS has a total average count of 1750 words (an average of 175 words per document).

Motivated by problems arising in the dating of historical and heritage texts, we set about in this paper to develop a method for assigning temporal rank orders to a sequence of documents when the date of issue is either missing or uncertain. In the historical context, the limited number and length of surviving manuscript texts presents a particular challenge in ordering. Our unsupervised method for document rank ordering relies on the principle that word usage changes gradually, on the scale of decades. Our method effectively captures changing word usage in the DEEDS and SOTU corpora by the bandwidth estimates. As shown (in section 6), the median of the correlation values for both corpora were significantly higher than the baseline from random ordering of sets of 10 documents. However, when the sizes of each of the documents are composed of fewer words, as in the case of the DEEDS-single, the TempSeq method doesn’t perform as well due to the lack of adequate number of words on which to base the necessary estimates.

In practice, a reliable document rank ordering method should furnish the opportunity to identify particularly informative words for estimating the correct document orderings (for example, words with relatively high values of $h_{amise,w,\sigma(l)}$ for the optimal ordering). Equally, such a method provides us with the potential to identify anachronistic words, which may have been inserted into documents for nefarious reasons, such as the case of forged documents mentioned in section 1.

In our current experimental design, we selected orderings of documents that were approximately twenty years apart and extending over 200 years; our procedures for ordering performed significantly better than the randomization. In the future, we shall examine the performance of our method when selected documents are separated at variable time intervals. We also plan to examine the temporal rank orderings of documents when the number of words in the documents is extremely limited, namely the particular case of Anglo-Saxon texts.

Acknowledgments: The authors gratefully acknowledge Prof. Paul Cumming of Bern University for his critical reading of this manuscript.