ViS-À-ViS: Detecting Similar Patterns in Annotated Literary Text

The ViS-À-ViS system was developed in an agile fashion, emphasizing continuous involvement of the “customer” (the literary scholar) in the development process. The development took place in cycles of two weeks, where at the end of each cycle the scholar examined the system, used it on real data, and provided feedback. The feedback helped to refine existing visualizations and to define new ones. For example, during the first iterations, the scholar has found that while the Gantt chart (Figure 1(a)) is useful for showing the density of tags across the text, it does not illustrate the quantitative nature of the data. As a result, the scholar asked the team to insert a new visualization (the Stacked Area Chart in Figure 1(b)).

In the rest of this section we explain the architecture of the tool and list key features that help in analyzing similarities between annotated (tagged) texts, in terms of detecting similar patterns.

One of the areas we are researching is automatic detection of similarities. While much work has been done on text reuse and other fully computerized methods of evaluating textual similarity, including in Hebrew [10], there has been less work done applying computerized methods to hand-tagged data. The hope is that computerized methods can efficiently leverage the time and effort put into creating the data, and that it can add value beyond what can be done with the plain text.

We apply a novel combination of existing methods to this task. Our initial phase is to vectorize each text for each tag we are using to evaluate the similarity. We convert the tag data to a binary vector, where each element in the vector represents whether the given character in that text has the given tag. Since we do not expect to see exact matches, and we expect to see shifting and warping, as well as comparing across texts of very disparate widths, we need a fuzzier metric. We apply dynamic time warping (DTW) [8] to the task, an algorithm originally applied to speech signal analysis, however it has since seen a wide range of applications [4].

The time parameterization is equally applicable to any similar feature vector. However, when we need to calculate similarities for a substantial number of texts, we then need to calculate similarities for each pair, and the additional quadratic calculation of DTW itself becomes prohibitively slow. We therefore use FastDTW[9] for the underlying alignment, as it provides similar accuracy in linear time. Finally, having aligned our sequences we then score our alignments. We use TAM [2] for a baseline score, as we find it provides substantially better results than simpler methods like Euclidean distance.

Having calculated the TAM for the given tag between a pair of texts, we then change it to a 0 to 1 scale, and apply one last modification of our own, namely, we multiply by a weight factor. We calculate our weight factor as $min(\frac{10*H(v1)}{|v1|},1)*min(\frac{10*H(v2)}{|v2|},1)$, applying a linear slope penalty for sparseness of the vectors (where H denotes the Hamming weight). We do this since we find that the tags tend to be fairly sparse, and while from a mathematical point of view treating two empty vectors as “perfectly similar” is more conventional than our treatment as perfectly dissimilar, given the underlying sparseness this would cause a large amount of useless results (the penalty can be easily neutralized for non-sparse corpora). This also allows a substantial speedup by being able to avoid some calculations.

Having completed calculating pairwise for a group of texts we can then derive a heatmap (Figure 3). We use this visualization to show the relationship between pairs, as well as allowing a view of groupings, emerging from blocks of similar texts.

In order to examine ViS-À-ViS as a tool for the literary scholar we conducted a short experiment. One section of the corpora under discussion was analyzed manually fifteen years ago by Münz-Manor, as part of his doctoral dissertation [5]. The research focused on the nature of figurative language in the corpus, namely the use of metaphors and similes. The main conclusion of the dissertation was that the corpus under discussion reveals poetic “disinterest” in using figurative language, a most uncommon practice in the universe of Hebrew verse or verse in general [7].

We wanted to come back to the results of the dissertation, this time using our tool, in order to corroborate or refute the original findings. In order to do so we transferred the original manual taggings from the dissertation into CATMA and then exported them to ViS-À-ViS. By that we were able to use data visualization in order to have a fresh look at the corpus. Some of the preliminary results were revealing; for example, the Sunburst chart (Figure 4) helped us refine the statistical distribution of the various figurative devices. On the one hand we were able to corroborate the finding that the simile is rather marginal in the corpus [6], a point that was already noted in the doctoral dissertation, and on the other that the synecdoche is much more common than was initially thought (marked with an arrow in the figure). In another case we were able to detect a specific poetic genre (the Hosanna poems) in which figurative language was much more pronounced than expected.

The ViS-À-ViS tool also helped to detect patterns that went unnoticed in traditional literary studies of the corpora, and to single out an important poetical phenomenon. It turns out that in one specific genre, the Avodah poems for the Day of Atonement [11], there is a sudden drop in the usage of metaphors and other figurative devices at a certain point in the poem followed by a small resurgence of figurative language towards the end of the poem (as illustrated in Figure 1(a) and 1(b) for a specific poem in the genre). Once we detected the fact that this pattern is consistent only in this group of poems we went back to the texts themselves and it was revealed that a change in the narrative governs this drop. Interestingly, deviation from this pattern enabled us to single out a poem of the Avodah genre which is much later from the rest of the poems (Figure 5). That is to say, a change in the pattern detected signified a poetic change in the text.

We took then another step and annotated, again by traditional human hermeneutical tagging, the Book of Psalms. Once this corpus of ancient Hebrew poetry was imported to our tool we were able to compare the findings concerning figurative language in both corpora. Figurative language is very common in the biblical book and its juxtaposition with the later corpus revealed as well meaningful differences that we would like to examine in the near future.

In order to evaluate the quality of the similarity algorithm we have added a user rating mechanism to the ViS-À-ViS tool. The mechanism is implemented using our board feature, that is described in Section 1. We created a list of test groups based on our automatic scores. Each group consisted of a target document, and five other candidate documents that were created with the three most similar documents to the target, one moderately similar (chosen from 4th-10th place), and one drawn at random. The literary scholar (Ophir Münz-Manor) was then presented with each group and was asked to rank the five documents in order of similarity to the target document. In order to avoid priming, the five documents were presented randomly.

The results of our initial evaluation were promising. While the algorithm was unable to precisely rank them, it correctly ascertained the groupings in most cases. Most promisingly, the least similar document was correctly identified roughly 70% of the time (vs. 20% by random chance). Given the specifics of the method where the least similar by the metrics was substantively different, being randomly chosen and not chosen for being of interest, it suggests that it was indeed capable of differentiating between similar and dissimilar texts. We also note that the scholar himself found the nuances of the choices to be difficult and unclear at times. In short, we believe that the system as it currently exists is already capable of suggesting preliminary matches and groupings of interest.

All in all, the innovative nature of the group of scholars and the project relates to the combination of “traditional” literary scholarship with state-of-the-art visualization and algorithmic tools and procedures. Moreover, the project facilitates discussions concerning the opportunities and limitations of literary data in a big-data setting, the meaning of the so-called “distant reading” in this context, the interaction between large-scale visualization and particular textual occurrences, and, finally, the similarities and differences between human and computerized “reading”.

Another unique feature of the project lies in the fact that we visualize and detect patterns not in the texts themselves but rather on the level of tagging. Since our tags are based at the moment on manual annotation of the literary texts we have access to the “hermeneutical stratum”, namely to a “second-order” data of human provenance. In other words, patterns exist not only in the text but also in the attached interpretation thereof.

For us, both the primary text and the tagged text are data and as such both are interesting. While the former can be called first-order data the latter might be called second-order data. That said, from a humanistic perspective the text and its interpretation(s) are both sides of one coin and we have interest in both of them. For that reason in the next stage we plan to add visualization capabilities of both the text and the tags, hence improving the pattern recognition abilities of the tool. One of the advantages of ViS-À-ViS will be its ability to visualize and analyze the correspondence between the two strata. Another feature would enable the juxtaposition of several human interpretations on the same text hence enabling also a comparative study of human hermeneutical multiplicity.

We have currently tested ViS-À-ViS on literary text with figure of speech tags. As a future work, we plan to use the system with a greater variety of literary tags, as well as to explore its applicability for more domains.