Traditional Readability Formulas Compared for English

We use the CCB corpus to calibrate formulas. The article excerpts included in CCB are divided into the categories of story, poetry, informational text, and drama. For the simplification of our approach, we limit our research to story-type texts. This left us with only 69 items to train with. But those are directly from the U.S. Common Core Standards. Hence, we assume with confidence that the item classification is generally agreeable in the U.S.

CCB is the only dataset that we use in the calibration of our formulas. All below datasets are mainly for feature selection purposes only.

WBT, the largest native dataset available in RA, contains articles targeted at readers of different age groups from the Weekly Reader magazine and the BBC-Bitesize website. In table 1, we translate those age groups into U.S. schools’ K-* format. We downsample to $625\frac{\text{item}}{\text{class}}$ as per common practice.

CAM (Xia et al., 2016) classifies 300 items in the Common European Framework of Reference (CEFR) (Verhelst et al., 2001). The passages are from the past reading tasks in the five main suites Cambridge English Exams (KET, PET, FCE, CAE, CPE), targeted at learners at A2–C2 levels of CEFR.

CKC (Lee and Lee, 2020b, a) is less-explored. It developed upon the reading passages appearing in the Korean English education curriculum. These passages’ classifications are from official sources from the Korean Ministry. CKC represents a non-native country’s official ESL education curriculum.

OSE is a recently developed dataset in RA. It aims at ESL (English as Second Language) learners and consists of three paraphrased versions of an article from The Guardian Newspaper. Along with the original OSE dataset, we created a paired version (OSE-Pair). This variation has 189 items and each item has advanced-intermediate-elementary pairs.

In addition, OSE-Sent is a sentence-paired version of OSE. The dataset consists of three parts: adv-ele (1674 pairs), adv-int (2166), int-ele (2154).

NSL (Xu et al., 2015) is a dataset particularly developed for text simplification studies. The dataset consists of 1,130 articles, with each item re-written 4 times for children at different grade levels. We create a paired version (NSL-Pair) (2125 pairs).

ASSET (Alva-Manchego et al., 2020) is a paired sentence dataset. The dataset consists of 360 sentences, with each item simplified 10 times.

We start by recalibrating five readability formulas. We considered Zhou et al. (2017) and the number of Google Scholar citations to sort out the most popular traditional readability formulas. Further, to make a fair performance comparison with our adjusted variations, we choose the formulas originally intended to output U.S. school grades but are based on 20th-century texts and test subjects.

Flesh-Kincaid Grade Level (FKGL) is primarily developed for U.S. Navy personnel. The readability level of 18 passages from Navy technical training manuals was calculated. The criterion was that $50\%$ of subjects with reading abilities at the specific level had to score $\geq 35\%$ on a cloze test for a text item to be classified as the specific reading level. Responses from 531 Navy personnel were used.

where sent is sentence, and # refers to "count of."

The genius of Gunning Fog Index (FOGI) is the idea that word difficulty highly correlates with the number of syllables. Such a conclusion was deduced upon the inspection of Dale’s list of easy words (Zhou et al., 2017; Dale and Chall, 1948). However, the shortcoming of FOGI is the over-generalization that "all" words with more than two syllables are difficult. Indeed, "banana" is quite an easy word.

Simple Measure of Gobbledygook (SMOG) Index, known for its simplicity, resembles FOGI in that both use the number of syllables to classify a word’s difficulty. But SMOG sets its criterion a little high to more than three syllables per word. Additionally, SMOG incorporates a square root approach instead of a linear regression model.

Coleman-Liau Index (COLE) is more of a lesser-used variation among the five. But we could still find multiple studies outside computational linguistics that still partly depend on COLE (Kue et al., 2021; Szmuda et al., 2020; Joseph et al., 2020; Powell et al., 2020). The novelty of COLE is that it calculates readability without counting syllables, which was viewed as a time-consuming approach.

Automated Readability Index (AUTO) is developed for U.S. Air Force to handle more technical documents than textbooks. Like COLE, AUTO relies on the number of letters per word, instead of the more commonly-used syllables per word. Another quirk is that non-integer scores are all rounded up.

Considering the value of traditional readability formulas as essentially the generalized definition of readability for the non-experts (section 1), what really matters is the included features. The coefficients (or weights) can be recalibrated anytime to fit a specific use. Therefore, it is important to first identify handcrafted linguistic features that universally affect readability. Additionally, to ensure breadth and usability, we set the following guides:

We utilize LingFeat for feature extraction. It is a public software that supports 255 handcrafted linguistic features in the branches of advanced semantic, discourse, syntactic, lexico-semantic, and shallow traditional. They further classify into 14 subgroups. We study the linguistically-meaningful branches: discourse (entity density, entity grid), syntax (phrasal, tree structure, part-of-speech), and lexico-semantics (variation ratio, type token ratio, psycholinguistics, word familiarity).

After extracting the features from CCB, WBT, CAM, CKC, and OSE, we first create feature performance ranking by Pearson’s correlation. We used Sci-Kit Learn (Pedregosa et al., 2011). We take extra measures (Approach A & B) to model the features’ general performances across datasets. Each approach runs under differing premises:

After 78 hours of running, we decided not to extract features from NSL. Computing details are in appendix E. Among the features included in LingFeat, there are traditional readability formulas, like FKGL and COLE. These formulas performed generally well but a single killer feature, like type token ratio (TTR), often outperformed formulas. Traditional readability formulas and shallow traditional features are excluded from the rankings.

Under premise A, each dataset poses a different linguistic environment to feature performance. Further, premise A takes human error into consideration and agrees that data labeling is most likely inconsistent in some way. The literal correlation value itself is not too important under premise A.

Rather, we look for features that perform better than the others, under the same test settings. Thus, approach A’s rewarding system is rank-dependent. In a dataset, features that rank 1-10 are rewarded 10 points, rank 11-20 get 9 points, … and rank 91-100 get 1 point. Since there are five feature correlation rankings (one per dataset), the maximum score is 50. The results are in Table 3, in the order of score.

Under premise B, the weak correlation of a feature in a dataset is solely due to the feature’s weakness to generalize. This is because all datasets are supposedly perfect. Hence, we only measure the feature’s absolute correlation across datasets.

Approach B’s rewarding system is correlation-dependent. In a dataset, features that show correlation value between 0.9-10 are rewarded 10 points, value between 0.8-0.89 get 9 points, … and value between 0.0-0.09 get 1 point. Like approach A, the maximum score is 50. The result is in Table 4.

First and the most noticeable, the top features under premise A & B are similar. In fact, the two results are almost replications of each other except for minor changes in order. We initially set two premises to introduce differing views (and hence the results) to feature rankings. Then, we would choose the features that perform well in both.

But there seems to be an inseparable correlation between ranking-based (premise A) and correlation-based (premise B) approaches. CorrNoV_S (Corrected Noun Variation) was the only new top feature introduced under premise B.

Second, discourse-based features (mostly entity-related) performed poorly for use in our final NERF. As an exception, ra_NNToT_C (noun-noun transitions : total) scored 28 under premise A and 26 under premise B. On the other hand, a majority of lexico-semantic and syntactic features performed well throughout. This strongly suggests that a possible discovery of universally-effective features for readability is in lexico-semantics or syntax.

Third, the difficulty of a document heavily depended on the difficulty of individual words. In detail, as_AAKuL_C, as_AAKuW_C, to_AAKuL_C, to_AAKuW_C showed consistently high correlations across the five datasets. As shown in Section 2, these five datasets have different authors, target audience, average length, labeling techniques, and the number of classes. Each dataset had at least one of these features among the top 5 performances.

The four features come from age-of-acquisition research by Kuperman et al. (2012), which now prove to be an important resource for RA. Such direct classification of word difficulties always outperformed frequency-based approaches like SubtlexUS (Brysbaert and New, 2009). Back to feature selection, we follow the steps below.

The steps above produce the same results for both approach A and B. The final selected features are as_AAKuL_C (psycholinguistic), as_TreeH_C (tree structure), as_ContW_C (part-of-speech), as_NoPhr_C (phrasal), as_SbL1C_C (word familiarity), CorrTTR_S (type token ratio). CorrNov_S (variation) only appeared under approach B, and we did not include it.

The final NERF (section 4.5) is brought in three parts. The first is lexico-semantics, which measures lexical difficulty. It adds the total sum of each word’s age-of-acquisition (Kuperman’s) and the sum of word familiarity scores (Lg10CD in SubtlexUS). The sum is divided by # sentences.

The second is syntactic complexity, which deals with how each sentence is structured. We look at the number of content words, noun phrases, and the total sum of sentence tree height. Here, content words (CW) are words that possess semantic content and contribute to the meaning of the specific sentence. Following LingFeat, we consider a word to be a content word if it has "NOUN", "VERB", "NUM", "ADJ", "ADV" as a POS tag. Also, a sentence’s tree height (TH) is calculated from a constituency-parsed tree, which we used the CRF parser (Zhang et al., 2020) to obtain. The related algorithms from NLTK (Bird et al., 2009) were used in calculating tree height. The same CRF parser was also used to count the number of noun phrase (NP) occurrences.

The third is lexical richness, given through type token ratio (TTR). This is the only section of NERF that is averaged on the word count. TTR measures how many unique vocabularies appear with respect to the total word count. TTR is often used as a measure of lexical richness (Malvern and Richards, 2012) and ranked the best performance on two native datasets (CCB and CAM). Importantly, these two datasets represent US and UK school curriculums, and TTR seems a good evaluator. What was interesting is that out of the five TTR variations from Lee et al. (2021); Vajjala and Meurers (2012), corrected TTR generalized particularly well.

Like section 3, we use the non-linear least fitting method on CCB to calibrate NERF. The results match what we expected. For example, the coefficient for word familiarity, which measures how frequently the word is used in American English, is negative since common words often have faster lexical comprehension times (Brysbaert et al., 2011).

All readability formulas, whether recalibrated or not, show near-perfect performances in ranking the simplicity of texts. On both OSE-Pair & NSL-Pair, we designed a simple task of ranking the simplicity of an item. Both paired datasets include multiple simplified versions of an original item. Each row consists of various simplifications. A correct prediction is the corresponding readability formula output matching simplification level (e.g. original: highest prediction, …, simplest: lowest prediction).

In OSE-Pair, a correct prediction must properly rank three simplified items. NERF showed a meaningfully improved performance than the other five traditional readability formulas before recalibration. NERF correctly classified 98.7% pairs, while the others stayed $\leq$95% (FKGL: 93.4%, FOGI: 92.6%, SMOG: 94.4%, COLE: 94.9%, AUTO: 92.6%). Recalibration generally helped the traditional readability formulas but NERF still showed better performance (FKGL: 97.8%, FOGI: 97.1%, SMOG: 94.4%, COLE: 89.9%, AUTO: 95.8%).

In NSL-Pair, a correct prediction must properly rank five simplified items, which is a more difficult task than the previous. Nonetheless, all six formulas achieved 100% accuracies. The same results were achieved before and after CCB-recalibration. This hints that NSL-Pair is thoroughly simplified.

Readability formulas seem to perform well in ranking several simplifications on a passage-level. But there certainly are limits. First, one must understand that calculating "how much simple" is a much difficult task (Table 5). Second, the good results could be because sufficient simplification was done. For more fine grained simplifications, readability formulas could not be enough.

We were surprised that some existing text simplification studies are directly using traditional readability formulas for sentence difficulty evaluation. Our results show that using a formula-based approach is particularly useless in evaluating a sentence.

We tested both CCB-recalibrated and original formulas on ASSET. Here, a correct prediction must properly rank eleven simplified items. Despite the task difficulty, we anticipated seeing some correct predictions as there were 360 pairs. SMOG guessed 37 (after recalibration) and 89 (before recalibration) correct out of 360. But all the other formulas failed to make any correct prediction.

OSE-Sent poses an easier task. Since the dataset is divided into adv-int, adv-ele, and int-ele, the readability formulas now had to guess which is more difficult, out of the given two. We do obtain some positive results, showing that readability formulas can be useful in the cases where only two sentences are compared. On ranking two sentences, NERF performs better by a large margin.

We argue that NERF is effective in fixing the over-inflated prediction of difficulty on medical texts. Such sudden inflation is widely-reported (Zheng and Yu, 2017) as the common weaknesses of traditional readability formulas on medical documents.

The U.S. National Institute of Health (NIH) guides that patient documents be $\leq$K-6 of difficulty. The most distinct characteristic of medical documents is the use of lengthy medical terms, like otolaryngology, urogynecology, and rheumatology. This makes traditional formulas, based on syllables, unreliable. But NERF uses familiarity and age-of-acquisition to penalty and reward word difficulty.

A medical term not found in Kuperman’s and SubtlexUS will have no effect. Instead, it will simply be labeled a content word. But in traditional formulas, the repetitive use of medical terms (which is likely the case) results in an insensible aggregation of text difficulty. In case various medical terms appear, NERF rewards each as a unique word.

Among recent studies is Haller et al. (2019), which analyzed the readability of urogynecology patient education documents in FKGL, SMOG, and Fry Readability. We also analyze the same 18 documents from the American Urogynecologic Society (AUGS) by manual OCR-based scraping. As Figure 1 shows, it is evident that NERF helps regulate the traditional readability formulas’ tendencies to over-inflate on medical texts. An example of the collected resource is given in appendix B.