Auto-Select Reading Passages in English Assessment Tests?

Not all reading materials apply the same. Depending on how the reading material is presented to language learners, generally good reading material might not be as appropriate in more specific contexts. In this paper, we investigate if automatic reading passage selection is a feasible task in test-based contexts.

Though not many, some studies (Kuo et al., 2014; Hsu et al., 2010) attempted to devise a reading material recommendation system to give generally helpful texts to students. Such systems attempt to give what is helpful to each user, based on one’s performance record. However, most English assessment exams (SAT, TOEFL, TOEIC) are standardized, presenting some unique challenges.

Choosing appropriate passages for reading comprehension sections of such exams is a lengthy process. SAT, a standardized test widely used for college admissions in the United States (Westrick et al., 2019), has lengthy technical specifications for test development, which include three steps: 1. item search/crafting, 2. content and fairness review, 3. pretesting (College-Board, 2017). Each step requires further technical training of already professionals, making test development troublesome.

The 1. test-specific technicalities merge with the 2. test developer’s expertise to select suitable reading passages. Intuitively speaking, both components are difficult to mimic even with the recent natural language processing (NLP) techniques. Hence, automatic reading passage selection for test papers has largely been left in the dark. But the topic is worthy enough of investigation.

Faster reading passage selection is definitely useful as it enables cheaper testing, diverse variations of test papers, better student preparation, etc. But would deeper research produce a usable passage selection system for tests? We try to answer.

We propose a multi-aspect text similarity search on a gold instance, instead of a bottom-up recommendation algorithm. A gold instance is a reading passage that has already appeared in previous test papers. Therefore, a gold instance has already gone through a standard passage selection procedure.

By searching for a text similar to the gold instance, instead of mimicking the full selection procedure, we can utilize the textual properties of the gold and skip steps difficult to imitate.

We need a multi-aspect text similarity search because the textual properties that make a passage appropriate are manifold (College-Board, 2017; Educational-Testing-Service, 2010). Though previous text similarity studies mostly focus on semantic similarity (Prakoso et al., 2021; Gomaa et al., 2013), we need to expand the scope to surface-level properties, lexical difficulty, etc.

CSAT tests the ability to study in college, based on South Korea’s high-school curriculum (KICE, 2014). Around 500,000 Korean students take the test every year. English is a required subject, and it puts much focus on reading comprehension. Each test paper has more than 20 reading passages taken from high-quality, previously published sources. We take CSAT as a running example.

Our proposed system consists of three main components: 1. database, 2. extractor, 3. filter. A database is a collection of stand-alone passages. The database is an arbitrary concept; it could be a gold instance’s original published source (considering a gold is an excerpt), the ACL Anthology, or even the whole internet. Developing a database is analogous to setting the system’s sensible search range. The passages in a database are run through extractors to give numerical representations of textual features.

Extractors are a collection of multiple single feature extractors. Some examples are a word per sentence extractor and a semantic encoding extractor. The numerical features (integer, float, or multiple numeric sequences), are then concatenated to the original passages to be saved in the database, awaiting for a gold instance to be passed.

When a gold instance¹¹1In this paper, ”instance” is used analogously to ”passage”. enters the system (Figure 1), it is first run through the same set of extractors (but not saved in the database). Then, the whole database goes through multiple filters to abandon dissimilar passages. The filters have pass conditions that compare the corresponding features between the database passages and gold instance’s to select a passage appropriate to replace gold.

Our database is a collection of instances, mostly a single paragraph, like in CSAT. All past CSAT English subject papers are publicly available²²2www.kice.re.kr. We take the test papers between 2021-2022; we searched the source articles.

We used optical character recognition (OCR) through Google Vision API on the retrieved source articles’ image files. We manually divided the result into paragraphs but we did not perform any inspection on the appropriateness of each item. We only went through basic preprocessing procedures. The database distribution is described in Table 1. The preprocessing steps are:

Then, we develop a series of extractors. The first is sentence-word counter using en_core_web_sm model pipeline from spaCy v.3.2.1 (Honnibal and Montani, 2017). We pass in each passage into a spaCy object and count items in the resulting token list and sentence list. Usages are publicly available³³3spacy.io/usage/spacy-101.

Second, we implement a lexical difficulty extractor using the Word Corpus of the Korean ELT curriculum (CoKEC-word) from Lee and Lee (2020). The dataset classifies 30608 words in 6 levels, with each corresponding to a specific grade range (A: suitable for grade 1 to 4 students, B: grade 5 to 8 students, C: grade 8 to 9 students, D: grade 9 to 11 students, E: grade 11 to 12 students, F: college).

Last, we implement semantic embeddings extractor using SBERT-based framework (Reimers and Gurevych, 2019). From the authors’ sentence-transformers library, we utilize the recommended model (all-MiniLM-L6-v2) (Wang et al., 2020).

A filter is a function of some data and a condition (filtered data = filter(data, condition)). Whereas a condition is a function of a gold instance, a database instance, and a parameter $\alpha$ (pass/fail = condition(gold instance, DB instance, parameter)).

Combining multiple filters in a sequence reduces the original database multiple times to leave only the passages that pass all conditions under given parameters. Below are the five conditions we use, in a consecutive manner from top to bottom.

For the above conditions, #Sent: number of sentences, WPS: #word per sentence, H_diff: highest word difficulty, HPW: #high difficulty word (grade 9 and above) per word. $\alpha$ is the parameter that sets the range. Semantic similarity is calculated through the cosine distance between gold instance’s (G) semantic encoding and database instance’s (D).

We randomly selected 10 gold instances from CSAT papers between 2021 and 2022. Since all CSAT reading passages are of similar lengths, the selected passages ranged from 134 to 181 words. From our database (Section 2.2), we ran the conditions (Section 2.4) to obtain 5 candidate reading passages for each gold instance.

The parameters we set for each condition are described in Table 2. These conditions are manually fine-adjusted to make the resulting number of instances as close to 5. As shown in the #Output column, it sometimes seemed impossible to give the desired #Output. This is due to the statistical distribution of the respective features in database.

The 5 instances that passed our conditions were concatenated to 5 dummy instances, from the same database but completely randomly selected. The 10 instances were shuffled for anonymity. Ultimately, we obtained 100 candidate instances along with 10 gold instances.

We presented the test set to 2 EFL teachers in South Korea, each with over 10 years of experience, and 1 South Korean EFL student who took CSAT in 2020. All 3 test subjects are employees in EFL educational firms or institutions.

For each trial, 10 candidate instances were shown along with the corresponding gold instance. The test subjects were asked to give suitability scores on a 5-point scale. Our question was "Is the candidate instance suitable as a CSAT reading passage?" They were also asked to give reasons as to why they gave the score.

We are interested in seeing if the filtered instances score meaningfully higher than the dummy instances. Table 3 shows the result score sums and correlations of the test outlined in Section 3.1. The Common European Framework of Reference (CEFR) levels is classified with the RoBERTa-RF-T1 model, retrieved from Lee et al. (2021). The model is state-of-the-art and gives 76.3% accuracy on the Cambridge dataset (Xia et al., 2016).

1. Multi-aspect similarity search gives meaningful statistical improvement from random selection. The filtered passages triumphed dummy passages in terms of suitability as a CSAT reading passage. This is an interesting phenomenon as the filters were not set to directly measure suitability. Rather, they measure similarity in terms of surface-level, lexical, and semantic textual features.

This hints that finding a passage similar to a gold instance has the potential to give a passage suitable for test papers. The filters used were relatively simple; with further improvements, the performance can improve. However, we failed to find meaningful correlations with any of the specific features. This leads us to our second insight.

2. "Suitability" is a multi-aspect textual property, definitely not captured on the surface level. From Table 3 and Fig. 2, it seems clear that one does not think a passage is suitable just by #words or #sentences. In addition, the classified CEFR level of a passage had no significant correlation with its suitability. The CEFR levels are analogous to a passage’s readability (Xia et al., 2016), which seems to have minimal correlation with suitability.

In fact, we find that no meaningful correlation and distribution exist between any tested single feature and suitability. On how filtered passages scored better than dummy passages, we believe that certain combinations of textual features lead to high suitability as a test reading passage.

One expects to see a certain type of a passage in a test paper. Such suitability is not measured by objective aggregation of textual features, unlike some other NLP tasks like readability assessment (Collins-Thompson, 2014). Hence, common prediction models like regression would not work well, complicating the task solution.

For a passage to be appropriate for reading comprehension tests, it is required that the passage is understandable by itself. The passage could be self-contained and not dependent on the preceding text, table, or figure. Otherwise, the uncertain parts should be easily inferable. Along with more sophisticated preprocessing methods that detect strings like "Table" or "Figure", we plan to develop inter-paragraph dependency detection algorithms.

The test outlined in Section 3 is based on a small population. Further, it is human-dependent and is not easily reproducible. In future works, we plan to introduce an objective evaluation method. This possibly consists of a classification dataset, where gold instances are mixed with its neighboring paragraphs from the same source article. Then, a model could be adjusted to choose the gold instance among the neighboring paragraphs.

In Section 3.1, we manually set the parameters $\alpha$ for each condition. This is yet another weakness that makes the study difficult to reproduce. For our filter-based method to be widely useful for both commercial and academic purposes, we need an algorithm that automatically adjusts condition parameters, based on the number of desired outputs.