PharmMT: A Neural Machine Translation Approach to Simplify Prescription Directions

The primary component of the proposed approach is a sequential model that “translates” physician-authored e-prescription text to normalized, patient-friendly language using an NMT framework. NMT models map a source sequence, $\mathbf{x:}~{}x_{1},x_{2},\ldots,x_{n}$ into a reference sequence, $\mathbf{y:}~{}y_{1},y_{2},\ldots,y_{m}$ by maximizing the conditional probability $p(\mathbf{y}|\mathbf{x})$ using an Encoder-Decoder framework DBLP:journals/corr/Neubig17. One such model is a recurrent sequence-to-sequence model, which consists of a bidirectional LSTM model schuster1997bidirectional with global attention as the encoder and a forward-sequence LSTM model hochreiter1997long as the decoder. Both encoder and decoder stages are configured as multi-layer models, with a hidden state in each layer, to sufficiently capture the deep semantic components in the e-prescription barone2017deep.

To compare against the performance of the recurrent sequence-to-sequence model, we also trained an attention-based transformer model NIPS2017_7181. Position embedding was enabled to capture sequence information and provide similar architectural complexity as a recurrent network. Both models were developed using the OpenNMT framework klein-etal-2017-opennmt; klein-etal-2018-opennmt, and the dropout probability was adjusted to prevent over-fitting srivastava2014dropout. Additional experimental details can be found in Section 4.2.

Once the machine translation module generates the simplified candidate directions, the candidates are checked for consistency of key components of the prescription. Numerical components – including dosage, frequency, and duration – are critical in prescriptions. Medication under-dosage often leads to poorer health outcomes; while over-dosage can be severe, even fatal.

The correctness of the numerical components in the simplified directions is checked by comparing against the source e-prescription. We incorporated two different numerical checking strategies – Token-based and NER-based. In the token-based checking, all numeric tokens that appear in the simplified direction were checked against numeric tokens in the source direction. The bag of tokens approach helps tag any simplified direction that “makes up” numeric values in a key component.

On the other hand, the token-based checker is also prone to generate more faulty consistency claims; for example, when the simplified direction swaps a dosage term with the frequency. To overcome this, we incorporated a pre-trained medication NER model XinyanNER to tag dosage, frequency, and duration components in both source and simplified directions, and compared them component-by-component. The NER model was trained over a medication extraction task henry20202018 and achieved an overall F1 score of 0.9571 over all medication components.

If the simplified direction is deemed consistent after the numeric check, the direction is considered as the final candidate for normalization. However, if the numeric check fails, the NMT output is discarded and the original source direction is used as the final candidate. This graceful backoff represents a trade-off between information accuracy and good language model performance.

Before the candidate direction from the numerical check phase is finalized, the candidate text undergoes pharmacy-specific post-processing and simplification. Two pharmacists identified common linguistic patterns in pharmacy directions, which were coded into normalization rules. Highly-abbreviated medical jargon was replaced, for example, by replacing the Latin term bid with its synonymous phrase twice a day. Action verbs appropriate for the form of the drug, such as inject (syringe), inhale (nebulizer), and take (capsule), were added. Numerical values in words or fractions were converted to digits (e.g. 1 1/2 was converted to 1.5). Abbreviations and other common medical expressions were normalized into standard and patient-friendly variants, e.g. inj or injector to injection. Overall, the normalization step included more than 300 rules. Table 2 shows examples of these normalization and simplification rules. The normalization module also served as the rule-based baseline.

The e-prescription corpus used in this study consists of all e-prescriptions dispensed by an online outpatient mail-order pharmacy from their dispensing software from January 2017 to October 2018. The corpus consists of a total of 530,988 e-prescriptions received from 65,139 unique physicians from all fifty US states Pharmacy_2020.

Each e-prescription direction in the data set is paired with the corresponding simplified text authored by a mail-order pharmacy team member. Table 1 shows some example e-prescription directions and the corresponding simplified text. In addition to the e-prescription, each direction in the corpus also contains auxiliary information about the name and strength of the drug. However, similar to the directions, physician-authored information often included chemical or ingredient names for the drug, while the pharmacist-translated direction contained generic or brand names for drugs.

In all, there are 120,402 unique e-prescription directions and 83,823 unique pharmacist-authored directions in the dataset. The difference in these numbers is due to the diverse writing styles of physicians and pharmacists. On an average, there were 6.33 e-prescription directions mapped to a single pharmacist-authored direction; while one e-prescription direction mapped, on an average, to 4.41 different pharmacist-authored directions.

To avoid information leak during the evaluation, we split our data into train, validation, and test sets so that there were no duplicates across the sets. Duplicate e-prescription directions were grouped and assigned to only one of the three data sets. Table 3 summarizes the distribution of the instances over the three data sets. None of the instances in the validation and test sets were used during the training phase. TODO: Explain

To prepare the data for training, the source e-prescription directions and reference pharmacist-authored directions are prepended with the drug name and strength, as described in Section 3.1.1.

We compared two classes of NMT models – Transformer-based model and BiLSTM/LSTM-based model under different pre-trained word embeddings. The results are summarized in Table 5. The reported values are mean and standard deviation over 10 independent iterations of training and validation using the same model hyper-parameters.

Both automated metrics were consistent in their ranking of the systems. On both metrics, LSTM-based models outperformed transformer-based models. One possible explanation for these results is that although transformer-based models are better at capturing long-range dependencies, their advantage is nullified by the relatively short sentences in this task. The average length of e-prescription directions is 10.42$\pm$4.55 tokens.

Models using the pre-trained clinical-domain word embeddings led to the highest performance on both metrics, and were statistically better than models that used general-domain word embeddings. Models using the randomly-initiated word representation performed the worst. The model performance improved further when ensemble learning was applied. The ensemble BiLSTM / LSTM model with clinical domain-adaptive word embeddings achieved the highest overall BLEU score of 66.61$\pm$0.29 and METEOR score of 82.73$\pm$0.10.

We also note that in Sec. 3.1.2, we stated our hypothesis that domain-adaptive word embeddings would outperform both the general-domain embeddings and randomly-initialized vector representations. This hypothesis was shown to be valid in our results in Table 5. Instead of the static, but domain-adaptive, word embeddings explored in this work, other alternatives such as contextual word embeddings could also be used, including BioBERT BioBERT and ClinicalBERT ClinicalBERT.

Next, we compared the translated directions generated by the PharmMT model against a rule-based baseline in which the e-prescription is passed directly through the Normalization module to produce the output. Table 7 shows some examples of rule-based translation against PharmMT output. These examples highlight three potential issues of rule-based systems, viz., contextual ambiguity, reordering of direction components, and sensitivity to misspelled tokens. In the first example, the rule-based approach fails to recognize ‘90’ as duration without any contextual clues, whereas PharmMT correctly normalizes it as ‘for 90 days’. Similarly, in the second example, unlike the rule-based approach, the PharmMT model correctly reorders the direction components by inserting dosage tokens (‘3.5 tablets’) before the form token, ‘tablet.’. Rule-based approaches are also more sensitive to misspelled tokens, compared to PharmMT, as shown in the third example.

The best performing NMT model achieved a BLEU score of 67.21 and a METEOR score of 82.90. After finalizing the best NMT model, the significance of the remaining components is shown using an ablation study. The results are summarized in Table 8.

The final output of the end-to-end system was evaluated by domain experts. Of the 300 pairs of e-prescription directions and their corresponding simplified texts, 86.7% (n=260) were labeled as Correct, 7.6% (n=23) as Missing; and 5.7% (n=17) as Wrong. The missing errors were primarily related to missing adjectives and adverbs (e.g. transdermal, slowly), or typographic omissions, such as brackets. The incorrect errors were primarily related to special directions (e.g. taking medications before meals, with food), formatting issues with dosage (e.g. 10-12), or complex directions based on days of the week (e.g. every day except Sundays). These results show that in 94.3% instances, the simplified output can be used as-is or after minimal changes to add the missing elements.

We further analyzed all non-‘Correct’ instances (n=17; 5.7%) identified during the manual evaluation. A major class of errors was complicated instruction and language patterns in the e-prescription. These directions were on average 16.86 words long, compared to an average of 12.57 words for ‘Correct’ instances. For example, one direction that was not simplified correctly was: 30 units with meals plus ssi 150-200 2 units, 201-250 4 units, 251-300 6 units, 301-350 8 units, greater than 351 10 units. This direction instructed patients to change dosage depending on the sliding scale for insulin (ssi).

Based on the hypothesis that shorter directions will have simpler language patterns, we evaluated a subset of test instances (n=11,977; 32.6%) that were under 12 words long. This ‘shorter length’ subset achieved an aggregate BLEU score of 71.14, while the complementary ‘longer length’ subset managed an aggregate BLEU score of 64.02.

We further investigated the performance of the numeric checker. The number of instances marked as inconsistent by the two approaches is summarized in Table 9. The stricter, NER-based numeric checker flagged 4,027 (17.33%) instances as inconsistent, while the token-based checker flagged only 1,390 (5.98%) of instances as inconsistent.

The normalization process is heavily influenced by the style preferred by pharmacists coding the normalization rules. Since the preferred style of the team that generated the original reference differed from the expert pharmacists in our team, the original reference data were themselves not normalized. This led to a reduction in BLEU scores when normalization was added (cf. Table 8).

To understand the upper bound of our trained models, we created a normalized version of the reference corpus. Using this as gold reference, the original reference corpus has a BLEU score of 82.48, and that of the PharmMT system was 62.68 (see Table 10). The table also shows the ratio of test instances that are normalized, to indicate how close the output is to the normalization rules: the lower the ratio, the closer it is. The results indicate that the NMT model learned more latent rules from the train data set than the hand-crafted normalization rules, while having a higher BLEU score.

Automated metrics such as BLEU and METEOR can only evaluate the translation results using a linguistic, token-level approach, but fail to capture the nuanced semantic-level information. However, in prescription directions, different words contain unequal useful information, and hence should be re-weighted during the evaluation process. As we noted in Sec. 3.2, consistency of key information is vital for patient safety. For example, in the translation shown in bottom of Table 6, both machine translation outputs NMT₁ and NMT₂ have only one token different from the reference. But, NMT₁ is labeled as ‘Correct’ in the manual evaluation while NMT₂ is labeled as ‘Wrong’ because of a serious error on dosage. However, both translations got similar BLEU and METEOR scores. In the future, we will focus on improving information consistency while maintaining high model performance.