\system: Evaluating Interactive Human-LM Co-writing Systems

Language models (LMs) have advanced significantly, showcasing previously unheard-of capabilities in solving a wide spectrum of generation and language understanding tasks (Rae et al., 2021; Chowdhery et al., 2022; Ouyang et al., 2022). This has spurred great academic and public interest in using LMs to build writing assistants, in which humans collaborate with LMs to paraphrase sentences (e.g., QuillBot), autocomplete sentences (Chen et al., 2019), write stories (Akoury et al., 2020), etc. Despite the interactive co-writing process, the co-writing systems are at present primarily tested in non-interactive settings (Yuan et al., 2022; Dang et al., 2022). Specifically, current studies commonly conduct evaluations only on the final, co-written article (Mirowski et al., 2022), or prior- and post-human assessment on perceiving LMs (Yuan et al., 2022; Singh et al., 2022), etc. Consequently, these evaluations fail to capture the delta (or dynamic shift) of human-LM interactions. Consider, for instance, a scientific paper writing task involving two types participants, freshmen and a university professors. While it may not be surprising that the professor-LM team achieves significantly better writing quality, this metric does not reflect the fact the impact of the LM on its users: Freshmen might have benefited significantly from the model in terms of the scientific paper structure, grammar correction, etc., whereas professors might have achieved similar writing performance even without the LM. In other words, the quality of the final article cannot genuinely reflect the co-writing system’s influence on users. Instead, we should assess users’ dynamic interaction improvements to indicate system capability, such as the relative quality change between two iterated articles from the same user.

In spite of the importance, to the best of our knowledge, there are few studies that examine how to assess the interaction shift in the iterative co-writing process, and how to depict an integrative view for evaluating interactive co-writing systems. To close this gap, we propose \system: a human-centered evaluation framework for human-LM interactive co-writing systems. We identify the key components and interaction aspects for evaluating the co-writing systems. We further collect the comprehensive types of evaluation metrics under each aspect, including the novel measurements designed for dynamic interaction assessments, and the other two conventional yet important aspects (i.e., human-LM interaction and writing artifact evaluation). Though a case study, we show that \systemcan be effectively used as a thinking tool for comprehensively evaluating and comparing the co-writing systems.

We aim to propose \systemas a guideline that can assist researchers in evaluating and analyzing interactive co-writing systems more comprehensively and fairly.

To this end, we focus on identifying key axes for co-writing systems. To begin with, grounded on the Co-Creative Framework for Interaction Design (COFI) model (Rezwana and Maher, 2022), we identify three key components in human-LM co-writing systems: it involves both human and LM collaborating on a shared writing artifact (e.g., essay, story, paper, etc.) as partners (Rezwana and Maher, 2022). Among these three components, humans are the primary decision-makers for interacting with LMs and writing artifacts. To reflect its importance, we design \systemto be a human-centered evaluation framework, meaning that the ultimate objectives of these metrics is help humans achieve their various writing needs (e.g.,, better user experience, higher-quality writing artifact, etc.).

As shown in Figure 1, \systemfurther recognizes three aspects, which are important around these components and also the interactions between them, including evaluate human-LM interaction, evaluate dynamic interaction trace, and evaluate writing artifact. Based on existing evaluation approaches, which might capture insufficient evaluation aspects to reflect the system capability holistically (Yuan et al., 2022; Mirowski et al., 2022; Dang et al., 2022) or focus much on one-time metrics that neglect dynamic interaction changes (Singh et al., 2022; Lee et al., 2022), \systemseeks to contribute in two folds. First, \systempresents a more integrative view of interaction evaluation supported by practical metrics. Besides, \systemexplicitly extracts a set of metrics to assess the dynamic change along iterations.

How can \systemguide practical evaluation? We use \systemto analyze existing evaluations, and reflect on what axes these metrics emphasize Concretely, we use an inductive approach to collect the practical metrics adopted in state-of-the-art co-writing systems (e.g., Wordcraft (Yuan et al., 2022), Integrative Leaps (Singh et al., 2022), Beyond Text Generation (Dang et al., 2022), Dramatron (Mirowski et al., 2022), etc.) Then we categorize them into subgroups and fit into \systemframework. Figure 1(b) depicts the categorized evaluation metrics for each \systemaspect. We next clearly define the aspects and metric subgroups in \system, and briefly explain the underlying motivation. Please see Table 1 in Appendix A.1 for more elaborated metrics and details.¹¹1Note that we do not aim to build enumerated lists of evaluation metrics. Instead, we focus on introducing the motivation and process of creating these evaluation aspects and subgroups, which can be generalized in broader use..

We consider \systemas a framework for researchers to fairly evaluate and compare co-writing systems. \systemcan be useful for researchers to: 1) identify key interactive evaluation aspects among human, LM, and writing artifact interactions during the co-writing process, 2) select appropriate metrics to assess and compare the co-writing systems, 3) comprehensively analyze and describe the human-LM interactive performance of the co-writing systems. Next we present a concrete use case of re-evaluating the Beyond Text Generation (BTG) (Dang et al., 2022) system to showcase how to use \systemfor evaluation.

Suppose the researchers have built the BTG system and need to evaluate its performance in an interactive setting. \systemcan help them assess the system comprehensively and compare it to existing baselines. For example, they can use the \systemframework to unpack their hypothesis into fine-grained evaluation requirements. A mapping may look like: “the BTG system can help humans write better articles (i.e., aspect3: evaluate writing artifact) by enabling better human-LM interactions (i.e., aspect1: evaluate human-LM interaction) in efficient ways (i.e., aspect2: evaluate dynamic interaction trace). With these aspects in mind, they can then dive into each aspect to select the appropriate metrics to support this statement. For instance, they can assess “writers’ perceptions of LM” by choosing enjoyable, preference metrics, and “how writers think about LM’s feedbacks” with usabiity, effective, coherence metrics, etc. Also, they can assess the dynamic interaction efficiency by analyzing the objective logging data (e.g., incorporation rate, latency). The final writing article can also be rated with a set of measures (e.g., external expert review, quality, satisfaction, ownership, etc.). Following the evaluation methods presented in Appendix A.2 (which captures the most common methods used for state-of-the-art system evaluations), they can map the metrics to actual user study designs. Note that the researchers apply all measurements to both the proposed BTG system and the baselines, and ideally with the same group of users, so that they can compare the performance of co-writing systems in a fair manner.

This work present \system: a human-centered framework to evaluate human-LM interactive co-writing systems. It provides a thinking tool for researchers to design comprehensive interaction evaluations and analyses. We further feature a use case study introducing how to use \systemstep-by-step for fair evaluations.

We especially Thank Dr. Ting-Hao ’Kenneth’ Huang for his insightful feedback, valuable support, and co-organizing the In2Writing Workshops!