Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple Wizards

Wizard of Oz is a standard approach to evaluating prototypes at various stages of development and at varying levels of fidelity (Dahlbäck et al., 1993). Often the most difficult, expensive, or impossible to develop aspects of the system are left up to WoZ for early testing. In one of the earliest examples, IBM created a WoZ platform for a user interface in their personal computer, which was still an uncharted territory in the mid-1980s (Green and Wei-Haas, 1985). Mander et al. (Maulsby et al., 1993) created Turvy, a WoZ agent for exploring futuristic human-agent interactions that were difficult or impossible to achieve in the 1990s. Users were asked to “teach” the agent by talking to it while pointing at objects, and the Wizard behind the curtain registered and responded to this information, allowing for smooth and prescient testing of this futuristic vision of technology. Many WoZ platforms are essentially early versions of the final products, and so do not have one-size-fit-all WoZ platform that can cater to all variations.

Within research spaces of HRI, the end goal may not be a new platform, but rather a new understanding of how people use and relate to computers. Platforms that allow for WoZ have thus become widespread within research, such as Aldebaran-SoftBank’s humanoid robot Nao⁵⁵5 https://www.softbankrobotics.com/emea/en/nao or the general platform ZBOS applied to a care robot called Zora ⁶⁶6https://zorarobotics.be/zbos-zora. In particular, it comes with a simple programmable interface, text-to-speech (TTS) tool, and 3D simulation environment that have been frequently co-opted for WoZ in research (e.g., (Seaborn et al., 2021; Riek, 2012; Sandygulova and O’Hare, 2018; Wigdor et al., 2016)).

Platform specificity (or “one-off” prototypes, such as Turvy) and researcher flexibility (as with the widespread uptake of the Nao) are common orientations to WoZ in research. However, there are some general WoZ tools and platforms that can be integrated into a system under development or customized for the research question under study. As Cambre and Kulkarni recently identified (Cambre and Kulkarni, 2020), these prototyping methods may be designed to allow for WoZ of a specific feature or function (e.g., elicitation methods, dialogue management systems) and not others. Many of these have been developed for or as a result of research. SUEDE (Klemmer et al., 2000) was a visual prototyping tool designed to help researchers quickly create speech UIs without needing to do much coding. WebWOZ (Schlögl et al., 2010) was one of the first web-based, cross-platform WoZ tools for research involving dialogues between people and/or computers. Marionette (Wang et al., 2017) was developed to fill the gap in WoZ platforms for human-vehicle interaction, an emerging area of research facilitated by the increased use of voice-based, hands-free smart agents in vehicles (or as the vehicle itself). WebApp (Avula and Arguello, 2020; Schlögl et al., 2013b) was designed to address the gap in text-based conversational interface WoZ tools for search, particularly collaborative search within Slack⁷⁷7https://slack.com/intl/zh-hk/. Commercial platforms, such as Voiceflow⁸⁸8See https://www.voiceflow.com/ and Alexa Skills Kit⁹⁹9See https://developer.amazon.com/en-US/alexa/alexa-skills-kit can also be adapted for research as well as other purposes, but may be costly, limited to certain technologies (e.g., Amazon Alexa), or otherwise restricted due to proprietary matters.

Most WoZ platforms for voice interaction, both classic and modern, have focused on the problem of speech recognition and speech output in information retrieval contexts: natural language processing (NLP) (Diaper, 1989; Richards and Underwood, 1984; Guindon et al., 1987; Jönsson and Dahlbäck, 1988; Whittaker and Stenton, 1989). A survey of HRI works revealed that 72.2% of WoZ approaches to robots were selected for this reason (Riek, 2012). Shamekhi et al. (Shamekhi et al., 2018) compared voice-only and bodied versions of conversational agent as a collaborator in a group-based decision-making activity wherein WoZ was used to detect speakers’ intent and provide a response. As voice interfaces have proliferated and matured, new factors and concerns have arisen. Children, for instance, have been identified as a new and potentially vulnerable user group. Yarosh et al. (Yarosh et al., 2018) created three WoZ speech interfaces to study the patterns of questioning and answering amongst children. Context also matters as research moves “into the wild” in field studies. Braun et al. (Braun et al., 2019) developed a multi-personality voice-based smart vehicle agent that was tested under realistic driving conditions. The Wizard was responsible for changing the response patterns of the agent at opportune moments.

As this body of works show, speech-based interface is a booming area of study for which robust and ideally general WoZ platforms are necessary. We contribute to it by extending the remote, web-based, open source, cross-platform, and multi-modal approaches explored so far to speech-based interaction and multiple Wizards. Next we turn to the challenge of supporting multiple Wizards in the handling of increasingly sophisticated interaction scenarios of the near future–by elaborating on the Wizards’ own experiences as users of such technologies in a collaborative context.

Taking on the role of a Wizard can be a challenging task. Training is often required, but it may not eliminate or mitigate the difficulties the Wizard may experience. Difficulties are compounded when the Wizard does not have assistance and communicate directly with others involved in conducting the study, for fear of revealing the fiction of the Wizardry to the participant (Porcheron et al., 2021a). Little is known about the exact challenges that Wizards face, as the Wizard’s experience is not often reported in the literature. Still, some challenges have been identified.

One challenge is the cognitive load. Shin, Oh, and Lee (Shin et al., 2019a) identified four areas in which a Wizard may experience cognitive overload when playing the role of a conversational robot as they need to multitask between paying attention, decision-making, task-execution, and reflection. Their solution was to introduce an autonomous agent collaborator to assist the Wizard in their Wizardry. Large et al.  (Large et al., 2016) tested a digital driving assistant (DDA) in a vehicle driving simulation environment. Their setup featured two Wizards, each responsible for one cognitively demanding task: conversing with the driver and assisting with search and navigation. This work reduced cognitive load by dividing labor.

Another challenge is multi-modality. Salber and Coultaz  (Salber and Coutaz, 1993a) developed a WoZ platform called Neimo to support multi-modal designs. They considered that each modality could be the responsibility of each individual Wizard, e.g., a Speech Wizard, Face Wizard, and Mouse Wizard. Cohen et al.  (Cohen et al., 2008) explored this notion explicitly in the design of a WoZ platform that allows Multi-Wizard control of the key interface components, pen and speech, dividing the labour between two Wizards roles: content and user pen output analysis.

A third challenge relates to NLP, which is the foremost reason for employing WoZ in voice-based systems, conversational agents, and speech interfaces because they entail open-ended responses. Yarosh et al. (Yarosh et al., 2018) provided Wizards with a list of preset responses but allowed for these to be changed on the fly. The necessity to allow for open-ended responses from the Wizard was also found by Vtyurina et al. (Vtyurina et al., 2017), who recognized that searching for preset passages could impede the performance of the Wizard compared to simply allowing them to write the passage off the cuff.

A related challenge comes from the fact that computers have yet to master the detection of implicit and/or nonverbal cues, whereas our Wizards certainly could. Vtyurina et al. (Vtyurina and Fourney, 2018) identified several ways in which users conveyed an expectation of a response from a conversational cooking assistant. For instance, when the user was ready to move on to the next step, they did not explicitly say so, but rather used a vague endorsements such as “okay,” which the Wizard could understand and act upon.

The literature so far has focused on how to support solo Wizardry. Multiple Wizards can be trained to carry out the work in shifts (e.g., (Yarosh et al., 2018)), but this can introduce inconsistent experiences for the participants. While it may be effective to apply a truly Multi-Wizard approach in which multiple Wizards working in concert, this approach has been far less explored with only a few reported examples. For instance, a couple of studies (Green et al., 2004; Marge et al., 2017) have considered dividing the labour involved in controlling a robot within a co-present WoZ context: one Wizard controlling the dialogue and the other controlling the robot’s movement. Another work extended this idea to a remote context (Bonial et al., 2017), however, none of them has explicitly evaluated the Wizards’ experience. A few (e.g., (Serrano and Nigay, 2010)) have proposed Multi-Wizardy in future work but these ideas have yet to materialize.

Whether considering solo- or Multi-Wizard contexts, evaluating the experience of the Wizard as a performer and/or user of the WoZ platform has received little attention in general. Riek (Riek, 2012) found that most studies constrained what the Wizard could do (90.7%) but not what they were able to perceive (11%). Very few reported measuring Wizard error (3.7%) or reported pre-experiment Wizard training (5.4%). In these cases, the effects of these factors on the act of Wizardry as well as the experience of the Wizard are not known.

In this work, we focus on the experience of WoZ from the standpoint of the Wizard, treating the Wizard as an essential user of the platform and their experience as a key variable in the success of the WoZ approach for end-users. Additionally, we evaluate not just the Wizard’s experience as a solo actor, but also their experiences within a collaborative Multi-Wizard context. To the best of our knowledge, this is the first work to do so, generally and within WoZ for speech interfaces. As our platform essentially supports intelligent features by leveraging input from multiple human Wizards , the design has similarities with some real-time crowd-sourcing platforms. Thus, we will now review related literatures on these plaforms to explicate the differences between our approaches.

Real-time crowd-sourcing platforms provide similar functionalities to our system, in terms of enabling the division of labor and in facilitating the workflow for combining their outputs. Researchers have made use of advanced interface designs (e.g., visualizations of collaborator actions and separate work-spaces) and task allocation mechanisms in workflows (e.g., complex-tasks-to-micro-tasks), enabling workers to collaborate freely and efficiently to complete the goal (Lasecki et al., 2015; Huang et al., 2017). Some have investigated synchronous crowd contexts wherein a large number of disparate users jointly manipulate a user interface) (Lasecki et al., 2011), whereas others, such as Lasecki et al. (Lasecki et al., 2011), aggregated disparate user entries into a single application workflow so as to return work results quickly to crowd-workers. In other words, this approach combines multiple real-time contributions into a single command. Similarly, Kim et al. (Kim et al., 2013) designed Cobi to reduce work conflicts by providing visual indicators of each worker’s task status, to decrease the time consumed in resolving conflicts among workers. In line with this approach, Wizundry also provides visual indicators for each Wizard, allowing for action transparency for efficient collaboration, especially by providing real-time cues where synchronized tasks may be possible. To address such challenges, Bernstein et al. (Bernstein et al., 2010) split complex writing and editing work into several micro-tasks, assigning various people to work synchronously but independently by applying a sequential, ‘Find-Fix-Verify’ method. This enabled pre-assignment of different tasks to each crowd-worker on separate working interfaces. Inspired by these design considerations, we designed our toolkit with separate work interfaces that support the division of speech-based interface simulation tasks, while enabling us to investigate the division of labour that can emerge among multiple Wizards.

Who manages the workflow is another key consideration. Lasecki et al. (Lasecki et al., 2015) considered this issue in the context of WoZ-driven systems, which are often one-offs that still require significant development work. They designed their system Apparition to provide a ”lightweight write-locking mechanism” so that workers could self-manage task delegation in real-time while also communicating with each other about what components they are modifying and what tasks they are working on. This mechanism allowed workers to self-manage task delegation in real-time. Similarly, we designed Wizundry to enable multiple independent Wizards to self-allocate simulation tasks, and to synchronize this information with other Wizards to improve the Multi-Wizard workflow that results in a better end-user experience.

Following these previous studies on crowd interface design, especially in view of multi-worker workflows, collaborative mechanisms, and the collocation of tasks, we designed our platform with operational independence from the Wizard’s role. As a research tool, the Wizundry intelligent user interface leans on these previous endeavours by supporting cooperation among multiple Wizards who may take on unpredictable, ill-defined, and shifting tasks in real-time. We also aimed to reduce the challenges and workload experienced by Wizards by increasing their team capacity. In the next section, we describe the design of our Multi-Wizard platform: Wizundry.

We designed Wizundry 1.0 to allow researchers to (1) design and orchestrate speech-based WoZ studies with Multiple Wizards; (2) simulate speech-based intelligent features (such as keywords spotting and editing through natural speech); and (3) study how to provide better end-user experiences. The Wizundry consists of two interfaces: the Wizard interface (Fig 2) and the end-user interface (Fig 2). The Wizard interface allows Wizards to (1) simulate a smart dictation interface by editing machine-transcribed speech text, (2) work collaboratively with others to accomplish tasks, and (3) test coordination workflows to improve the experience for end-users. The end-user interface is independent, and was used to test the simulated speech interface. This toolkit provides a modular and configurable interface that builds on two core functionalities: Speech-to-Text (STT) and Text-to-Speech (TTS), which can be used together or separately in each interface.

We conducted a user study to test the initial design of Wizundry 1.0 and get a preliminary understanding of the Multi-Wizard experience. A pilot study with two groups (G1 and G2) and a main study with four groups (G3-6) were conducted. We were especially interested in their strategies and cognitive workload, one of the key human factors hinted at in previous work. To this end, we: 1) compared Wizard behaviour in solo- and multi-Wizard conditions; 2) observed the strategies that Wizards naturally employed while performing in pairs; and 3) assessed the usability, UX, and subjective cognitive workload.

Wizards were able to develop, employ, and modify their own approaches and strategies on the fly within the dyad-Wizard context of Wizundry 1.0. Dyads were able to freely define and agree on a workable approach to text- and audio-based Wizardry. Much of the current literature (e.g., (Klemmer et al., 2000; Yarosh et al., 2018; Shin et al., 2019a; Wang et al., 2017; Wigdor et al., 2016)) has focused on how a single Wizard conducts WoZ experiments or on the various ways designers can reduce the Wizard’s workload. In our study, we emphasized observing how participants create and adapt within dynamic and complex, yet supportive and collaborative Wizarding contexts. We first present initial findings from the study, and then describe the revisions we subsequently made to Wizundry and the study procedures.

Wizards were able to tackle a variety of challenges as they experimented with the various novel strategies to collaborate in Wizundry 1.0. Most dyads focused on how to handle the demands of the WoZ dictation task, but neglected teamwork as a result. In addition, communication and personality factors influenced the dynamic within the dyad, which then affected performance. When unexpected events happened outside the established cooperative approach, some participants initiated communication with the other Wizard, positioning themselves as the leader or dominant personality. Often this led to performance improvements, with the dominant personality proactively changing their behaviour as needed. In short, Wizundry 1.0 was easy to use in most cases, and the addition of a partner could be useful.

Wizundry 2.0 includes three interfaces: (1) the Landing Interface, (2) the Wizard Interface, and (3) the End-user Interface (see Fig 5). Here, we describe how we translated our findings into design goals and then implemented them in Wizundry 2.0.

We conducted a case study to evaluate Wizundry 2.0 with three Wizards collaborating together to simulate more advanced intelligent system features in addition to dictation. We targeted an imagined future dictation system that supports time-critical conversational tasks, such as an interviewer managing an interview process or debaters engaging in debates using agendas. Traditional dictation software cannot support such time-critical tasks well because it takes time to read lengthy transcriptions that have no structure, summaries, or highlights.

For our case study, we chose an interview speech notes processing task because it is complex enough to require multiple Wizards when simulated, as well as covers most transcription and content analysis requirements for research purposes, such as initial qualitative processing of transcribed text. We aimed to test whether and how Wizundry 2.0 can help researchers prototype and test future interfaces. We evaluated the effectiveness of our Multi-Wizard platform based on the following aspects: Can it help researchers simulate a near-future system that has advanced intelligent features, one that is hard to simulate by one Wizard alone? Can it support fast iterations of the simulated interface by adapting to the cooperative strategies of multiple Wizards? Can it help researchers learn how to improve the end-user experience? We describe the details of the study next.

We first present the findings from the Wizard collaboration results, including four cooperation strategies and perceptions of the effectiveness of Multi-Wizard collaboration as a triad. We then cover reactions from the Wizards and End-Users who acted as the Interviewer and Interviewee. Finally, we review the interaction patterns that we observed in each group.

Both studies 1.0 and 2.0 showed that having Multiple Wizards sharing the workload of simulating a real-time intelligent system is a feasible approach, sometimes necessary. Can Multiple Wizards provide a better experience than a single Wizard? Our studies indicated that this is certainly possible. In study 1.0 we tested with a relatively simple system that handles only text input and editing. The task distribution between Wizards were mostly uneven (Fig 3), where one Wizard did most of the work and the other was assisting. An interesting effect we observed was that not only the action of other Wizards, but also the anticipation of action by other Wizards brought much stress to the team in fear of conflicts. Since being a Wizard for a real-time interactive system is such a time-critical task, the uncertainty that emerge from overlapping task areas amongst participants was highly detrimental because participants had to pay extra attention to avoiding conflicting edits. Due to this cost and the coordination overload, we received mixed feedback from Wizard participants about whether it was actually helpful to have one more Wizard doing the task. Therefore we designed study 2.0 to test a scenario where clearer division of labor can be made and where the Wizards needed to simulate more advanced system features. The result was positive for study 2.0 as Wizards found the teammates’ work helpful and essential, and the system was able to support their collaboration effectively. From 1.0 to 2.0 our findings show that a Multi-Wizard approach works better for simulating complex systems with clearer separation of responsibilities among Wizards.

Besides sharing workload, we found that Wizundry is an effective platform for collecting and testing creative ideas for designing intelligent system behaviours. The Wizards in our studies came up with novel collaborative strategies and system behaviors. In study 1.0, the Wizard participants came up with interesting ways of dividing their tasks: content based, interface-controller based and input-device based approaches. Some dominant collaboration styles also emerged to reduce conflicts. Under the dyadic approach, Wizards focused on how to divide work and complete the same task. Wizundry was designed to allow Wizards to self-organize and divvy up tasks. We discovered that some participants actively support their partner, which was usually disruptive. In study 2.0, the Wizards redesigned their workflow by making sub-tasks sequential, or even by using the speech boxes to negotiate the pace of work with the end-user. These findings were a surprise to us and made us realize an unexpected use of Wizundry platform as an ideation + evaluation tool for designing intelligent system features with very fast iteration capability.

The Dyads and Triads studies showed the possibility for facilitating cognitively demanding tasks for wizards. Wizundry presented a novel Multi-Wizarding platform to participants acting as Wizards, assisting tasks that were cognitively demanding for them. Wizards were tasked with simulating advanced functions and features while dynamically responding to unpredictable requests, in a speech-based AI interaction context. As such systems become more innovative, users will continue to expect more advanced functionality and open-ended engagement from Wizards. The ability of each Wizard to grapple with the demands of such advanced tasks and system complexity will affect the degree to which the simulation is perceived as realistic and believable. Essentially, this is a matter of the cognitive demands placed on each Wizard. For us, this finding meant designing Wizundry to address these challenges in appropriate and useful ways. Including multiple Wizards to offload cognitive and task labor was but one design strategy. The used approach to the design of Wizundry appears to have helped facilitate Wizards in handling cognitive demands across different tasks.

Although previous works explored the challenges of being a Wizard (Shin et al., 2019b), we are not aware of any existing work of formal studies evaluating Wizard experiences. Our findings revealed empirical insights on the challenges of Wizarding and their collaboration, leading to questions for future research.

The primary challenge we faced was the coordination between Multiple Wizards. Although we improved on reducing potential conflicts from version 1.0 to 2.0 by conducting our study in a shared physical space where Wizards can interact more fully and in real-time, it still happened from time to time. In both studies there was a voluntary emergence of leaders in order to help coordinate the actions of the team and even to interact with the end-user to negotiate the task rhythm. Such collaboration style was mostly appreciated by other team members. However, all the Wizards in both studies reflected the high temporal pressure of the task, which led to little, or no time, for verbal communication between Wizards. Work conflicts occurred amongst multiple Wizards in our studies, especially when the responsibilities were not clearly separated. These findings are inline with previous works in terms of coordination and synchronization, we also found challenges in our experiments, especially with editing conflicts and workload management, which are similar to other real-time collaborative work contexts involving multiple people (e.g., e-work and online teaching) (Knister and Prakash, 1990; Nof, 2003). The collaborative cursor provided in our system was a basic function to provide awareness of Wizards’ actions among them. Moving forward, much more could be done to further improve this.

Previous studies on computer-supported working mechanisms addressed these challenges by enhancing the exchange of information between team members, such as updating decision-making algorithms and establishing alternative communication channels (Shamekhi et al., 2018). Another way was to design advanced user interfaces and use rhetorical visualisation to enhance users’ interaction effectively, which may help reduce the cognitive load (Huang et al., 2017). Some studies focused on using designed workflow to handle and segment complex tasks, especially in the crowd-sourcing field (Knister and Prakash, 1990; Kulkarni et al., 2012). To address editing conflicts in the co-writing system, some solutions aim to track the working actions and present the area of issues in the system interface, allowing the user to resolve it themselves (Knister and Prakash, 1990). Such solutions could be used in future development of Wizundry systems to better assist Wizards coordination. Moreover, we can draw further inspiration from literatures in “Social translucence”, a theory-based design approach proposed by Erickson and Kellogg in 2000 (Erickson and Kellogg, 2000). They proposed a framework for designing coordination mechanisms and collaboration norms by deploying a shared visualization of each user’s activities. Following this line of thinking, we can imagine that future versions of Wizundry could visualize other signals of Wizards’ activities, such as their gaze focus locale, to provide more timely collaborative awareness.

The second challenge we identified in both studies was the synchronization and communication between Wizards and the end-user. The end-user’s locale of attention was not obvious for the Wizards. In a dictation and text processing scenario, this was particularly problematic as the lengthy dictation led to Wizards losing track of where the End-User or other Wizards are. We believe this is an interesting design challenge that can be tackled in future work, to enhance communication between users and Wizards or intelligent systems. Moreover, our findings in the end-user experience of study 2.0 showed the importance and promising effects of providing customized assistance as requested by the end-user. It also appeared to us that it may be better to give part of interface control to end-users, to support close collaboration between the user and the Wizards, rather than simply letting the user verbally tell the system what she/he wanted. We will explore this in our future work. Last but not the least, findings in study 2.0 showed that prior knowledge of the end-user’s agenda, goals and preferences were much needed for Wizards to assist them effectively. Thus a prior “system configuration” with end-users’ input is a necessary step for future studies alike, since what the Wizards felt would be a good experience may be completely off for end-users (example Group 2 in Study 2.0). Human-simulated systems carry human biases determined by their personal experiences and backgrounds. Such biases can be more than rule-based or data-trained computer systems, and vary a lot across different Wizards.

The Wizards in our experiments occupied the same physical room to complete their respective tasks. Since they were in the same space, they could freely communicate face-to-face and use each other’s screens to negotiate their work during the experiment. In the discussion sessions where the Wizards were making strategies, we observed Wizards discussing with each other by pointing each other’s screen and using hand gestures to facilitate explanations. These verbal and gestural communication was effective and efficient in supporting their discussion. However, we rarely observed such communication when they were performing the tasks. This was due to the high temporal pressure when trying to follow the speech input of End-Users - they did not have time to communicate. Therefore, we believe Wizundry could be used with Wizards located remotely, as long as they plan ahead before tasks starts with a video conferencing call while sharing screens.

While our case studies focus on smart dictation interfaces with specific features, the findings can be generalized into other Speech-to-Text applications and development of human-AI interaction. First, our findings provide insights and design inspirations for the development of Speech-to-Text applications such as the voice typing interfaces embedded in modern smartphones, AI-powered transcription tools like Otter.ai, smart note-taking tools for meetings, conferences, etc. The dictation and editing tasks we tested are fundamental for speech-based text input interfaces, and the intelligent text processing features (keyword spotting, text categorization and summarization) we provided in study 2 are being developed in the field of NLP and Knowledge Graph for years (Yoo and Jeong, 2020; Haase et al., 2017). As these technologies get increasingly powerful, they will be used more and more to power real-time interfaces.

Second, although our findings in particular task strategies and system features are specific for dictation-based tasks, the higher-level findings can be generalized for the design of other human-AI interactions. For instance, our findings suggest that other researchers could implement a Multi-Wizard platform for other research contexts, e.g., multimodal interfaces or human-robot interaction, and use it as an ideation and fast iteration tool for designing intelligent system features.

Our identified cooperation strategies between Wizards and their effects on End-User experiences are generalizable to different contexts of use. We found: a sequential Pipeline strategy was easier for the Wizards but hindered End-User experiences; coordination overhead was heavy regardless of whether it was spontaneous or managed by a leader. Future work shall be aware of this trade-off when designing their own platform.

What we learned about clear task distribution in view of temporal pressure highlight potential challenges in other contexts too. Work conflict would also appear in other contexts. The importance of providing transparency of each Wizard’s actions shall be universal. A modular system did provide the flexibility and support to divide tasks and customize interfaces for individual Wizard, therefore shall be effective in other contexts as well.

Last but not least, our findings about the mismatched perceptions on task performance and the need for Wizards and End-User communication and collaboration were also generalizable to other contexts. It was interesting to see the Wizards’ perception of their task performance could be the opposite to what the End-User experiences. This suggests a need for ways to direct Wizards as to set their priorities better aligned to the ones of End-Users. We also found it was important to provide the possibility for End-Users to correct Wizards’ work and give real-time requests. In addition, we found that prior knowledge about the agenda of the end-users could be helpful for Wizards to prepare for the task and pay attention to the important information.

Wizundry is a platform supporting two user groups– Wizards and end-users. To have certain control over confounding factors, we employed researchers as confederates posing as end-users for both studies. The confederates were given pre-formulated scripts to read (Study 1.0) or a topic guide for the interview task (Study 2.0). Now that we have gained understanding on the Wizards’ side of the system by controlling the end-user side, our future work will conduct studies with real participants as end-users.

Our study design from 1.0 to 2.0 were iteratively adjusted. In Study 1.0 we let the Wizards decide their collaborative strategy as in how they divided responsibilities and controls. The observations from this study made us realize a clearer division of roles was beneficial or even necessary for a multi-wizard system due to the temporal demand of the task. Therefore in Study 2.0 we enforced a role division by default, which allowed us to dive deeper into the understanding of how workflow can be managed and their impact on end-user experiences. This 2-step methodology may be used in designing future studies for other cooperative systems like Wizundry.

At last, we shall be aware of the limitations of our approach. Our own researchers are not eligible to evaluate the system as real end-users, thus their reported experiences were not taken as objective evaluation of our interface. However, collecting the end-users’ experience was important for our study. The purpose of having them report their experiences entailed two objectives: 1) to provide end-user feedback to Wizards to test whether they can effectively adapt their strategy and workflow to achieve a better end-user experience; 2) to compare the Wizards’ collective performance across groups, as the researcher’s subjective biases would be consistent across different Wizard groups.

There were tradeoffs for dividing labor: start with clearer divisions in strategies versus without any plans. Wizards who started without a plan tended to have incorrect assumptions regarding the division of labour. Even though everyone’s initial intention was to enhance productivity, the resulting unclear divisions of labor were averse to the group’s ability to work effectively. The efficiency of completing tasks and sharing the workload were considerations for dividing labor. On the other hand, starting with clear division would enable each wizard to define his or her tasks and adhere to the specified execution plan. However, excessive focus on individual tasks may result in a lack of communication and coordination.

Multi-Wizard approaches, regardless of whether there is a defined strategy for assigning tasks to different Wizards, offer significant potential for WoZ contexts, especially involving voice and speech. Wizundry allows for combining modular features (e.g., Feature Selection List in Landing Interface) with independent Wizard work. Such Multi-Wizard configurations can support a single Wizard at the start of a WoZ study, which can then grow into a dyadic or Multi-Wizard setup as needed, based on the division of labor. By evaluating diverse groups of Dyads and Triads, we found that the challenges and the trade-offs were different for each labor allocation method that individual groups came up with and optimization. Researchers can explore the right combination of attendance, system features, and best-performing cooperation strategies with a greater diversity of Wizards and teams. Further, the collaborative space of each Wizard in each context may not be static. We observed that dyads began spontaneously changing strategies when they realized that their task performance was unsatisfactory. This may have been affected by the physical environment of the experiment, including device placement and personal operating preferences. We did not record the individual habits of each Wizard, which future work can consider.

The Multi-Wizard approach enables co-creation and co-learning amongst independent wizards in order to successfully complete the WoZ experiment. This approach provides the opportunity to brainstorm ideas, avoid self-centered perspectives, and expand creativity by drawing on the insights of others. Combining these diverse ideas of wizard members helps create more effective solutions to demanding tasks. By leveraging multiple Wizards’ knowledge, experience and brain power, we have the opportunity to rapidly prototype more powerful systems, such as more intelligent features, multi-modality input, etc. However, this is not the solution to address all the problems. The creativity and openness of advanced forms of human-AI interaction presents new challenges for researchers who wish to conduct WoZ studies. Having Wizards simulate a system still has its inherent limitations, including the relatively long reaction and computation time humans need, and the cost of communication and coordination between multiple of them. Future WoZ methods need to address these challenges through other approaches.

Wizundry presented in this paper is our first attempt to systematically explore possible ways to improve the methodology of WoZ studies by supporting Multiple Wizards in close cooperation. We learned important lessons through the major and minor iterations from Wizundry 1.0 to Wizundry 2.0. While this is by far not the end of the exploration, this work lays the ground by providing a web-based toolkit that is modular and extendable, for future studies in the research community. Based on the feedback we received from our studies, the next version of Wizundry will seek new ways of reducing temporal pressure for Wizards. It appeared to us that the temporal pressure of being a Wizard cannot be alleviated by adding more Wizards. Other approaches will be explored, including better visualization of the end-user and Wizards actions. We will also explore other ways of providing feedback to end-users beyond providing speech boxes.