The EMPATHIC Project: Mid-term Achievements

The Automatic Speech Recognition (ASR) component turns speech from a continuous stream of audio into structured data, containing likely words and their alternatives, labelled with the confidence level for each, start time (as an offset from the stream start), and duration. So far, our main achievements with this component are:

• •

  The development ASR.online, a new method of reading data into the ASR engine which uses the opensource GStreamer framework²²2https://github.com/GStreamer/gstreamer to continuously stream the audio through the ASR executable. Compared with our previous approach of buffering audio and processing small chunks, ASR.online reduces the latency between the speech and the transcription. The average latency has been reduced to approximately 500 milliseconds, as a result of both overall performance improvements as shown in Table 1, and the immediate transmission of transcription data.

• •

  The training of acoustic models for French, Spanish and Norwegian, the languages which will be used in the EMPATHIC field trials. The acoustic models of the ASR component were trained using the Kaldi ASpIRE recipe³³3https://github.com/kaldi-asr/kaldi/tree/master/egs/aspire. The training data consists of 1067 hours of Spanish speech, 271 hours of French and 228 hours of Norwegian, augmented at the DNN-HMM training stage by the addition of noise from RWCP, AIR and Reverb2014 databases using the reverberation algorithm implemented in the Kaldi framework. The training process took approximately two weeks, and used two systems with 32 CPU cores and 64GB RAM each, with a total of 6 NVIDIA Pascal architecture GPUs. 3-gram language models were adapted using transcription data from the training set. These models contain a vocabulary of approximately 67,000 words in the Spanish model, 60,000 words in French and 48,000 in Norwegian. The Norwegian model uses Bokmål, one of two official written standards of Norwegian. The models were tested using the NIST SCLITE utility⁴⁴4https://github.com/usnistgov/SCTK, which scores the best-path output against the ground-truth for correct words, substitutions, deletions, insertions and an overall word error rate. The results are shown in Table 2

The Natural Language Understanding (NLU) component translates the output of the ASR into semantic units to be processed by the Dialog Management (DM) component. The main mid-term achievements in the conception of the NLU component concern the development of two multi-lingual methods for topic classification.

First we had to detect the user’s end-of-turn pauses. Following previous approaches (e.g. (Roddy et al., 2018; Shannon et al., 2017)), we addressed this question as a classification problem. As features for classification, we use the temporal profile of speech, and the syntactic and semantic information encoded in the utterances. As classifiers, we used different variants of deep neural models. A distinguished feature of our approach is that we implemented an ASR simulator that allows us to evaluate the sensitivity of the End-of-Turn-Detection (EOTD) with respect to particular characteristics of the speaker (speech profiles), and to errors in the ASR output. This validation step is essential, since in the implementation of dialogue systems an early mistake in any of the system components can produce a negative cascading effect in the performance of the subsequent elements of the pipeline.

Our NLU component is expected to treat user utterances in three + one different languages, i.e. Spanish, French and Norwegian + English. In the literature, such systems are usually referred to as multi-lingual models, and different strategies have been proposed to develop them. Learning becomes a challenging task for a multi-lingual system since languages are diverse in their grammar. Furthermore, the availability and quality of the corpora with which machine learning models are usually trained is not equal for the languages. Thus, we focused on topic classification. Our approach was to create a modular system where information available in one language is transferred or exploited while learning models for the other languages. We implemented two strategies: the first based on the use of the Wordnet semantic network and synsets (Miller, 1995), the second based on parallel corpora.

Wordnet semantic synsets encapsulate information about different senses of commonly used words. This information is language independent, and can thus be used not only to obtain the equivalent word in another language, but also to obtain a set of sense-connected synsets, by computing the closure set that includes the hyperonymies. Thus, once we have a group of sense-connected synsets, we can generate the set of words that represent them for each language.

Our second strategy focuses on training one model for one particular language for which we have quality labeled data. By means of parallel corpora, we then extrapolate the obtained labels to a second language, so as to later train a specific model. Labeling is a tedious work, yet thanks to this strategy, we only have to label in one language while still obtaining a corpus for each of the languages.

The Dialogue Management (DM) component maintains the state and manages the flow of the conversation or, in other words, determines the action a system has to perform at each step.

For the EMPATHIC VC we used a DM providing an advanced management structure based on distributed software agents. It enforces a clear separation between the domain-dependent and the domain-independent aspects of the dialogue control logic. The domain-specific aspects are defined by a dialogue task specification, and a domain-independent dialogue engine executes the given dialogue task to manage the dialogue. The dialogue task specification is defined by a tree of dialogue agents, where each agent is responsible for managing a sub-part of the dialogue. For instance, Figure 1 shows the high level dialogue task structure for the EMPATHIC VC where the “Introduction” agent is responsible for handling the introductory dialogue of the EMPATHIC VC, the “Nutrition” agent is responsible for handling the nutrition dialogues, etc.

The aim of the system is to have a virtual coach that helps users in certain aspects of their lives. To this end, we have emulated a GROW (Goal - Reality - Obstacle - Will) (Sayas, 2018d) coaching model into the DM’s dialogue strategy. The GROW model is a structured method based on problem solving, goal setting and goal-orientation. The Model is divided into four phases that propose four questions to guide the user towards obtaining and achieving a goal. These questions are asked in a pre-established order. In the first session, this order must be respected to facilitate the user to follow the thread and be able to explore its goal and the necessary steps to achieve it. In the following sessions, the order can be changed or specific phases be chosen. Using the GROW dialogue model, the following dialogue topics have been implemented:

• •

  Introduction dialogues to make the users feel comfortable with the system and to obtain some basic information. This dialogue is carried out the very first time a user dialogues with the system.

• •

  Sport and Leisure (Sayas, 2018a) (Sayas, 2018c) dialogues based on users’ leisure time activities. The aim of these dialogues is to explore users’ leisure time activities, and if necessary, nudge them towards a more active lifestyle.

• •

  Nutrition (Sayas, 2018b) dialogues focused on users’ nutritional habits. The goal of these dialogues is to explore the nutritional routines of the users and, if necessary, try to make the users vary those routines in order to form nutritional habits that are potentially more healthy.

The Natural Language Generation (NLG) component maps the abstract dialogue acts provided by the DM to natural language constructs (in Spanish, French or Norwegian), written in orthographic form. So far, the NLG component has been developed from a reduced database of coaching turns. The data have been extracted from video recordings of real user sessions with a professional coach, and some handmade dialogues created by a professional coach. In the process of labeling, two types of labels have been used: (1) based on the GROW coaching model (Alexander, 2010), and (2) based on linguistic features needed to construct the text. Considering the restrictions in the amount of training data, the NLG currently represents a rule-based system, using a template-based approach (Oh and Rudnicky, 2000). Future work, however, aims to design a new NLG component based on a seq2seq neural network (Dušek and Jurcicek, 2015) (note: this plan of implementing a seq2seq neural model only concerns the NLG part of the EMPATHIC VC and will not affect the other components).

A Text-To-Speech (TTS) component converts any text into a spoken message. For EMPATHIC, TTS speaking styles should be fully compatible with the role of the VC communicating with elderly users. The Acapela TTS system employs a range of internally developed technologies, such as unit selection or parametric synthesizers based on Hidden Markov Models (HMM’s) or Deep Neural Networks (DNN’s). During the project, all of these technologies are adapted for the EMPATHIC communication and compared during evaluation with end-users. Professional speakers will be recorded to capture the communicative role of the VC with the aim to reflect the expressive possibilities of the dialogue system by coherent audio responses to the user’s emotional state so as to support the credibility, naturalness and adaptability of the full dialogue chain. So far, Acapela already recorded a Spanish professional speaker enacting a VC for the elderly and trained a snit selection, and initial DNN synthesis systems on this corpus. We run evaluations of this elderly coach style TTS in terms of naturalness and intelligibility. A Spanish coaching TTS voice is already available and integrated in the mid-term version of the EMPATHIC VC. Upon the evaluation of the Spanish system, Acapela will fine-tune the process and develop the French and Norwegian voices.

For initial prototyping purposes (cf. Section 4.4), we used a multi-step creation process (cf. Figure 2) to build five virtual agent coaches (3 female and 2 male) named Natalie, Alice, Lena, Christian and Adam. The first of the steps depended on the origin of the 3D model. The coaches Alice and Adam were created based on 2D images. For this, we had to create their 3D model from a 2D image using CrazyTalk⁵⁵5https://www.reallusion.com/crazytalk/ and then exported this to the RLhead format. For the coaches Christian, Natalie and Lena, who were created based on 3D models predefined by iClone⁶⁶6https://www.reallusion.com/iclone/, we skipped the CrazyTalk step. Second, we imported the RLhead to the Character creator. This tool helped us create realistic-looking 3D human models. We fixed the 3D model design, imported 3D clothing designs and generated humanoid animations with extensive customization tools. After that, a model was imported into iClone. At this step, we used iClone to blend character creation, animation and scene design into a real-time engine, and to edit them in 3DXchange. Next we exported the model and animation using 3Dxchange into the FBX format. Then, we imported the FBX file as a new resource into Unity3D⁷⁷7https://unity3d.com/fr. The transition from iClone to unity degrades the appearance of the model. To solve this, we had to correct the texture and optimize the shader of the model. Still in this step, we added the lip-synchronization using the SALSA plugin. The audio was processed so as to automate four basic mouth positions, which are the basis for lip-sync approximation. Instead of pre-processing or mapping shapes to audio markers, mouth movements were procedurally applied to a minimal set of mouth shapes to provide variation. It uses a combination of waveform analysis and four mouth shapes to produce high-quality lipsync approximation. The last step was to build the WebGL format within the Unity environment.

As a common definition, we can consider our target population as “Young Olds”, aged 65 to 79 years, with a healthy and active life, characterized by the continuation of their former lifestyle after retirement, yet focused more on enjoyment and leisure activities. It is important to highlight the two main indicators used for this definition. The first indicator concerns the healthy life years at birth; this indicator shows the average of life years sans diseases. The second indicator concerns the healthy life expectancy based on self-perceived health; this indicator shows the population’s self-perception of health. Thus, it include factors such as economic status, emotional problems and social relations. This is a subjective indicator based on surveys, perception records and/or self-assessments⁸⁸8https://ec.europa.eu/eurostat/data/database.

Therefore, the term “Young Olds” refers to people older than 65, who perceive their health as good or very good. Yet although, their self-perception is good, “Young Olds” may already have some type of disease or sickness⁹⁹9Encuesta Europea de Salud en Espana 2017, Pub. Pub. Instituto Nacional de estadística (INE).

Before describing the priorities defined by our analysis, it is worth to highlight four basic recommendations to promote user acceptance of ICT by seniors, defined by the European Active and Assisted Living (AAL) programme. Those recommendations are ¹⁰¹⁰10http://www.aal-europe.eu/wp-content/uploads/2015/02/AALA_Knowledge-Base_YOUSE_online.pdf:

• •

  Provide clear additional value and benefit of the solution;

• •

  Balance between supporting the users and activating them;

• •

  Maintain simplicity on the interaction user-solution;

• •

  Provide joyful experiences;

Those recommendations act as a starting point for integrating priorities into a solution design.

Regarding our analysis, family has been revealed as the main priority across different countries. On average, more than two thirds of seniors with children see them several times per month. This ratio is even higher when children live close by. In addition, the use of ICT tools is significantly higher when it is used for communication purposes with family members (especially children and grandchildren). Therefore, the involvement of family members as part of the solution will strongly increase the acceptance and usability of the solution.

Another key point to promote the ICT use among elderly is to show the clear benefit of a solution. For example, seniors are willing to learn to use ICT tools if this provides more interaction with their younger family members.

As mentioned before, more than half of our target population perceives their health as “good” or “very good”, although a common fear is the physical decline associate to ageing. This could be one strong reason why, independent of the country, more than half of the people we talked to perform some type of physical activity on a weekly basis (note: most commonly walking). Supporting those activities, providing motivational strategies, gamification tools and/or professional advice may thus be seen an opportunity to increase the acceptability of ICT. Even better, it would not only support and enrich a leisure activity that is already common among the target population, but also promote “healthy habits” and “well-being”.

Following a similar approach, nutrition is an area that combines “joyful” experiences and “healthy habits”. Cooking, planning meals, going out to restaurants, increasing expenditure on food etc. are related activities that are more promoted with representatives of the target population than with younger people. In that sense, changes in the nutrition habits can be seen as an indicator for physical or physiological changes. On the other hand, promoting and motivating a healthier nutrition habit can positively influence a human’s physical and emotional status.

While motivational factors and physical as well nutritional habits were similar with our target populations in Spain, France and Norway, we did also find differences. Those mainly relate to the familiarity and use of the Internet. Here it was shown that Seniors from Norway have a significantly higher percentage of people that use the Internet on a regular basis, in particularly when compared to Spain (note: France lies in between). This information is relevant, as personal experience with technology is a relevant factor influencing technology acceptance.

One aspect to focus on in human-agent interaction is the user’s level of technology acceptance. The concept was introduced by Davis (Davis, 1989) in the attempt to explain people’s acceptance (or not acceptance) of an interactive system. It led to the development of the Technology Acceptance Model (TAM), a questionnaire where acceptance is assessed in terms of a user’s perceived usefulness, and perceived ease of use (Davis, 1989) of the system. TAM was extended into TAM2 in 2000 (Venkatesh and Davis, 2000), adding two theoretical constructs that accounted for a user’s social influence and for how well a user’s work goals are supported by the interactive system. TAM2 evolved into the Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh et al., 2003) and later into UTAUT2, where hedonic motivations (the fun or pleasure derived from using a technology), price values (trade-off between perceived benefits and monetary costs), and habits were added to the original questionnaire, as further determinants theorized to affect user’s behavioral intentions and use behavior (Venkatesh et al., 2012). Finally, the Almere questionnaire was developed as a further evolvement of UTAUT2 objecting that the latter was developed without accounting for variables that relate to social interaction with robots or virtual agents and without considering seniors as potential users (Tsiourti et al., 2014; Heerink et al., 2010). Following the same reasoning, Hassenzahl developed the AttrakDiff questionnaire¹¹¹¹11AttrakDiff(tm)Internet Resource – http://www.attrakdiff.de. (Hassenzahl, 2018, 2004), a four cluster test, where each cluster, composed of 7 items, was assessing a desired user requirement.

It must be noted that, currently, all the theoretical formulations of questionnaires aiming to assess the user experience of an interactive system are, to a certain extent, dated, since those systems are increasingly more complex, showing humanoid appearance, and human features. Thus, even though the theory for defining user experience is still valid, new concepts have to be accounted for, in order to have a fair assessment of modern interactive systems. This is why, to date, there are no systematic investigations devoted to assessing the role of virtual agents’ features exploiting the above mentioned questionnaires. Furthermore, seniors have only been involved in a very limited number of studies on virtual agent’s. In the few they have been involved in, it has been shown that they clearly enjoy interacting with a speaking synthetic voice produced by a static female agent (note: these were 65+ aged seniors in good health (Cordasco et al., 2014)), and that such seniors are less enthusiastic than impaired people in recognizing the agent’s usefulness (Yaghoubzadeh et al., 2013). The only comparison among user’s age we know about, was conducted by Straßmann & Krämer (Straßmann and Krämer, 2017). The study was “a qualitative interview study with five seniors and six students” and showed that senior users prefer embodied human like agents over machine or animal-like ones. No information were obtained on the gender of the agents as well as their pragmatic and hedonic features, as advocated by the TAM, UTAUT, and AttrakDiff questionnaires discussed above.

Therefore, the Empathic project aims to “develop causal models of [agent] coach-user interactional exchanges that engage elders in emotionally believable interactions keeping off loneliness, sustaining health status, enhancing quality of life and simplifying access to future telecare services”. To this end, an initial research step was the development of an ad-hoc questionnaire to assess the pragmatic and hedonic features of the to be developed EMPATHIC VC. The questionnaire was developed through an iterative process that involved several experiments and the exploitation of the theoretical concepts already advocated by the authors of TAM 2, UTAUT2, and AttrakDiff, and the inclusion of new theoretical considerations regarding our users’ age. The goal of the questionnaire was also to provide information on the user’s preferences regarding the agent’s physical and social features, including its face, voice, hairdo, age, gender, eyes, dressing mode, attractiveness, and personality.

The first experiment with the aim to build and evaluate such a questionnaire (and start collecting said data) was conducted in Italy and involved 45 healthy seniors (50% female), aged 65+ (Esposito et al., 2018b). As this study was conducted before any of the agents described in Section 2.6 were built, our stimuli were based upon the four conversational agents proposed by the Semaine project¹²¹²12http://www.semaine-project.eu/, each possessing different personality features able to arise user specific emotional states i.e., Poppy (female, expressing optimism), Obadiah (male, expressing pessimism), Spike (male, expressing aggression) and Prudence (female, expressing a high degree of pragmatism) (Ochs et al., 2010). For each agent, a video-clip was extracted from the videos available on the Semaine website. In order to contextualize them to the Italian culture they were renamed, using names very popular in the local area (i.e. Serena, Gerado, Pasquale, and Francesca). Agent’s names and video durations were carefully assessed by four people. The final set of stimuli consisted of 4 video clips, each 10 secs long showing the agent’s half torso, all of the same dimensions, acting as if they were speaking while the audio was mute.

The preferences the target group had towards each of the proposed agents were assessed through a first version of our questionnaire, structured in 4 clusters each containing 7 items devoted to assess the practicality, pleasure, feelings, and attractiveness experienced by participants while watching the agent video-clips. The items proposed in each cluster exploited the theoretical foundation inherent to the UTAUT2, Almere, and AttrakDiff questionnaires. Results showed seniors’ positive tendency to initiate an interaction with the agent, with a strong preference towards agents with a positive personality. That is, Francesca and Serena scored always significantly higher than Pasquale and Gerardo for the pragmatic, hedonic and attractiveness features. Although seniors had not been informed of the agents’ personality, somehow they had perceived negativity or positivity from the dynamics of their facial expressions, which triggered a behavior of acceptance/rejection, suggesting that participants have preferences for positive facial dynamics. These results, however, required deeper investigations as all agents showing positive facial dynamics were females, hinting towards a potential gender influence on the processing of emotional facial expressions (cf. (Marsh et al., 2005; Rotteveel and Phaf, 2004)). Furthermore it must be noted that in this first study agents were dynamically moving their lips as if they would be speaking. Their voice was, however, muted which might have had an effect on people’s perceptions.

These aspects were accounted for in a subsequent experiment, conducted again in Italy (Esposito et al., 2018a), but this time with agents created with BOTLIBRE¹³¹³13http://www.botlibre.com. Also these agents, which were selected by 3 experts, showed half their torso, with definite clothing. Again, so as to contextualize, we named them in accordance with typical names for the region, i.e. Michele, Edoardo, Giulia and Clara. Each agent was provided with a different synthetic voice, producing the following sentence pronounced in Italian: “Hi, my name is Clara / Edoardo / Giulia / Michele. If you want, I would like to assist you with your daily activities!”. The synthetic voice was created through the website Natural Reader¹⁴¹⁴14http://www.naturalreaders.com. The voices (recorded using the software Audacity¹⁵¹⁵15https://www.audacityteam.org/) were embedded into each agent’s video-clip which had an average duration of approx. 6 seconds. The proposed agents did not show a particular personality. Care was taken to ensure that no emotion was depicted by their faces and they were shown on the same background, avoiding exaggerated cloths colours or facial features. This second experiment, again devoted to assessing senior’s preferences towards each of the proposed agents, exploited a new version of our questionnaire. It had 6 clusters:

• •

  Cluster 1: 10 items focusing on the usefulness, usability, and accomplishment of the tasks of the proposed system, i.e. the system’s pragmatic qualities (PQ). High scores in the PQ dimensions indicate that the users perceive the systems as well structured, clear, controllable, efficient and practical.

• •

  Cluster 2: 10 items focusing on motivations, i.e. the reason why a user should own and use such an interactive system, i.e., the system’s stimulating hedonic qualities (HQS). A system receiving high scores in the HQS dimensions is meant to be original, creative, captivating.

• •

  Cluster 3: 10 items focusing on how captivating, and, of good taste the system appears, i.e. the system’s hedonic qualities of feeling (HQF). A system receiving high scores in the HQF dimensions, is considered presentable, professional, of good taste, and bringing users close to each other.

• •

  Cluster 4: 10 items on the subjective perception of the system’s attractiveness (ATT), which is the hedonic dimension that gives rise to behaviors as increased use, or dissent, as well as, emotions as happiness, engagement, or frustration.

• •

  Cluster 5: 4 items assessing the type of professions seniors would endorse to the proposed agents, among which were welfare, housework, security, and front desk jobs.

• •

  Cluster 6: 3 items assessing agent’s age range preferences.

Each questionnaire item required a response given on a 5-point Likert scale ranging from 1=strongly agree to 5=strongly disagree (3=I don’t know). Since all clusters contained positive and negative items evaluated on a 5-point Likert scale, scores from negative items were corrected in a reverse way. This implies that low scores summon to positive evaluations, whereas high scores to negative ones. Each participant was first asked to provide answers to the items related to demographic information and user technology savviness, then they were asked to watch each agent’s video-clip and immediately after to complete the items from the remaining 6 clusters.

Various modules of the EMPATHIC VC will require ongoing development and improvement, e.g. to advance speech recognition and language understanding, or foster user experience and acceptance. In order to simulate such modules while they are being developed, the goal was to build and consequently use a simulated Wizard of Oz (WOZ) component. WOZ constitutes a prototyping method that uses a human operator (i.e., the so-called wizard) to simulate non- or only partly-existing system functions. In language-based interaction scenarios, like the ones envisioned by EMPATHIC, WOZ is usually used to explore user responses and the consequent handling of the dialogue, to test different dialogue strategies or simply to collect language resources (i.e., corpora) needed to train technology components. In EMPATHIC, however, the goal is to use WOZ beyond this traditional prototyping stage, and make it a fallback safety net for situations in which the automated coach may be unable to respond. That is, the goal is to develop a system component, which initially serves as a prototyping tool supporting the research on language-based interaction and dialogue policies, but then becomes an always-on backup channel dealing with those user requests the system is incapable of handling by itself. A first version of this tool has been built and consequently used in several user studies.

A total of 176 WOZ user studies (à 2 sessions) have so far been conducted (i.e., 68 in Spanish, 54 in French, and 54 in Norwegian). The following insights are based on feedback provided by wizards, who simulated the EMPATHIC VC, as well as study participants.