Engineering Reliable Interactions
in the Reality-Artificiality Continuum

According to the Merriam-Webster Dictionary, interaction is the “mutual or reciprocal action or influence” [45]; its definition covers both social interaction carried out by humans and socially-capable entities via communication, and physical interaction among physical entities and/or humans.

The reality-virtuality continuum was introduced by Milgram in 1994 [46] and applies to interaction in the physical space dimension, going from real to virtual. Given that the word interaction is also strongly characterized by a social dimension, the reality-virtuality continuum paradigm should be considered inside a bidimensional space, as shown in Figure 1, rather than inside a unidimensional one. The physical dimension, drawn on the horizontal axis, involves interaction with an environment, be it real, augmented or virtual. The social dimension, associated with the vertical axis, involves communicative interaction via natural language with companions, be them human or artificial.

A virtual environment is composed of virtual elements, only; an augmented reality environment is the combination of both virtual and real objects. Though the virtual elements should be consistent with the real world surrounding the user for both augmented and virtual scenes, the perceptual issues and the alignment constraints are different. Indeed, in immersive virtual reality the environment is coherent since the whole environment is virtual and only an interaction coherence is required. Whereas in augmented environments the visual alignment issues can rise among virtual and real parts of the scene, even for small spatial errors. While Milgram’s reality-virtuality continuum in the physical dimension is commonly accepted, we are not aware of an analogous “social interaction reality-artificiality continuum” paradigm, as depicted on the vertical axis. In our vision, social interaction goes from real to artificial, the latter involving humans and artificial companions, further classified as disembodied, like chatbots and software agents, and embodied, like robots and other autonomous systems equipped with a hardware structure. The various forms of social interaction can be combined with the various forms of physical interaction, leading to the identification of different areas in Figure 1. Although the best interaction style for most people is in the real physical space, and with real persons (the RR bottom-left area in Figure 1), sometimes this interaction is not possible. Before the Covid-19 pandemia, the impossibility to interact in the RR area was mainly associated with persons living in rural and alpine areas not well connected with villages nearby, and with hospitalized or disabled people, namely with fragile categories at risk of social exclusion. Covid-19 made instead clear that billions of humans can be suddenly prevented from moving in the RR area for a long time. Today, frailty of interactions is a global condition, and our well-being is dramatically affected by their lack: social distancing prevents interaction and the effects of prolonged isolation may worsen or trigger mental health problems¹¹1https://www.sciencenews.org/article/coronavirus-covid-19-social-distancing-psychological-fallout,
https://www.nationalgeographic.com/history/2020/04/psychologists-watching-coronavirus-social-distancing-coping/,
https://www.forbes.com/sites/traversmark/2020/04/17/social-distancing-is-harming-mental-health-more-than-physical-health-according-to-new-survey-data/..

In our futuristic vision, interaction will be possible even when no real environment, and no real companions, are available: the real or artificial companion may be immersed in a real, augmented, or virtual environment, and the human user may interact with it in natural language. Virtual Reality (VR) and Augmented Reality (AR) may be the media through which users interact with their companions, to overcome isolation when the RR area is not accessible. To make our vision feasible, the results already achieved in integrating robotics & VR/AR should be merged with the results in the chatbots & VR/AR field, and should be further extended to cope with the peculiar issues of both areas and with the challenges raised by their merge. We name the logical vision resulting from this merge, that we depict in Figure 1, RAC, for Reality-Artificiality Continuum.

In this position paper, we overview some of the challenges raised by the design and development of the RAC vision. In particular, we discuss some techniques suitable to design and implement reliable interactions in RAC, with special attention to applications related with well-being. We borrow the definition of reliability by [22]: “ For systems offering a service or function, reliability means that the service or function is available when needed. A software system is reliable to the extent that it meets its requirements consistently, namely that it makes good decisions in all situations. Among the requirements we envisage for a system implementing the RAC vision, we mention

The above requirements may be further specialized for the different areas of the bidimensional RAC space. From one hand the reliability of social interactions can be preserved in immersive VR since all the agents are coherent among them since all are virtual. Whereas in augmented environments users can feel differences among virtual and real agents. On the other hand, immersive VR may not be well accepted by some people with special needs, since VR creates a completely alternative reality.

Despite these differences between the different types of interaction, which create further specific challenges, the four requirements above are the most important for our RAC vision, and should always be met. Designing and implementing a system providing (some of) the interaction functionalities sketched in Figure 1 in such a way that it meets the four requirements above is what we mean by “engineering reliable interactions in the Reality-Artificiality Continuum”.

In the last 10 years, the interest of the research community towards social inclusion in general, and inclusion of fragile people in particular, has been growing year after year. Social inclusion was one of the eleven priorities for Cohesion Policy in 2014-2020 (“thematic objective 9”²²2https://ec.europa.eu/regional_policy/en/policy/how/priorities.) and several European Community research programs have funded, or are going to fund in the near future, projects aiming to address these needs³³3http://www.europeannetforinclusion.org/,
http://www.interreg-alcotra.eu/it/decouvrir-alcotra/les-projets-finances/pro-sol,
https://eacea.ec.europa.eu/erasmus-plus/news/social-inclusion-and-common-values-the-
contribution-in-the-field-of-education-and-training-2019_en.. At the end of April 2020, a third of the world population was on some form of a coronavirus lockdown, meaning their movements are restricted and controlled by their respective governments⁴⁴4https://www.statista.com/chart/21240/enforced-covid-19-lockdowns-by-people-affected-per-country/: many of us became fragile, and the need for social inclusion is now affecting at least 2.5 billions people. One way to achieve social inclusion is by boosting interaction.

The starting point of our vision is hence the Milgram’s reality-virtuality continuum [46], or Mixed Reality (MR) continuum, a commonly accepted paradigm, which describes Virtual and Augmented Reality (VR/AR) applications like a unitary approach, rather than distinct concepts, and which is depicted in the RR, AR, VR areas in Figure 1. The MR continuum is a scale between completely real environments with no computer enhancements, towards virtual environments that are totally generated by the computer. In between these two extremes are Augmented Reality and Augmented Virtuality (AV), which are a combination of real and virtual worlds. Similarly, the devices could go from non-immersive and standard visualization devices, e.g. monitors or tablets, to highly immersive systems, like the VR head-mounted displays, typically used for VR applications, but also for wearable AR [33].

So far, the most critical issues when engineering VR/AR applications involve their perceptual issues, undesired effects, and effective use in contexts different from the entertainment: these are open and largely discussed problem, e.g. addressing the wrong estimation of distances [12, 27, 30, 39, 40, 49]. This negatively affects interaction, for example in industrial or medical training tasks, but how to interact in a natural way in such environments is also important for their acceptance, especially by fragile people [6]. Indeed, immersion, presence and interactivity are all factors that influence VR acceptance and use [14, 47]. Some studies analyze acceptance of VR for specific classes of subjects, e.g. for cognitive deficits caused by diverse disorders or traumatic brain injury [15], with positive results both in terms of patients’ involvement and positive effects on their rehabilitation, also considering immersive VR head-mounted displays, or for e-Health systems for the elderly [9, 13]. The problem of acceptance of VR/AR has recently been studied also in education [16] and tourism [37].

Moving towards the acceptance of disembodied artificial companions in the real physical environment (RD area in Figure 1), despite the longstanding history of studies on the connections between sentiments, emotions and interaction in natural language with conversational agents and chatbots, with ad-hoc task also in the most recent SemEval competition [53], studies dealing with adoption and acceptance of chatbots by humans are very recent [50], especially when chatbots are used in the healthcare domain [34]. One requirement for a conversational agent to be accepted by its user is its capability to adapt to her/him. The literature on conversation adaptation is also very recent and almost limited [28, 31]. Apart from Amazon Sumerian⁵⁵5https://aws.amazon.com/sumerian/., it seems that no frameworks for integrating chatbots into VR/AR applications (AD and VD areas in Figure 1) exist, which makes the RAC vision extremely novel and original. Sumerian was launched in 2018 [21] and some authors criticize it for its lack of transparency and its questionable data policy, concluding that it is inappropriate for health-related applications [42]. Moreover, the use of applications developed through Amazon Sumerian implies a payment, which contrasts with the first RAC requirement.

As far as the AE and VE areas in Figure 1 are concerned, many solid proposals to integrate robotics and VR/AR exist, but these proposals do not take human-robot natural language interaction into account. Seminal works on robotics and VR/AR integration date back to the turn of the millennium [11, 24] and recent scientific proposals, some of which near-market, span many different domains. Training is among the most lively ones: in September 2019, Guerin and Hager patented a system and methods to create an immersive virtual environment using a VR system that receives parameters corresponding to a real-world robot that may be simulated to create a virtual robot based on the received parameters. Users may train the virtual robot, and the real-world robot may be programmed based on the virtual robot training⁶⁶6https://patentimages.storage.googleapis.com/e4/81/62/ca61f9719139a0/US20190283248A1.pdf.. Just one month later the Toyota Research Institute released a video showing how they use VR to allow human trainers to teach robots in-home tasks⁷⁷7https://youtu.be/6IGCIjp2bn4.. The just published paper by Makhataeva and Varol [41] surveys about 100 AR applications in medical robotics, robot control and planning, human-robot interaction and robot swarms from 2015 and 2019. Recent experiments demonstrate the value in using modern VR/AR technologies to mediate human-robot interactions [52] and boost robots acceptance by their users.

Finally, we mention the very active field of natural language human-robot interaction, positioned in the top-left RE area in Figure 1. Robots must understand the natural language, and robotic systems capable of executing complex commands issued in natural language that ground into robotic percepts have been designed and implemented [26], grammar based approaches to human-robot interaction have been experimented and compared with data driven ones [7], algorithms targetted to recognize motion verbs such as “follow”, “meet”, “avoid” have been proposed [32]. However, to be believable and pleasant companions, robots should also generate natural language. The reasons why so few current social robots make use of sophisticated generation techniques are discussed by Foster in her paper on “natural language generation for social robotics: opportunities and challenges” [23].

The analysis of the state of the art shows that many building blocks that are needed to implement the vision put forward in Figure 1 are already available to the research community, at least as working prototypes. Nevertheless, the RAC building is still to come.

We believe that many years will pass by before we will see antropomorphic robots interacting with humans in natural language inside a virtual reality (VE area in Figure 1, the most challenging one). However, some approaches may boost the integrated RAC vision take off, and allow its developers to engineer interactions that are reliable. In this section we review some of them.

The different combinations of physical and social interaction raise different research challenges and offer different opportunities for the users. As an example, a pure virtual experience could be well accepted by a young user, but it might not be applied to elderly, who might more benefit from a communication-intensive, voice-based interaction where words are carefully selected from a vocabulary of words that are familiar to that specific person, and that (s)he can easily recognize and understand; for this kind of person, interaction with the augmented reality should not be an immersive one.

To engineer reliable interactions in RAC, users should be profiled, the real environment where they operate should be recognized and abstracted into a model, and interactions should dynamically adapt to their profile and real environment, which may change over time, and at the same time should follow some patterns, or protocols, that are “safe by design”.

We disregard the profiling issue, already well covered by the literature [17, 51], and we discuss how reliability of interaction could be addressed by exploiting ontologies and runtime verification.

Once the objects in the real environment are detected and the corresponding bounding box created, the virtual scenes should be composed on-the-fly⁸⁸8The bounding box is the volume that fully encloses the real object. A virtual scene is a digital counterpart of a real scene, where the objects are virtual and not real., and in a (semi-)automatic way, by inserting objects of the same shape, position and spatial properties of the corresponding real ones. These virtual objects should be semantically consistent with the virtual scene. The scene is composed on-the-fly in order to handle objects that can move in the real scene and it is necessary an automatic procedure to carry out this process. A human operator can contribute to the creation of a virtual scene by working on the semantic consistency of the objects and on the kind of environment that the operator wants to implement. The whole scene can be built off-line to decrease the computational cost and to handle on-the-fly only the objects that can change their position in the real scene.

For example, if the virtual scene represents a bush, and a real object has been detected with a parallelepipedal bounding box starting from the ground and 70cm tall, it might be substituted, in the virtual scene, by a shrub. If the height is 1.70cm, it might be substituted by a tree. If the virtual scene represents a shore, those two objects might be substituted by a chair and a beach umbrella, respectively, and if it is a urban environment, they might be a trash bin and a pole with a road sign.

In order to associate virtual objects with real objects in a semantically consistent way, we need to represent the knowledge about which virtual objects make sense into a virtual scene, also depending on their geometric features, via one or more ontologies. To the best of our knowledge, the connections between VR/AR and ontologies have been explored in a very limited way, and the idea of exploiting ontologies in VR/AR to make virtual and real objects consistent is an original one. The most similar proposal dates back to 2011 and applies to 3D virtual worlds; it aims at labelling some things with tags like “chair” and “table” and associate functional specifications to enable computers to reason about them, but no real-to-virtual semantic alignment is foreseen [18].

Once ontologies enter the RAC, they can also serve as a means to ensure semantic interoperability among different users and to ease natural language interaction. This would be a more standard way to use ontologies in RAC, since the relationships between natural language processing and ontologies have been studied since the advent of the semantic web [43, 44, 56].

The exploitation of ontologies makes both physical and social interaction reliable because, as observed by Uschold and Gruninger more than 20 years ago, “a formal representation makes possible the automation of consistency checking resulting in more reliable software” [55].

As an example, by writing disjunction axioms in the ontology it would be possible to formally ensure that a physical object is never rendered with a virtual appearance, if that virtual look may hurt a user with some given profile. In a similar way, ontologies might ensure that the words selected by the artificial companion when generating sentences are compliant with the profile of the user, and that they are properly disambiguated when uttered by the user. To make an example, a user with a young profile might use “paw” as a slang acronym for “Parents Are Watching” while a cat lady would use it with its most common meaning. By avoiding misunderstandings and choices of words or virtual shapes unsuitable for some categories of users, reliability of interactions would increase.

Runtime verification is a formal method aimed at checking the current run of the system. It takes as input a data stream of time-stamped sensor or software values and a requirement to verify, expressed in the form of a temporal logic formula, most commonly Linear Time Temporal Logic [8], Mission-time Linear Temporal Logic (MLTL) [38], First-Order Linear Temporal Logic [5], or via trace-based formalisms like Trace Expressions [3]; the output is a stream of $\left<\mathrm{time},\mathrm{verdict}\right>$ tuples containing a time stamp and the verdict from the valuation of the requirement (true, false, or unknown when a three valued logic is adopted), evaluated over the data stream starting from that time step.

The runtime verification formalism and engine we have been developing in the last seven years, Trace Expressions, has proven useful both to monitor agent interaction protocols (AIP, [19, 48]) taking place in purely software multiagent systems [4, 20], also when self adaptation of agents is a requirement [2], and to verify event patterns taking place in physical environments like the Internet of Things [35, 36]. The Runtime Monitoring Language, RML⁹⁹9https://rmlatdibris.github.io/, is a synthesis of these years of research: it decouples monitoring from instrumentation by allowing users to write specifications and to synthesize monitors from them, independently of the system under scrutiny and its instrumentation.

RML would be suitable to monitor physical interactions in any portion of the RAC bidimentional space. To make an example inspired by the current lockdown experience, the number of human users in the same room at the same time should be no more than 2, they should keep a 1m distance, and no user could stay in the room for more than 45 minutes. If users were equipped with sensors suitable for indoor localization, RML could monitor the balance of “user X enters the room at time T1” and “user X leaves the room at time T2” events and check that the number of users in the room and the time spent there meet the lockdown rules. A further continuous check that if two users are in the room, they keep the mandatory distance, could also be easily implemented.

What would be definitely harder, but worth exploring, is the runtime monitoring of natural language interactions. Let us consider the following situation: Alice is interacting with her artificial companion, which is aware that Alice profile is “very polite”. The artificial companion expects that after a greeting, Alice answers with a greeting, and after the accomplishment of some task, Alice thanks. There are many ways Alice can greet her companion or express her gratitude, so there are many different concrete utterances that should be tagged ad “greeting” or “gratitude”. It would be up to some sophisticated natural language processing component to tag utterances in the right way. Assuming that utterances are managed as events, each tagged with its event type ($greets$, $thanks$), the interaction protocol with very polite users might require that ${robot}\stackrel{{\scriptstyle{greets}}}{{\Longrightarrow}}{alice}~{}{:}~{}{alice}\stackrel{{\scriptstyle{greets}}}{{\Longrightarrow}}{robot}~{}{:}~{}\epsilon$ and ${robot}\stackrel{{\scriptstyle{task\_completed}}}{{\Longrightarrow}}{alice}~{}{:}~{}{alice}\stackrel{{\scriptstyle{thanks}}}{{\Longrightarrow}}{robot}~{}{:}~{}\epsilon$ where ${A1}\stackrel{{\scriptstyle{Tag}}}{{\Longrightarrow}}{A2}$ means that $A1$ uttered a sentence tagged as $Tag$ to $A2$, and ${:}$ is the prefix operator meaning that after one event, another must take place. $\epsilon$ represents the end of the interaction.

Being able to correctly tag utterances on the fly would allow RML to detect violations of the protocol. For example, Alice might not thank her companion for some task it carried out. While this may be due to any reason, not necessarily a serious one, it might also be a signal of something wrong in Alice’ mood, and might require some special action to take, or some ad hoc sub-protocol to be triggered.

The idea that some sentences can be collapsed into the same category, and hence tagged with the same “event type”, is already supported by many languages for building chatbots. As an example AIML, the Artificial Intelligence Markup Language¹⁰¹⁰10http://www.aiml.foundation/doc.html, provides the srai mechanism for this purpose, and IBM Watson Assistant¹¹¹¹11https://www.ibm.com/cloud/watson-assistant/ achieves the same goal via the notion of “intent” exemplified by a list of sentences.

At the state of the art, runtime verification of natural language interactions is out of reach. However, by integrating existing approaches to chatbot development and RML we might be able to move a step forward in this direction.

As a final remark, we observe that although the RAC integrated vision is still far to come, we believe that it represents a promising direction towards the satisfaction of anywhere-anytime social inclusion needs. Many ethical, social, psychological and medical aspects should be addressed if the RAC became available, besides the technical ones. As part of our close future work, we are going to address some of the latter: on the one hand, we will work on the alignment of virtual and real objects via ontologies. On the other, we will explore the possibility to integrate runtime verification mechanisms in human-chatbot interactions.