Spoken Dialogue System Based on Attribute Vector
for Travel Agent Robot*

The basic idea of this system is to fully understand the user intention and recommend a sightseeing spot with features that match the user intention. To achieve this, an “attribute vector” representing the characteristics of each spot was defined in advance. Next, the user intentions were ascertained through dialogue and expressed as a “user vector.” Finally, it was compared with the “attribute vectors” of the candidate spots to make a recommendation.

The details of the algorithm are as follows.

1. 1.

   The characteristics for each sightseeing spot are extract from sightseeing information¹¹1It consists of two files, SightBasic.csv and SightOption.csv. provided by the convention organizer and create an “attribute vector.”

2. 2.

   The “attribute vectors” of the two candidate spots are compared, and the user intentions are questioned about the attributes with different values. The answers are analyzed to create a “user vector” that represents the user intentions.

3. 3.

   The obtained “user vector” is compared with two “attribute vectors” of sightseeing spots, and the recommended spot is communicated to the user with reasons based on the result.

The algorithm can make recommendations by focusing on the differences between the two candidate spots, and a highly satisfactory dialogue can be achieved by correctly reflecting the user intentions.

When making a recommendation in this manner, the characteristics of sightseeing spots must be extracted and expressed in an “attribute vector.” Sightseeing spots can have a variety of characteristics with a great deal of variation. To collect such information, manually collect information on each spot in advance and accumulating it in the database would be effective. However, this competition had a rule that stated: “the system should be able to work as soon as a list of sightseeing spots is given,” and collecting information manually is not allowed.

Although the competition allowed crawling various web pages and collecting information automatically, we decided to generate “attribute vectors” only from the information provided by the convention organizers.

Attributes used in this system are as follows.

• •

  Type of sightseeing spot
  Four attributes: Art museum; Park; Museum; Observatory

• •

  Facilities
  Three attributes: Whether admission is charged or free; Whether parking is available; Whether it can be enjoyed even in the rain.

• •

  Type of recommended customers
  Five attributes: Children; Ladies; Babies; Alone; Customers with pets.

• •

  Recommended season
  Four attributes: Spring; Summer; Autumn; Winter.

A program was created to automatically extract the attribute values from the information. The attribute values were three: applicable (“yes”), not applicable (“no”), and no information (“do not care”). For attributes related to the “type of sightseeing spot,” “yes” was given to only one of the four types, and “no” was given to the rest. For the attributes related to “facilities,” “yes” or “no” were selected for each. For the attributes related to “type of recommended customers” and “recommended season,” positive information, such as “recommended for children,” was provided, whereas “not recommended for children” was not provided. Therefore, “yes” or “do not care” was set.

The purpose of this dialogue was to determine which of the two sightseeing spots the user would prefer to visit. Therefore, we found the differences in the attributes of the two spots and asked the user which attribute value they preferred in these attributes. The “user vector” was created by the answers to represent the user intentions.

The details of the algorithm are as follows.

• •

  Advance preparation

  1. 1.

     For all attributes used in “attribute vector,” define a list of questions to be asked if there is a difference in the value of the attribute. For example, if the attribute is about the “type of sightseeing spot,” define questions such as “Do you like art museums?” For an attribute related to the “type of recommended customers,” define questions such as “Will you go with your children?”

  2. 2.

     For each question, prepare a set of keywords and the update rules to determine which attribute values of “user vector” should be updated if and when a keyword appears in the user’s response. For a single question, assume various responses and construct a list of as many keywords as possible that may appear in the responses. For example, for the question “Will you go with your children?,” if the answer is “yes,” the attribute “recommended for children” is set to “yes,” the other attributes related to the “type of recommended customers” are set to “no,” and all other attributes are set to “do not care.”

     Furthermore, for each of the possible answers from the user, an appropriate response sentence is defined, such as “You are with your child; therefore, we would like to recommend a place for your child.”

• •

  In the dialogue

  1. 1.

     Before starting the dialogue, all attribute values in the “user vector” are set to “do not care.”

  2. 2.

     Compare two “attribute vectors” of the selected sightseeing spots and extract all attributes with different attribute values. Ask questions for each extracted attribute.

  3. 3.

     Extract keywords from the user’s answer and update attribute values in the “user vector” based on the update rules. In this step;

     • –

       If the update rule is “yes,” update the attribute value to “yes.”

     • –

       If the update rule is “no,” update the attribute value to “no” only if the value was “do not care” before the update. If the value was “yes” before updating, then ignore it.

     • –

       If the update rule is “do not care,” ignore it.

The algorithm realizes that the user intentions can be acquired by focusing on the difference between two selected sightseeing spots to make appropriate recommendations.

After the “user vector” is obtained, it is used to recommend the sightseeing spot that matches the user intentions. However, in this competition, the sightseeing spot that the travel agency wants to recommend is defined separately from the user intention and one of the objectives was to induce the user to choose this spot through dialogue.

To achieve this objective, the following three attribute sets were calculated and the recommendation method was changed according to the number of elements. In the following explanation, let $V_{r}$ be the “attribute vector” of the spot recommended by the travel agency, $V_{n}$ the vector of the other spot, and $V_{u}$ the “user vector.”

• •

  $M(V_{r},V_{u})$
  A set of attributes that have the attribute value “yes” in both $V_{r}$ and $V_{u}$. The more these attributes are, the more the user intention matches that of the spot recommended by the travel agency.

• •

  $U(V_{r},V_{u})$
  A set of attributes whose value in $V_{r}$ is “yes” and those in $V_{u}$ is “no.” The more these attributes are, the less the user intention matches that of the recommended spot.

• •

  $U(V_{n},V_{u})$
  A set of attributes whose value in $V_{n}$ is “yes” and those in $V_{u}$ is “no.” The more these attributes are, the less the user intention matches that of the other spot.

Depending on the number of attributes in these three sets (the number of attributes in set A is denoted by $|A|$), the following recommendations are made.

• •

  $|M(V_{r},V_{u})|\geq|U(V_{r},V_{u})|$
  In this case, because the number of attributes that match the user intention is larger than the number of attributes that do not match the user intention, a sightseeing spot recommended by the travel agency is recommended.

  Hence, the attributes included in $M(V_{r},V_{u})$ are listed as reasons. Furthermore, the attributes included in $U(V_{n},V_{u})$ are provided to explain that other spot does not match the user intention and guide not to select it.

• •

  $|M(V_{r},V_{u})|<|U(V_{r},V_{u})|$
  In this case, the recommended spot does not match the user intention. First, the attributes included in $U(V_{r},V_{u})$ are provided to describe that the recommended spot does not match the user intention, and then explain preferable points in the recommended spot and unpreferable points in the other spot based on $M(V_{r},V_{u})$ and $U(V_{n},V_{u})$. Finally, the robot recommends the recommended spot to the user.

• •

  $M(V_{r},V_{u})=U(V_{r},V_{u})=\phi$
  In this case, the user did not answer “yes” or “no” to the attributes of the recommended spot. Because the user intention is unknown, the user is encouraged to select the recommended spot by explaining the characteristics of the spot, such as “Generally recommended for children.”

The algorithm can guide the user to select the sightseeing spot recommended by the travel agency while reflecting the user intention.

The results of each user utterance were analyzed. A total of 483 user utterances were observed in the entire dialogue, of which 174 (36.0%) were correctly understood by the system and responded as expected.

For the remaining user utterances, the causes of incorrect responses were classified. The results are listed in Table I. Note that some of the utterances were counted more than once because they were related to more than one cause.

The data presented in Table I reveals that 139 utterances (28.8%) were voice activity detection (VAD) errors in the speech recognition system. In VADs, even if the user uttered an utterance, it was not recognized as a speech segment and speech recognition was not performed. In the scene where the timeout was set (69 utterances), the dialogue continued as if nothing had occurred. However, in other situations (70 utterances), the system did not react to anything, and the dialogue continued when the user spoke again. Furthermore, 42 utterances (8.7%) were incorrectly recognized as user intentions owing to speech recognition errors. These two errors were caused by the speech recognition system. The Google Speech Recognition Engine[4] was used in this system; this was expected to improve recognition accuracy in noisy environments.

The next most common cause was missing keyword registrations (26.3%). This system extracted keywords from the user’s utterances, and used them to determine the user intentions and generate response sentences. The keywords were manually registered by assuming the user’s utterances in advance, but this assumption was insufficient. Consequently, the appropriate dialogue could not be realized. This can be significantly prevented by registering additional keywords as required; however, registering all keywords is not realistic. This is the principal limitation of keyword-based systems. Note that in some cases, the notations of the registered keywords were different from those of the recognition results (such as hiragana and kanji), and the correct response could not be obtained although the keywords were registered.

In the experiment, all subjects were given a questionnaire regarding their impressions of the system[3]. The relationship between the content of the dialogue and user satisfaction obtained from the questionnaire was analyzed to determine the factors that affected the level of satisfaction.

The questionnaire asked about the level of satisfaction with nine items on a 7-point scale and the total score of these items was defined as the “overall satisfaction score.” The average score was 40.1 out of 63 points, which is not high; however, the system was evaluated to some extent.

The correlation coefficients between the overall satisfaction score and some possible items were calculated to determine the types of responses of the system that increased satisfaction. The results are presented in Table II.

Table II revealed that the percentage of appropriate responses had a correlation coefficient of $+0.42$, thus indicating a significant impact on the impression of the dialogue system. Particularly, this percentage was correlated with several items in the questionnaire: “Were you satisfied with the choice of tourist attractions to visit?” ($+0.48$); “Did you trust the robot?” ($+0.41$), and “Did you use the information obtained from the robot to select the sightseeing spot?” ($+0.41$). These results also show that the dialogue system was trusted and recognized as worth consulting.

However, the number of utterances by the user had a negative correlation coefficient of $-0.37$. As the number of utterances here included “restatements” (the user repeats the same utterance because the system did not react to it despite the user’s utterance), restatements were assumed to reduce satisfaction. However, the correlation coefficient between the number of restatements and overall satisfaction score was $-0.29$; therefore, this could not be concluded as the only cause. As the number of user utterances increases, the number of inappropriate responses by the system may also increase, thereby reducing the level of satisfaction. Note that the number of restatements also had a higher negative correlation with questions “Were you able to obtain sufficient information about the sightseeing spots?” ($-0.40$) and “Would you like to visit this travel agency again?” ($-0.38$), thus indicating a negative impact on the objective of consulting.

Moreover, neither the percentage of incorrect responses nor “I see” responses exhibited a significant correlation with the overall satisfaction score. Although a more detailed analysis is required, it does not seem to directly lower the level of satisfaction.