Affective Manifolds: Modeling Machine’s Mind to Like, Dislike, Enjoy, Suffer, Worry, Fear, and Feel Like A Human

Imitating any alive creature, we can model a machine. For example, we can have vegetable-like, animal-like, and human-like machines which can play the roles of robots as vegetables (plants), animals, and humans. These machines have some advantages and benefits to biological organisms. For example, an alive vegetable-like machine does not wilt. An alive animal-like machine, such as a dog robot, does not dye or can be taught very fast.

An alive human-like machine is more robust to damages and can work in dangerous environments. Moreover, they do not get tired from working. Although, as we will show in this paper, we can add the ability of getting tired or having pain to the machines. It is possible to have multiple types of human-like machines with different abilities where the worker robots do not posses the ability of pain and tiredness but the living robots have these characteristics to live among humans. Nonetheless, this may raise the ethical dilemmas of racism/distinction among robots and racism/distinction between human and robot, which can be serious problems in the future. This reminds us that affective computing usually faces ethical challenges [15].

Diffusion MRI and fMRI experiments [9] have shown that different parts of brain are activated based on various mood characteristics, such as joy, suffer, love, fear, etc. Accordingly, we can define mind as a set of affective manifolds, each of which corresponds to a group of characteristics of mind. This models the different parts of machine’s mind. The affective manifolds play the role of latent variables for characteristics of mind. Some example effective manifolds will be mentioned in Section 2.4.

An example affective manifold is the love manifold containing the states of “love”, “like”, “dislike”, and “hate”. More detailed examples for affective manifold will be provided in Section 2.4.

Any linear dimensionality reduction method, such as Fisher discriminant analysis [7], can be used to learn an affective subspace.

An illustration of parts of machine’s mind is in Fig. 1. Eq. (6) simulates the brain of human whose different parts are activated for different tasks, according to the diffusion MRI and fMRI experiments [9].

An example for two non-overlapping affective manifolds is where one manifold contains the states “frown”, “neutral lips”, “smile”, and “laugh”, and the other manifold includes the states “feared”, “worried”, “enjoy”, and “laugh”. The state “laugh” is shared between the two characteristic groups of these affective manifolds.

We can define various affective manifolds. As many effective manifolds as required can be defined and used. Some example affective manifolds are provided in the following.

We can use different learning methods for different manifolds depending on their complexities. The dimensionality of every affective manifold or subspace, denoted by $p$, depends on the complexity of that characteristic group. The more complex the characteristic is, the larger dimensionality of manifold is required. Both complexities of states and the number of states have impact on complexity:

• •

  The more number of states a characteristic group has, the more complex the affective manifold is, so a larger dimensionality is required for the affective manifold.

• •

  The more complex the states of a characteristic group are, the more complex the affective manifold is, so a larger dimensionality is required for the affective manifold.

The goal is to learn a nonlinear mapping $f$, with learnable parameters $\theta$, from a $d$-dimensional input space to a $p$-dimensional embedding space:

The $d$-dimensional data can be any signal captures by any sensor of the machine/robot. The input signal stimulates one of the states in the affective manifold, the same way as an input signal to human’s brain stimulates a state in human’s mind (for example, we can get sad by looking at a picture or listening to a piece of music). The input signals are measured by any sensor of the machine, where the sensors can model the five senses of a human. For example, the input signals can be images, observed by the cameras (eyes) of robot, sound signals heard by microphones, or any other signals. The $p$-dimensional embedding space is the affective manifold. Therefore, the mapping (7) is a mapping from the input signal to the affective manifold in the machine’s mind. The pipeline of mapping the input signals to the affective manifolds in the machine’s mind is illustrated in Fig. 1.

We can learn an affective manifold using deep metric learning [6]. One useful network for deep metric learning is the Siamese network [2] containing two or three networks sharing their weights. The mapping $f$ can be a Siamese neural network whose weights are the learnable parameters $\theta$.

Based on the psychological experiments, philosophical studies, and biological facts, we can define the desired relative distances (or margins) between the states of an affective manifold. For this, in the embedding space of the manifold, we define the relative margins between the states of that manifold. Note that the actual distances are not important but the relative distances matter. Scaling up or down the actual distances can only have impact on the speed of convergence of training. In addition to the scale of distances, the rotation and mirroring are not important in manifold learning because the relative distances are preserved under these transformations.

Figure 2 depicts an example for imagination of the desired relative distances between the states of an affective manifold. Section 5 will provide examples for the relative distances of states in simulations. Note that defining the margins between the states of an affective manifold requires empirical social experiments, biological facts, psychological studies, or philosophical arguments. Some psychological and philosophical analyses used for determining the margins will be discussed in Section 5.

If an affective manifold has $s$ states, we define a symmetric margin matrix $\boldsymbol{M}\in\mathbb{R}_{+}^{s\times s}$ whose elements are the desired margins between the states. Let $m_{i,j}$ denote the $(i,j)$-th element of the matrix $\boldsymbol{M}$, i.e., the desired margin between the $i$-th and the $j$-th states of the affective manifold.

For learning an affective manifold, we can learn an embedding space where the variance of instances within every state is minimized and the inter-state distances are fixed to the desired distances. For this goal, we prepare mini-batches of $d$-dimensional anchor-positive-negative triplets. In every triplet, the anchor and positive instances belong to the same state but the negative instance is from another state.

In every mini-batch, let the $j$-th $d$-dimensional anchor, positive, and negative instances be denoted by $\boldsymbol{x}^{j}_{a}\in\mathbb{R}^{d}$, $\boldsymbol{x}^{j}_{p}\in\mathbb{R}^{d}$, and $\boldsymbol{x}^{j}_{n}\in\mathbb{R}^{d}$, respectively. If the bach size is denoted by $b$, every mini-batch is $\{(\boldsymbol{x}^{j}_{a},\boldsymbol{x}^{j}_{p},\boldsymbol{x}^{j}_{n})\}_{j=1}^{b}$. The embeddings of triplets in the affective manifold are $f(\boldsymbol{x}^{j}_{a})\in\mathbb{R}^{p}$, $f(\boldsymbol{x}^{j}_{p})\in\mathbb{R}^{p}$, and $f(\boldsymbol{x}^{j}_{n})\in\mathbb{R}^{p}$. We denote the state of the instance $\boldsymbol{x}$ by $s(\boldsymbol{x})$; hence, the states of a triplet are $s(\boldsymbol{x}^{j}_{a})$, $s(\boldsymbol{x}^{j}_{p})$, and $s(\boldsymbol{x}^{j}_{n})$.

Inspired by the triplet loss [18] and the contrastive loss [8], we propose a loss function for learning an affective manifold. We define the following loss function for training the affective manifold:

where $\lambda_{p}>0$ and $\lambda_{n}>0$ are the weighting parameters controlling the relative importance of the positive loss $\mathcal{L}_{p}$ and the negative loss $\mathcal{L}_{n}$. The functions $\mathcal{L}_{p}$ and $\mathcal{L}_{n}$ are the loss functions for the positive and negative pairs:

where $\|.\|_{2}$ denotes the $\ell_{2}$ norm.

The loss function (9) is the mean squared error of distances between the anchor and positive of triplets in a mini-batch. The loss function (10) is the mean squared error of difference of the desired margin and the current distance between the anchor and negative of triplets in a mini-batch. Therefore, on the one hand, the overall loss function (8) reduces the distances between instances of every state so that the veriance of every state is reduced. On the other hand, this loss function makes the distances between every two states equal to the desired margin between those two states.

It is noteworthy that machine’s every affective manifold can change gradually over time using transfer learning [20]. For this, we further train the already trained network using new triplets from new input signals. This transfer learning simulates experiencing new things by the machine. This is inspired by the gradual change of human’s mind and characteristics over time by experiencing.

For every $d$-dimensional input signal, the machine shall have one of the states of an affective manifold. After training the network, we pass the input $\boldsymbol{x}\in\mathbb{R}^{d}$ to the network and we get its embedding $f(\boldsymbol{x})\in\mathbb{R}^{p}$ at the output of network. We can have various approaches for inferring the state of the input signal in the embedding space. One possible approach is explained in the following. Let the mean of embeddings of the training instances in the $i$-th state be denoted by $\boldsymbol{s}_{i}\in\mathbb{R}^{p}$. The state of the input signal can be determined as:

Using other distance metrics such as the Mahalanobis distance or other norms is also possible. By inferring the state, the machine’s mind becomes able to possess a state of that characteristic group in reaction to the input signal. For example, the machine becomes happy or sad by looking at a specific picture or listening to a specific sentence/music.

If two machines have minds, containing affective manifolds, they can interact with each other and react emotionally accordingly. For example, they can talk to one another and see each other and react to the chat or the body language of the other machine (if the machines are implemented in robots). In this meaning, two machines (or robots) can interact with each other similar to how two humans interact, if they have sufficient number of affective manifolds in their minds. For instance, two machines can fall in love with each other, dislike one another, or get angry at each other (e.g., if one of them verbally insults the other one and the other machine gets sad).

If a machine or robot has a mind with affective manifolds, it can interact with human(s) or living creatures. This also opens the gate for forming positive and negative emotions between human and machine. For example, the robot and human can interact and get happy, get sad, or even fall in live with each other. Sci-fi movies have tried to show the possibility of such situations. For example, in the movie “Her” directed by Spike Jonze, a man falls in love with the operating system of his computer. Another example is the movie “Ex Machina”, directed by Alex Garland, in which a human falls in love with a robot.

Any positive and negative emotions and feedback can happen between human and machine. It is possible to have these emotions under control if the number of states in the affective manifolds are controlled (for example, if we remove some controversial states from the machine’s mind). Forming emotions between human and machine may open the gate for them to even fall in love with each other [17]; some researchers have even gone a step further and raised the possibility of marriage of human and robot in the future [13]. Obviously, this raises serious ethical challenges which need to be addressed [19]; however, we should note that affective computing is always faced with ethical difficulties [15].

The affective manifolds can also be useful for human-human interaction. This might seem strange as there is not any machine in the sides of this interaction. However, note that it is possible to model a human’s mind as a machine’s mind. This simplifies (reduces) the complex human’s mind to a simpler mind but with sufficiently good imitation. The modeled mind can almost produce same behavior as the human’s mind if it has enough affective manifolds with enough number of states and is trained well enough.

An example usage of this application is using this model in matchmakers for dating apps. For example, the behavior of the human users, based on their characteristics and their likes/dislikes, can be used to train an affective manifold with states “match” (“like”) and “non-match” (“dislike”) for every user. Then, for each user, the characteristics of other users are fed as input signals to the learning model to see where every other user falls in the affective manifold. The user with the closest position in the state of “match” (“like”) will be the best match for that user (see Eq. (11)).

We used a Siamese network for affective manifold learning. The backbone network included two convolutional layers followed by two fully connected layers. The Parametric Rectified Linear Unit (PReLU) activation functions, max pooling, and dropout were used in the network. The batch size was set to $b=32$ and the embedding dimensionality of affective manifold was set to $p=2$ for better visualization of results. We set $\lambda_{p}=\lambda_{n}=1$ to have equal weights in the loss function (8). For every experiment, we trained the network for ten epochs. For the sake of simulation, we used the MNIST dataset [12] as the input signal where every digit is associated to one of the states.

The affective love manifold can have the states “hate”, “dislike”, “like”, and “love”. Based on two different physiological observations, we can define the margins either linearly or nonlinearly.

The affective joy manifold models the level of joy in the machine’s mind. This manifold can contain the states “suffered”, “feared”, “worried”, “enjoying”, “relaxed”, and “bored”. One possible way, but not the only way, to define the margins between these states is illustrated in Fig. 4. As this figure shows, suffering is the opposite of relaxing and being bored in terms of comfort. We set the state “feared” away from “suffered” and “worried” with the same distances. However, among “feared” and “suffered”, it seems better to have “feared” closer to “worried” because suffering is more severe. Among the negative joy levels, the state “worried” can be the closest to the state “enjoying” which is positive. For the neutral joy levels, which are “relaxed” and “bored”, we define the margins equal from the state “enjoying”.

The reason for why we put the positive state “enjoying” between the negative states (i.e., “suffered”, “feared”, and “worried”) and the neutral states (i.e., “relaxed”, and “bored”) is explained in the following. Previously, humans thought that suffering and joy are the two opposite ends of the levels of joy in life. However, philosophical investigations into joy and meaning of life have shown that enjoying is merely a short period of time, between suffering and boredom, in every experience of life. Human starts from the suffer of not having something, tries and obtains it, then enjoys it for some short time, and then the human becomes bored of that. This cycle starts again for obtaining something else. In fact, boredom, and not enjoying, is the opposite of suffering in life. This was initially thought through by Gautama Buddha. In the modern philosophy, Arthur Schopenhauer has addressed this concept:

“Life swings like a pendulum backward and forward between pain and boredom.”

Arthur Schopenhauer, 1788–1860.

According to the above explanations and Fig. 4, we can calculate relative margins based on the desired angles and relative positions of states:

where “suffered”, “feared”, “worried”, “enjoying”, “relaxed”, and “bored” are the first to sixth states.

The trained embedding space for the affective joy manifold is illustrated in Fig. 3-b. As this figure shows, the states are correctly learned to follow the desired margins determined in Eq. (14) and depicted in Fig. 4. Again, note that the relative distances are important and the overall scale, rotation, and mirroring do not matter.