Leading or Following? Dyadic Robot Imitative Interaction Using the Active Inference Framework

The stochastic generative model is used for prior generation, as illustrated in the future prediction part (after time step 4) in Fig. 1.

PV-RNN is comprised of deterministic variables $\mathbf{d}$ and random variables $\mathbf{z}$. An approximate posterior distribution $q_{\phi}$ is inferred based on the prior distribution $p_{\theta}$ by means of error minimization on the generated prediction $\mathbf{X}$. The generative model can be factorized as:

	$\displaystyle\footnotesize p_{\theta}(\bar{\bm{X}}_{1:T},{\mathbf{d}}_{1:T},\bm{z}_{1:T}|\bm{d}_{0})=$		(2)
	$\displaystyle\prod_{t=1}^{T}p_{\theta_{\bar{\bm{X}}}}(\bar{\bm{X}}_{t}|{\mathbf{d}}_{t})p_{\theta_{{d}}}({\mathbf{d}}_{t}|{\mathbf{d}}_{t-1},\bm{z}_{t})p_{\theta_{z}}(\bm{z}_{t}|{\mathbf{d}}_{t-1})$

Although $\mathbf{d}$ is a deterministic variable, it can be considered to have a Dirac delta distribution centered on $\tilde{\bm{d}}$ as $\bm{\sigma}(\bm{d}-{\tilde{\bm{d}}})$. $\bar{\bm{X}}$ is conditioned directly on $\bm{z}$ through $\tilde{\bm{d}}$. At the initial time step, $\tilde{\bm{d}}$ is set to $0$. Otherwise, $\tilde{\bm{d}}$ is recursively computed, for which the internal state before activation is denoted by $\bm{h}$. This internal state $\bm{h}$ is a vector, calculated as the sum of the internal states of the current level $l$ and its connecting layers of the previous time step $t-1$ plus the latent $\bm{z}$ in the same layer of the current time step $t$:

	$\displaystyle\bm{\tilde{d}}^{l}_{t}$	$\displaystyle=\text{tanh}(\bm{h}^{l}_{t})$		(3)
	$\displaystyle\bm{h}^{l}_{t}$	$\displaystyle=\left(1-\frac{1}{\tau^{l}}\right)\bm{h}^{l}_{t-1}+$
		$\displaystyle\frac{1}{\tau^{l}}\left(\bm{W}^{ll}_{dd}\bm{\tilde{d}}^{l}_{t-1}+\bm{W}^{ll}_{zd}\bm{z}^{l}_{t}+\bm{W}^{ll+1}_{dd}\bm{\tilde{d}}^{l+1}_{t-1}+\bm{W}^{ll-1}_{dd}\bm{\tilde{d}}^{l-1}_{t-1}\right)$

$\tau^{l}$ denotes the layer-specific time constant. With larger value for $\tau^{l}$, slower timescale dynamics develop, whereas with a smaller value set, faster timescale dynamics dominate. $\bm{W}$ represents connectivity weight matrices between layers and their deterministic and stochastic units. The output with size $N_{x}$ is computed as mapping from $\bm{\tilde{d}}^{1}$ as:

$\displaystyle\mathbf{X}_{t}=\bm{W}^{ll}_{Xd}\bm{\tilde{d}}_{t}^{1}+\bm{b}_{X}$

(4)

The prior distribution $p_{\theta}(\bm{z}_{t})$ is a Gaussian distribution represented with mean $\bm{\mu}_{t}^{p}$ and standard deviation $\bm{\sigma}_{t}^{p}$. The prior depends on $\bm{\tilde{d}}_{t-1}$ by following the idea of a sequence prior [26], except at $t=1$ where it follows a unit Gaussian distribution.

	$\displaystyle p_{\theta}(\bm{z}_{1})$	$\displaystyle=\mathcal{N}(0,I)$		(5)
	$\displaystyle p_{\theta}(\bm{z}_{t}|\tilde{\bm{d}}_{t-1})$	$\displaystyle=\mathcal{N}(\bm{\mu}^{p}_{t},(\bm{\sigma}^{p}_{t})^{2})\text{ where $t>1$}$
	$\displaystyle\bm{\mu}^{p}_{t}$	$\displaystyle=\text{tanh}(\bm{W}^{ll}_{d\mu^{p}}\bm{\tilde{d}}_{t-1})$
	$\displaystyle\bm{\sigma}^{p}_{t}$	$\displaystyle=\exp(\bm{W}^{ll}_{d\sigma^{p}}\bm{\tilde{d}}_{t-1})$

Based on the work on variational autoencoders, we use the reparameterization trick to formulate the latent prior of $\bm{z}_{t}$ as mean $\bm{\mu}^{p}_{t}$ and standard deviation $\bm{\sigma}^{p}_{t}$. The reparameterization trick was introduced by Kingma and Welling [27] to make random variables differentiable for backpropagating errors through the network for learning. The same consideration is taken for the posterior of $\bm{z}_{t}$ in the inference model as well (cf. below Eq. 7).

Posterior inference is performed during learning and afterward, during action and perception. Fig. 1 illustrates information flow in the posterior inference in a time window from time step 2 to time step 3. The inference model for the posterior is described as:

$\displaystyle\footnotesize Q_{\phi}(\bm{z}_{t}|\bm{\tilde{d}}_{t-1},\bm{e}_{t:T})=\mathcal{N}(\bm{z}_{t};\bm{\mu}^{q}_{t},\bm{\sigma}^{q}_{t})$

(6)

where $\bm{e}_{t}$ denotes the error between the target $\bar{\mathbf{X}}_{t}$ and the predicted output $\mathbf{X}_{t}$. Like the prior $p_{\theta}$, the posterior $q_{\phi}$ is also a Gaussian distribution with mean $\bm{\mu}_{t}^{q}$ and standard deviation $\bm{\sigma}_{t}^{q}$. For $\bm{z}_{1:T}$ it is defined as:

	$\displaystyle q_{\phi}(\bm{z}_{t}|\bm{e}_{t:T})$	$\displaystyle=\mathcal{N}(\bm{\mu}^{q}_{t},\bm{\sigma}^{q}_{t})$		(7)
	$\displaystyle\bm{\mu}^{q}_{t}$	$\displaystyle=\text{tanh}(\bm{A}^{\mu}_{t})$
	$\displaystyle\bm{\sigma}^{q}_{t}$	$\displaystyle=\exp(\bm{A}^{\sigma}_{t})$

Since computing the true posterior is intractable, an approximate posterior $q_{\phi}$ is inferred by maximizing the lower bound, analogous to Eq. (1). Here, the adaptation variable $\mathbf{A}_{1:T}$ forces the parameters $\phi$ of the inference model to represent meaningful information. The lower bound of PV-RNN can be derived as:

	$\displaystyle\footnotesize L(\theta,\phi)$	$\displaystyle=\sum_{t=1}^{T}(\frac{1}{N_{X}}E_{q_{\phi}(\bm{z}_{t}|\bm{\tilde{d}}_{t-1},\bm{e}_{t:T})}\big{[}\log p_{\theta_{\bar{X}}}(\bar{\bm{X}}_{t}|\bm{\tilde{d}}_{t})\big{]}-$		(8)
		$\displaystyle\sum_{l}^{L}\frac{\mathbf{w}^{l}}{N_{z}^{l}}D_{KL}\big{[}q_{\phi}(\bm{z}_{t}|\bm{\tilde{d}}_{t-1},\bm{e}_{t:T})||p_{\theta_{z}}(\bm{z}_{t}|\bm{\tilde{d}}_{t-1})\big{]})$

where the first term is the accuracy and the second term is the complexity (for details referred to [16]). $N_{x}$ and $N_{z}^{l}$ are the number of sensory dimensions and the number of the latent random variables at the $l^{th}$ layer, respectively. $\mathbf{w}^{l}$ serves as a weighting parameter for the complexity term in layer $l$ and is referred to as the meta-prior [16]. The meta-prior represents the strength for regulating the closeness between the posterior and the prior distributions. In $t=1$, $\mathbf{w}^{l}_{1}$ is set with 1.0. $\mathbf{w}^{l}_{2:T}$ is set to a specific value when the sequence prior [26] is used after time step 1. In the posterior inference, all learning-related network parameters of $\theta$, $\phi$, and the adaptive variable $\mathbf{A}$ are updated to maximize the lower bound by back-propagating the error from time step $T$ back to $t_{1}$ [28].

Two robots equipped with the PV-RNN model interact during synchronized imitation. In the interaction, the robots predict proprioception $\mathbf{X}^{pr}_{t+1}$ and exteroception $\mathbf{X}^{ex}_{t+1}$ for the next time step. The predicted $\mathbf{X}^{pr}_{t+1}$ regulates joint angle movements of a robot by considering a PID controller. This movement $\mathbf{X}^{pr}_{t+1}$ can then be sensed by the other robot in terms of exteroception $\bar{\mathbf{X}}^{ex}_{t+1}$. This is provided through the kinematic transformation of joint angles $\mathbf{X}^{pr}_{t+1}$ (cf. Fig. 2).

While in the training phase, the error signal is taken from the proprioceptive $\bar{\mathbf{X}}^{pr}$ as well as the exteroceptive $\bar{\mathbf{X}}^{ex}$ target sequences, in the interaction phase the error signal for each robot is taken only from $\bar{\mathbf{X}}^{ex}$¹¹1Considering only $\bar{\mathbf{X}}^{ex}$ for the error term in interaction settings assumes that the PID controller generates only negligible position errors for the joints.. During interactions, prediction errors ${\mathbf{e}}^{x}$ are generated and propagated bottom-up throughout all layers, as well as time steps in the posterior inference window, in terms of the latent error ${\mathbf{e}}^{z}$. This updates posterior distributions in the network and minimizes the variational free energy. In this phase, only $\mathbf{A}_{1:T}$ is updated without updating network parameters $\theta$ and $\phi$.

A robot that is trained with A$20\%$B$80\%$C will first generate an A movement, and then transition to B with 20 percent probability and to C with 80 percentage probability. We found that smaller $\mathbf{w}$ settings are less stable in reproducing the probabilistic structure of the training data. The BC-ratio was either greater or less than $20\%$ for B or greater or less than $80\%$ for C. Networks trained with larger meta-priors become more reliable in regenerating the probabilistic training sequence (BC-ratio in Table II). In addition to the capacity of learning the probability distribution of the training data, we found that smaller meta-priors show noisier pattern generation. Non-classified movements were as high as $22\%\pm 4$ with $\mathbf{w}=0.01$ and decreased to $6\%\pm 0.6$ with $\mathbf{w}=3.4$.

Repeatability in generating sequences in prior generation was examined by conducting a divergence analysis. Sequences are considered diverged when a comparison per time step of $\mathbf{X}^{pr}$ exceeds a threshold³³3As the threshold for the divergence analysis, we use the mean squared error of joint angle data $[-180,180]$ of $\mathbf{X}^{pr}$. The threshold is set to $55$.. Out of 20 regeneration sequences, we randomly select one as a reference and calculate the average divergence step of the other sequences to this the reference. Out of 400 time steps of prior generation, sequences diverged from the reference around time step $43$ for networks trained with smaller $\mathbf{w}$. With increasing $\mathbf{w}$, repeatability of the trajectories increased. Here the divergence step was around $139$ (cf. divergence step $t$ in Table II).

Loose regulation of the complexity term results in noisier, less repeatable prior generation performance. Also the learned probability for transition to either B or C is not accurate. This observation changes with increasing meta-prior. The larger $\mathbf{w}$, the more accurate the learned transition probability becomes. Also, prior generation becomes more repeatable by developing more deterministic dynamics with the initial sensitivity characteristics (i.e., the sequence is generated solely depending on the latent state in the initial time step). For subsequent dyadic robot interaction experiments, we empirically select the meta-prior setting $\mathbf{w}=0.005$ and $\mathbf{w}=3.4$ as two representatives of tight and loose regulation of the FEP complexity.

In Experiment 1, $\mathbf{R}^{1}_{0.005}$ adapts to the probabilistic transition of $\mathbf{R}^{2}_{3.4}$ by increasing the probability of performing B from $22\%$ in the stand-alone condition to $70\%$ in the dyad (Table III Experiment 1). Both robots are performing more B than C with a BC-synchronization of $56\pm 23\%$ which is significantly higher than the chance rate of $32\%$⁴⁴4We assume that B and C are independent probabilistic events. Then we can consider the probabilities for a robot $R$ to perform either a B movement as $\mathbf{P}^{R}(B)$ or a C movement as $\mathbf{P}^{R}(C)$. The actual BC-synchronization chance level can then be calculated as: $\mathbf{P}^{1}(B)\times\mathbf{P}^{2}(B)+\mathbf{P}^{1}(C)\times\mathbf{P}^{2}(C)=0.8\times 0.2+0.8\times 0.2=0.16+0.16=0.32$..

Fig. 5 shows an example of how prediction of the future and posterior inference of the past proceed as time passes from time step $199$, $229$, to $259$ for both robots. We observe that the intended future behavior (the prior generation) of $\mathbf{R}^{1}_{0.005}$ is not consistent with the actually performed actions after posterior inference. On the other hand, in the case of $\mathbf{R}^{2}_{3.4}$, the performed action complies with its prediction.

This behavior can be explained by looking at exemplar priors $\mu_{i}^{lp}$ and posteriors $\mu_{i}^{lq}$ for layer $l$ and neuron $i$ between two robots. In layer $1$, selected posterior network dynamics $\mu_{1}^{1q}$ and $\mu_{2}^{1q}$ are deviating from prior dynamics $\mu_{1}^{1p}$ and $\mu_{2}^{1p}$ for robot $\mathbf{R}^{1}_{0.005}$. Whereas the dynamics of $\mathbf{R}^{2}_{3.4}$ are mostly overlapping (cf. Fig. 4(a) and supplementary movie). More specifically, the average KL Divergence $\mathbf{e}^{z}$ of $\mathbf{R}^{1}_{0.005}$ is larger for all layers ($({\mathbf{e}}^{z,1},{\mathbf{e}}^{z,2},{\mathbf{e}}^{z,3})=(109.1,1.4,0.06)$) than for $\mathbf{R}^{2}_{3.4}$ ($({\mathbf{e}}^{z,1},{\mathbf{e}}^{z,2},{\mathbf{e}}^{z,3})=(0.4,0.0003,0.00001)$). This means that $\mathbf{R}^{2}_{3.4}$ tends to behave as intended because the posterior is attracted by the prior. On the other hand, $\mathbf{R}^{1}_{0.005}$ tends to adapt to $\mathbf{R}^{2}_{3.4}$ since the posterior is rather attracted by the observation than by the weaker prior belief.

Note that $\mu_{1}^{3q}$ and $\mu_{1}^{3p}$ in layer 3 change only slowly with time. This indicates that these latent variables represent how movement primitives transit from deterministic states to non-deterministic states using their slower timescale properties characterized by $\tau^{3}$.

When two robots with loose complexity regulation interact, both robots maintain their learned preferences in terms of probability in generating either B or C. $\mathbf{R}^{1}_{3.4}$, which learns a $76\%$ transition to C in a stand-alone situation, shows its preference to C in the dyad with probability of $83\%$. $\mathbf{R}^{2}_{3.4}$, which in a stand-alone condition would maintain its preference to B with a probability of $75\%$, shows $61\%$ percentage transition to B in the interaction. BC-synchronization rate turns out to be low as $31\pm 24\%$, which is almost equal to the chance rate. Examining the network dynamics of the prior and posterior distributions shows that the robots executed movements based upon their prior action intention without adapting their posteriors to observations of the other robot’s movement (cf. Fig. 4(b) and supplementary movie).

When two robots with tight regulation of complexity interact, both try to adapt their own action to the one demonstrated by the other. Indeed, Fig. 4(c) shows that the prior and posterior do not comply, but deviate. Whether trained with the probabilistic transition of A20%B80%C or A80%B20%C, both robots significantly reduce the tendency to perform their own intended behavior C or B, respectively. This is evidenced by changes of the BC-ratio from stand-alone compared to the dyadic setting (TABLE III Experiment 3). BC-synchronization rate is $42\pm 20\%$ which is higher than the chance rate but not significantly. The interaction becomes noisier, compared to results of Experiments 1 and 2 (cf. Fig. 4(c) and supplementary movie), which indicate that tight regulation makes robots more sensitive to temporal fluctuations in observations of their counterparts.