Persistent-Transient Duality in Human Behavior Modeling

We consider the problem of modeling the sequential behaviors of $N$ entities in a dynamic system, where each entity $i$ is represented by a class label $c_{i}$ and sequential features $X_{i}=\left\{x_{i}^{t}\right\}_{t=1}^{T}$. After observing $T$ steps, we predict the features in the next $L$ steps, $Y_{i}=\left\{y_{i}^{t}\right\}_{t=T+1}^{T+L+1}$. The entity classes include the human and object ($c_{i}$=”human”/”object”), which decide the entity’s feature spaces and behaviors.

In this paper, we use a customized attention function defined over the query $q\in R^{d}$ and the identical key/value pairs $V=\{v_{j}\}_{j=1}^{N}\in R^{d\times N}$:

$\displaystyle\text{Attn}\left(q,V\right)=\sigma\left(\sum_{j=1}^{N}\text{softmax}_{j}\left(\boldsymbol{W}_{\alpha}^{T}\left[\boldsymbol{W}_{q}q;\boldsymbol{W}_{v}v_{j}\right]\right)\boldsymbol{W}_{v}v_{j}\right)$

where $\left[\cdot;\cdot\right]$ is the concatenation, $\sigma$ is a non-linear activation function, $\boldsymbol{W}_{v}$, $\boldsymbol{W}_{q}$, and $\boldsymbol{W}_{\alpha}$ are learnable weights.

We model the persistent-transient duality of human behavior by a hierarchical neural network called Persistent-Transient Duality Networks (PTD) (See Fig. 2). The network has three main components: The Persistent Channel, the Transient Channels, and the Transient Switch.

The Persistent channel oversees the global view of the scene, including the humans and other entities. It has the form of a recurrent relational network where entities interact in the spatio-temporal space spanned by the video. The temporal evolution of each entity is modeled as:

$$h_{i}^{\mathcal{P},t}=\text{RNN}_{c_{i}}\left(\left[z_{i}^{\mathcal{P},t},m_{i,\mathcal{T}\rightarrow\mathcal{P}}^{t}\right],h_{i}^{\mathcal{P},t-1}\right),$$

(1)

where $\text{RNN}_{c_{i}}$ is a recurrent unit that corresponds to the class of the $i^{\text{th}}$ entity, maintaining hidden states $h_{i}^{\mathcal{P},t-1}$ and consuming input vector $z_{i}^{\mathcal{P},t}$. This input is formed as $z_{i}^{\mathcal{P},t}=\left[x_{i}^{t};m_{i}^{t}\right]$, where $x_{i}^{t}$ is the entity’s feature and $m_{i}^{t}$ is the message aggregated from the entity’s neighbor through attention, $m_{i}^{t}=\text{Attn}\left(u_{i}^{t},\left\{u_{j}^{t}\right\}_{j\neq i}\right)$ with $u\_^{t}=\left[x\_^{t};h\_^{\mathcal{P},t-1}\right]$.

The unit also uses an optional transient-persistent message $m_{i,\mathcal{T}\rightarrow\mathcal{P}}^{t}$ which is non-zero if a corresponding transient process (Sec. 2.4) is currently active.

The Persistent channel generates two outputs from its hidden state: the future prediction $\hat{y}_{i}^{\mathcal{P},t}$ and the message $m_{i,\mathcal{P}\rightarrow\mathcal{T}}^{t}$ to the Transient channel:

$$\hat{y}_{i}^{\mathcal{P},t}=\text{MLP}\left(h_{i}^{\mathcal{P},t}\right),\hskip 8.5359ptm_{i,\mathcal{P}\rightarrow\mathcal{T}}^{t}=\text{MLP}\left(h_{i}^{\mathcal{P},t}\right).$$

(2)

The channel’s prediction output $\hat{y}_{i}^{\mathcal{P},t}$ are combined with those from the Transient channel as detailed in Sec. 2.6.

This persistent channel is equivalent in modeling with the major state-of-the-art HOI-M by using relational recurrent models [2]. Our key novelty is the consideration of the second side of the duality - the Transient process, presented in the next section.

Within the persistent-transient duality, the Transient process allows the model to zoom in at relevant context and take the local viewpoint of the human entity when it starts to interact with objects. This human-specific process is implemented by a Transient channel. The egocentric property of this channel separates it from the global view of its parent persistent channel and reflects in three aspects of feature representation, computational structure, and inference logic.

The life cycles of the Transient processes are managed based on the situation of human’s activity by a neural Transient Switch (See Fig. 3b).

The switch first considers the current persistent state and the surrounding environment of the center entity $r$ to update its switch RNN $h_{r}^{s,t}$:

$\displaystyle h_{r}^{s,t}$

$\displaystyle=\text{RNN}_{s}\left(\left[h_{r}^{\mathcal{P},t},m_{r}^{s,t}\right],h_{r}^{s,t-1}\right),$

(6)

where $m_{r}^{s,t}=\text{Attn}\left(h_{r}^{s,t-1},\left\{\bar{x}_{l}^{t}\right\}_{l\neq r}\right)$, and $\bar{x}_{l}^{t}=f_{\text{ego}}\left(x_{l}^{t},x_{r}^{t}\right)$ defined in Sec. 2.4.

The RNN unit is important in maintaining the switch’s sequential properties, making it a state-full machine that can handle patterns of on and off switchings and avoid spurious decisions caused by noises. The switch-on probability $p_{r}^{t}$ is:

$$\hat{p}_{r}^{t}=\gamma_{r}^{t}\cdot\text{sigmoid}\left(Wh_{r}^{s,t}\right),$$

(7)

where $W$ is learnable weights. The discount factor $\gamma_{r}^{t}\in[0,1]$ responds to the distance from the subject to the nearest neighbor: $\gamma_{r}^{t}=\exp\left(-\beta\cdot\text{min}\left\{\left\|d_{lr}^{t}\right\|_{2}\right\}_{l\neq r}\right),$ where $d_{lr}^{t}$ are the center-leaf geometrical distances and $\beta$ is a learnable decay rate. This factor acts as a disruptive shortcut gate that modulates the switching decision based on the spatial evidence of the interaction.

Finally, the binary switch decision $\hat{s}_{r}^{t}$ is decided by thresholding the switch score: the switch is on ($\hat{s}_{r}^{t}=1)$ when $\hat{p}_{r}^{t}>=0.5$, and is off otherwise. When it changes from $0$ to $1$, a new Transient process is created at time $t$ for person $r$. This transient process will keep running until the switch turns off, then the persistent process again becomes the single operator.

In PTD, future motions are predicted by unrolling the model into the future of $L$ time steps. At each future time step $t$, the predictions from persistent channel $\hat{Y}^{\mathcal{P},t}$ (Eq. 2) and those from Transient channel $\hat{Y}^{\mathcal{T},t}$ (Eq. 5) are combined with the priority on the Transient predictions. For a human entity, if its Transient channel is activated, the Transient prediction will be chosen; otherwise, the Persistent prediction will be used. For an object entity, if it receives an active outward Transient edge, it will take that channel’s prediction. If it receives multiple outward edges, it uses the prediction from the channel with the highest transient score $\hat{p}_{r}^{t}$. Otherwise, it uses the persistent prediction by default.

The model is trained end-to-end with two losses: prediction loss and switch loss, $\mathcal{L}=\mathcal{L}_{\text{pred}}+\lambda\mathcal{L}_{\text{switch}}.$

The Prediction loss $\mathcal{L}_{\text{pred}}$ measures the mismatch between predicted values $\hat{Y}$ and groundtruth $Y$, $\mathcal{L}_{\text{pred}}=||\hat{Y}^{T:T+L}-Y^{T:T+L}||_{2}^{2}.$

The Switch loss $\mathcal{L}_{\text{switch}}$ is used to supervise the Transient Switch and is implemented as: $\mathcal{L}_{\text{switch}}=\text{BCE}\left(\hat{P}^{1:T+L},P^{1:T+L}\right)$, where $\hat{P}^{t}$ are the switch scores (Eq. 7) of all human entities, and $P^{t}$are binary ground-truth switch scores collected at time step $t$.