AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching

Temporally normalized frame difference [19] is proved to be robust for rPPG recovery in motion and bad illumination scenarios. In DeepPhys [16], a new type of normalized frame difference is designed as the network input. However, some important information will be lost due to the normalized clipping, which limits the capability of representation learning. Here we adopt the original RGB frames as input and design a novel Temporal Difference Convolution (TDC) for describing the temporal differences in feature levels. For simplicity, we describe TDC with 3$\times$3$\times$3 kernel and channel number 1. The case with larger kernel size and channel number is analogous. $w(i)$ denotes the learnable weight at local position $i$. As illustrated in Fig. 2, besides weighting the spatial information in the local region $\mathcal{R}_{t}$ at current time $t$, the TDC also aggregates the temporal difference clues within local temporal regions $\mathcal{R}_{t-1}$ and $\mathcal{R}_{t+1}$, which can be formulated as

where $p^{t}_{0}$ denotes current location at time $t$ on both input $I$ and output $O$ feature maps while $p^{t-1}_{n}$, $p^{t}_{n}$ and $p^{t+1}_{n}$ enumerate the locations in $\mathcal{R}_{t-1}$, $\mathcal{R}_{t}$ and $\mathcal{R}_{t+1}$, respectively. For instance, local receptive field region $\mathcal{R}_{t-1}$, $\mathcal{R}_{t}$ and $\mathcal{R}_{t+1}$ can be represented as $\left\{(-1,-1),(-1,0),\cdots,(0,1),(1,1)\right\}$. The hyperparameter $\theta\in[0,1]$ tradeoffs the contribution of temporal difference. The higher value of $\theta$ means the more importance of temporal difference information. Specially, TDC degrades to vanilla 3D convolution when $\theta=0$.

Overall, the advantages of introducing temporal difference clues into vanilla 3D convolution are in two folds: 1) cascaded with deep normalization operators, TDC is able to mimic the temporally normalized frame difference [19] in feature levels, and 2) temporal central difference information provides fine-grained temporal context, which might be helpful to track the local ROIs for robust rPPG recovery.

All existing end-to-end rPPG networks [15, 16, 17, 18] are designed manually, which might be sub-optimal for feature representation. In this paper, we first introduce NAS to automatically discover the best-suited backbone for the task of remote HR measurement. Our search algorithm is based on two gradient-based NAS methods [23, 24], and more technical details can be referred to the original papers.

As illustrated in Fig. 3(a), our goal is to search for cells to form a network backbone for the rPPG recovery task. As for the cell-level structure, Fig. 3(b) shows that each cell is represented as a directed acyclic graph (DAG) of $N$ nodes $\left\{x\right\}^{N-1}_{i=0}$, where each node represents a network layer. We denote the operation space as $\mathcal{O}$, and Fig. 3(c) shows nine designed candidate operations. Each edge $(i,j)$ of DAG represents the information flow from node $x_{i}$ to node $x_{j}$, which consists of the candidate operations weighted by the architecture parameter $\alpha^{(i,j)}$. Specially, each edge $(i,j)$ can be formulated by a function $\tilde{o}^{(i,j)}$ where $\tilde{o}^{(i,j)}(x_{i})=\sum_{o\in\mathcal{O}}\eta_{o}^{(i,j)}\cdot o(x_{i})$. Softmax function is utilized to relax architecture parameter $\alpha^{(i,j)}$ into operation weight $o\in\mathcal{O}$, that is $\eta_{o}^{(i,j)}=\frac{exp(\alpha_{o}^{(i,j)})}{\sum_{{o}^{\prime}\in\mathcal{O}}exp(\alpha_{{o}^{\prime}}^{(i,j)})}$. The intermediate node can be denoted as $x_{j}=\sum_{i<j}{\tilde{o}}^{(i,j)}(x_{i})$ and the output node $x_{N-1}$ is depth-wise concatenation of all the intermediate nodes excluding the input nodes.

In general, the architecture of the cells within four blocks are shared for robust searching. We also consider the flexible and complex setting when all the four blocks to be searched are varied. We name these two configurations as ‘Shared’ and ‘Varied’, respectively. In addition, we also compare the operation space with and without TDC in Section III-B.

In the searching stage, $\mathcal{L}_{train}$ and $\mathcal{L}_{val}$ are denoted as the training and validation loss respectively, which are all based on the overall loss $\mathcal{L}_{overall}$ described in Section II-C. Network parameters $\Phi$ and architecture parameters $\alpha$ are learned with the following bi-level optimization problem:

After convergence, the final discrete architecture is derived by: 1) setting $o^{(i,j)}=arg\,max_{o\in\mathcal{O},o\neq none}\,p_{o}^{(i,j)}$, and 2) for each intermediate node, choosing two incoming edges with the two largest values of $max_{o\in\mathcal{O},o\neq none}\,p_{o}^{(i,j)}$.

Besides designing the network architecture, we also need an appropriate loss function to guide the networks. However, existing negative Pearson (NegPearson) [17, 18] and signal-to-noise ratio (SNR) [15] losses only constrain in the time or frequency domains, respectively. It might be helpful for the network to learn more intrinsic rPPG features with both strong spectral distribution supervision in the frequency domain, and fine-grained signal rhythm guidance in the time domain. The loss in the time domain can be formulated as

where $X$ and $Y$ indicate the predicted rPPG and ground truth PPG signals, respectively. $T$ is the length of the signals. Inspired by SNR loss, we also treat HR estimation as a classification task via frequency transformation, which is formulated as

where $PSD(X)$ is the power spectral density of the predicted rPPG signal $X$, while $HR_{gt}$ denotes the ground truth HR value. $CE$ means the classical cross-entropy loss. Finally, the overall loss in the time and frequency domain is $\mathcal{L}_{overall}=\lambda\cdot\mathcal{L}_{time}+\mathcal{L}_{fre}$, where $\lambda$ is a balancing parameter.

Two problems hindering the remote HR measurement task caught our attention, and we propose two data augmentation strategies accordingly. On one hand, head movements could cause ROI occlusion. We propose data augmentation strategy ‘DA1’, which is to randomly erase or cutout partial spatio-temporal tubes within a random time clip (less than 20% spatial size and 20% temporal length), which mimics the situation of ROI occlusion. On the other hand, HR distribution is severely unbalanced as a reversed-V shape. We propose data augmentation strategy ‘DA2’, which is to temporally upsample and downsample the videos to generate extra training samples with extreme small or large HR values. To be specific, the videos with HR values larger than 90 bpm would be temporally interpolated twice while those with HR smaller than 70 bpm are downsampled with sampling rate 2, to simulate half and doubled heart rate, respectively.

Three public datasets are employed in our experiments. The VIPL-HR [14] and MAHNOB-HCI [25] datasets are utilized for intra-dataset testing with subject-independent 5-fold cross-validation while the MMSE-HR [6] is used for cross-dataset testing. Performance metrics for evaluating the average HR include the standard deviation (SD), the mean absolute error (MAE), the root mean square error (RMSE), and the Pearson’s correlation coefficient ($r$).

All ablation studies are conducted on Fold-1 of the VIPL-HR [14] dataset. PhysNet [17] is adopted as the backbone for exploring the impacts of loss functions and TDC. The NAS searched backbone is used for exploring the effectiveness of data augmentation strategies.

Impact of Loss Functions. Fig. 4(a) shows that $\mathcal{L}_{fre}$ plays a vital role for accurate HR prediction, which reduces 4.3 bpm RMSE compared with $\mathcal{L}_{time}$. The lowest RMSE (9.9 bpm) could be obtained when supervised with $\mathcal{L}_{overall}$, indicating the elaborate supervision in both time and frequency domains enables the model to learn robust rPPG features.

Impact of $\theta$ in TDC. According to Eq. (1), $\theta$ controls the contribution of the temporal difference information. As illustrated in Fig. 4(b), compared with the vanilla 3D convolution (i.e., $\theta$=0), TDC achieves lower RMSE in most cases, indicating the temporal difference clues are helpful for HR measurement. Specially, we consider $\theta=0.2$ and $\theta=1$ in our search operation space because of their better performance (RMSE=9.07 and 9.1 bpm, respectively).

Impact of NAS Configuration. As shown in Fig. 4(c), the searched backbones always perform better than that designed manually without NAS. It is interesting that NAS with shared cells is more likely to search excellent architectures than that with varied cells, which might be caused by the inefficient search algorithm and limited amounts of data. Moreover, better-suited networks could be searched with TDC operators.

Impact of Data Augmentation. Fig. 4(d) illustrates the evaluation results of various data augmentation strategies. It is surprised that only with ‘DA1’ (without ‘DA2’), it harms the performance slightly (0.5 bpm RMSE increased). The reason might be that the model learns rPPG-unrelated features due to the random spatio-temporal cutouts especially under the conditions of unbalanced data distribution. However, with the help of ‘DA2’ to enrich the samples with extreme HR values, the strategy ‘DA1+DA2’ improves the performance by 5%.

Results on VIPL-HR.  As shown in Table I, all three traditional methods (Tulyakov2016 [6], POS [8] and CHROM [7]) perform poorly because of the complex scenarios (e.g., large head movement and various illumination) in the VIPL-HR dataset. Similarly, the existing end-to-end learning based methods (e.g., PhysNet [17] and DeepPhys [16]) predict unreliable HR values as their Pearson’s correlation coefficient are quite low ($r\leq 0.2$). In contrast, our proposed AutoHR achieves comparable performance with state-of-the-art non-end-to-end learning based method RhythmNet [14], which can be regarded as a strong end-to-end baseline for the challenging VIPL-HR dataset. Note that RhythmNet needs the strict and heavy preprocessing procedure to eliminate external disturbances while our AutoHR learns the intrinsic rPPG-aware features automatically without any preprocessing.

Results on MAHNOB-HCI.  We also evaluate our method on the MAHNOB-HCI dataset, which is widely used in HR measurement. The video samples are challenging because of the high compression rate and spontaneous motions, caused by facial expressions for example. As shown in Table II, the proposed AutoHR achieves the lowest MAE (3.78 bpm) among the traditional and end-to-end learning methods, which indicates the robustness of the learned rPPG features. Our performance is on par with the latest non-end-to-end learning based method RhythmNet [14]. It implies that with excellent architecture and sufficient supervision, the end-to-end learning fashion is possible for robust HR measurement.

In this experiment, the VIPL-HR database is used for training and all videos in the MMSE-HR database are directly used for testing. It is clear that the proposed AutoHR also generalizes well in unseen domain. It is worth noting that AutoHR achieves the highest $r$ (0.89) among the traditional, non-end-to-end and end-to-end learning based methods, which signifies 1) our predicted HRs are highly correlated with the ground truth HRs, and 2) our model learns domain-invariant intrinsic rPPG-aware features.