Estimating Distances Between People using a Single Overhead Fisheye Camera with Application to Social-Distancing Oversight

In this approach, to estimate the 3D-world distance between two people we adopt the unified spherical model (USM) proposed by Gayer and Danilidis [9] for fisheye cameras and a calibration methodology to find this model’s parameters developed by Bone et al. [3]. This model, derived in [3], enables computation of an inverse mapping from image coordinates to 3D space as described next.

Consider the scenario in Fig. 2 where the center of the 3D-world coordinate system is at the optical center of a fisheye camera mounted overhead at height $B$ above the floor and a person of height $H$ stands on the floor. Let a 3D-world point $\bm{P}=[P_{x},P_{y},P_{z}]^{T}\in\mathbb{R}^{3}$ be located on this person’s body at half-height and let $\bm{P}$ appear at 2D coordinates $\bm{x}$ in the fisheye image.

Bone et al. [3] showed that the 3D-world coordinates $\bm{P}$ can be recovered from $\bm{x}$ with knowledge of $P_{z}$ and a 5-vector of USM parameters ${\bm{\omega}}$ via a non-linear function $G$:

$$\bm{P}=G(\bm{x},P_{z};{\bm{\omega}}),$$

(1)

In order to estimate ${\bm{\omega}}$, an automatic calibration method using a moving LED light was developed in [3]. In addition to $\bm{\omega}$, the value of $P_{z}$ is needed since this is a 2D-to-3D mapping. However, based on Fig. 2 we see that $P_{z}=B-H/2$.

In practice, we can only get a pixel-quantized estimate $\widehat{\bm{x}}$ of $\bm{x}$ from which we can compute an estimate $\widehat{\bm{P}}$ of $\bm{P}$ using (1). Let $\widehat{\bm{P}}_{A}$ and $\widehat{\bm{P}}_{B}$ denote the estimated 3D-world coordinates of person $A$ and person $B$, respectively, based on the centers of their bounding boxes $\widehat{\bm{x}}_{A}$ and $\widehat{\bm{x}}_{B}$. Then, we can estimate the 3D-world Euclidean distance $\widehat{d}_{AB}$ between them via:

$$\widehat{d}_{AB}=||\widehat{\bm{P}}_{A}-\widehat{\bm{P}}_{B}||_{2}.$$

(2)

In this approach, we train a neural network to estimate the distance between person $A$ and person $B$. Since the distance between two points in a fisheye image is invariant to rotation, we pre-process locations $\bm{x}_{A}$ and $\bm{x}_{B}$ before feeding them into the network. First, we convert $\bm{x}_{A}$ and $\bm{x}_{B}$ to polar coordinates: $\bm{x}_{A}\rightarrow(r_{A},\theta_{A})$ and $\bm{x}_{B}\rightarrow(r_{B},\theta_{B})$, where $r_{\cdot}$ denotes radius and $\theta_{\cdot}$ denotes angle. Then, we compute the angle between normalized locations as follows:

$$\theta:=(\theta_{A}-\theta_{B})\textrm{ mod }\pi.$$

(3)

Note that by its definition, $0\leq\theta\leq\pi$. Finally, we form a feature vector associated with locations $\bm{x}_{A}$ and $\bm{x}_{B}$ as follows: $\bm{V}=[r_{A},r_{B},\theta]^{T}$. We chose a regression Multi-Layer Perceptron (MLP) to estimate the 3D-world distance between people (in lieu of a CNN) since the input vector is a 3-vector with no required ordering of coordinates for which convolution would be beneficial. We collected a training set of images, where for each vector $\bm{V}$ we know the ground-truth distance $d_{AB}$, and trained the MLP, $F:\mathbb{R}^{3}\mapsto\mathbb{R}$, as a regression model that performs the following mapping:

$$\widehat{d}_{AB}=F(\bm{V}).$$

(4)

In training, we used the mean squared-error (MSE) loss:

$$\mathcal{L}=\frac{1}{M}\sum_{i=1}^{M}||\widehat{d}_{AB_{i}}-d_{AB_{i}}||^{2},$$

(5)

where $M$ is the batch size.

While the geometry-based approach can be tuned for specific height of a person through $P_{z}$ (1), the neural-network approach would require a training dataset having a large number of annotated examples at multiple heights. Since this is extremely labor intensive, we train the MLP at a single height of 32.5 inches (details in Section 4.1) which corresponds to one-half of $H=$ 65 in, an average person’s height. Recall that we assume a person’s location in the image is the center-point of the person’s bounding box. Therefore, for a standing, fully-visible 65-inch person this center-point matches the 32.5-inch training height well. However, there would be a mismatch for people of other heights or when a person is partially occluded, for example by a table. In the latter case, the detected bounding box would be above the table and so would be the bounding-box center. To compensate for this height mismatch between the training and testing data, we propose a test-time adjustment in the MLP approach.

This height adjustment can be thought of as lowering the center-point of a person in terms of pixel coordinates. An illustration of this idea is shown in Fig. 1 where the red point represents the center of the red bounding box and $h$ its height. In the process of height adjustment during test time, we move the actual center (red point) of the bounding box along the box’s axis pointing to the center of the image (white-dashed line) to produce an adjusted center (green point). This displacement is defined as $\alpha\times\frac{h}{2}$ and we consider a range of values for $\alpha$ (see Fig. 5). A value of $\alpha>0$ corresponds to moving the bounding-box center towards the image center, i.e., we reduce the height of a detected person.

In order to train the MLP, we need ground-truth distance data. We placed a 9 ft $\times$ 9 ft chessboard mat on classroom tables of equal height (32.5 inches) in 19 different locations as shown in Fig. 3. The mat at location #2 was carefully placed with sides parallel to those of the mat at location #1 and forming a contiguous extension of mat #1 (as if two mats were placed abutting each other). Similarly, the mat at location #3 was placed contiguously with respect to the mat at location #2 and so on until location #8. The same procedure was repeated for locations #9-#16 with the mat at location #9 carefully aligned with the mat at location #1 and the distance between them carefully measured (121.5 inches). In order to provide ground-truth data in the center of camera’s field of view, the mat was also placed directly under each of 3 cameras (locations #17, #18, #19 shown in orange) with no alignment to mats at other locations.

All the black/white corners of chessboard images were annotated, resulting in numerous $(\bm{x}_{A},\bm{x}_{B})$ pairs. Since we know that each square is of size 12.5 inches and the neighboring chessboards are abutting and aligned, we could accurately compute the 3D distances between physical-mat points corresponding to $\bm{x}_{A}$ and $\bm{x}_{B}$ The overall process can be thought of as creating a virtual grid with 12.5-inch spacing placed 32.5 inches above the floor of the classroom.

In order to test both of the proposed approaches, we collected a dataset with people in a 72 ft $\times$ 28 ft classroom. First, we marked locations on the floor where individuals would stand (Fig. 4). We measured distances between all locations marked by a letter (green disk) which gives us ${10\choose 2}=45$ distances which turned out to be distinct. For locations marked by a number (yellow squares), we measured the distances along the dashed lines (20 distinct distances). Using this spatial layout, we collected and annotated two sets of data.

• •

  Fixed-height dataset: One person of height $H=$ 70.08 inches moves from one marked location to another and an image is captured at each location. This allows us to evaluate our algorithms on people of the same known height.

• •

  Varying-height dataset: Several people of different heights stand at different locations in various permutations to capture multiple heights at each location. We use this dataset to evaluate sensitivity of our algorithms to a person’s height variations.

In addition to the 65 distances ($45+20$), we performed $8$ additional measurements for the fixed-height dataset and $2$ additional measurements for the varying-height dataset.

Depending on their location with respect to the camera, a person may be fully visible or partially occluded (e.g., by a table or chair). In order to understand the impact of occlusions on distance estimation, we grouped all the pairs in both testing datasets into 4 categories as follows:

• •

  Visible-Visible (V-V): Both people are fully visible.

• •

  Visible-Occluded (V-O): One person is visible while the other one is partially occluded.

• •

  Occluded-Occluded (O-O): Both people are partially occluded.

• •

  All: All pairs, regardless of the occlusion status.

Table 1 shows various statistics for both datasets: the number of pairs in each category, the number of distances measured and their range as well as the number of pairs with distance in three ranges: 0ft–6ft, 6ft–12ft and $>$12ft.

In both datasets, we found bounding boxes of people using a state-of-the-art people-detection algorithm for overhead fisheye images [6] from which we computed their locations (bounding-box centers). To measure the real-world distances between people, we used a laser tape measure.

In the geometry-based approach, to learn parameters ${\bm{\omega}}$ of the inverse mapping $G$ (1) we used the method described by Bone et al. [3]. This method requires the use of 2 fisheye cameras, but is largely automatic and has to be applied only once for a given camera type (model and manufacturer). In the experiments, we used one camera at a time (3 cameras are installed in the test classroom – locations #17-#19 in Fig. 3) and report the results only for the center camera due to space constraints. Results for other cameras are similar.

In the neural-network approach, we used an MLP with 4 hidden layers and 100 nodes per layer. In training, we used MSE loss (5) and Adam optimizer with 0.001 learning rate.

In Tables 2 and 3, we compare the performance of both methods when estimating the distance between people on the fixed-height and varying-height datasets, respectively. We report the mean absolute error (MAE) between the estimated and ground-truth distances:

$$\textrm{MAE}=\frac{1}{N}\sum_{i=1}^{N}|\widehat{d}_{AB_{i}}-d_{AB_{i}}|$$

(6)

where $N$ is the number of pairs in the dataset while $\widehat{d}_{AB_{i}}$ and $d_{AB_{i}}$ are the estimated and ground-truth distances for the $i$-th pair $AB$, respectively.

It is clear from Table 2 that the geometry-based approach using $H/2=$ 35.04 inches (to compute $P_{z}$) consistently outperforms the same approach using $H/2=$ 32.5 inches, which, in turn, significantly outperforms the neural-network approach trained on chess mats placed at the height of 32.5 inches. While it is not surprising that knowing a test-person’s height of $H=$ 70.08 inches improves geometry-based method’s accuracy, it is interesting that even assuming $H/2=$ 32.5 inches the geometry-based approach significantly outperforms the MLP optimized for a fixed height of 32.5 inches during training.

Similar performance trends can be observed in Table 3 for the varying-height dataset but with larger distance-error values than in Table 2. This is due to the fact that in the varying-height dataset people have different heights, so a selected parameter $H$ in the geometry-based algorithm or a training height in the neural-network algorithm cannot match all people’s heights at the same time.

Note that for two fully-visible people of the same and known height (Table 2), the geometry-based algorithm has an average distance error of less than 10 inches. This error grows to about 21 inches for all pairs (visible and occluded). For people of different and unknown heights (Table 3), the average error for pairs of fully-visible individuals (for $H/2=$ 35.04 inches) is slightly above 1 ft and for all pairs it is less than 3 ft. While these might seem to be fairly large distance errors, one has to note that the distances between people are as large as 58.5 ft (702 inches).

In terms of the computational complexity, on an Intel(R) Xeon(R) CPU E5-2680 [email protected] both algorithms can easily support real time operation although the geometry-based algorithm is significantly faster. For example, suppose 3D-world distances are to be computed between all pairs of 100 image locations. The geometry-based algorithm can first map all pixel coordinates to 3D world coordinates (1) and then compute the Euclidean distance for all ${100\choose 2}=$ 4,950 pairs. This, on average, takes 4 $\mu s$. The neural-network algorithm has to apply the MLP to all 4,950 pairs separately taking on average 949 $\mu s$.

As we discussed in Section 3.3, the centers of the detected bounding boxes may not reflect the true height of a person due to occlusions. In this context, we proposed a method to adjust a bounding-box center location during testing to compensate for the occlusion effect.

Here, we evaluate the impact of this height adjustment on each method’s performance. For a fair comparison, we use $H/2=$ 32.5 inches (table height) in the geometry-based method. Recall that the neural-network approach was trained on chess mats placed at this height.

The value of MAE as a function of height-adjustment parameter $\alpha$ is shown in Figs. 5(a) and 5(b) for the fixed-height dataset. When the true bounding-box centers are used ($\alpha=0$), the MAE for the neural-network approach and V-V pairs (blue line) is close to 20 inches. However, when the centers are lowered by about 10% ($\alpha\approx$ 0.10), the MAE for V-V pairs drops to about 9 in. Similar trends can be observed in Figs. 5(c) and 5(d) for the varying-height dataset. If the true bounding-box centers are used, the MAE for V-V pairs is above 20 inches for the neural-network approach. However, when the centers are lowered by about 15% ($\alpha\approx$ 0.15), the MAE for V-V pairs drops to around 12 inches. Analogous trends can be seen for other types of pairs and for all pairs, as well as for the geometry-based approach.

In Tables 4 and 5, we show the lowest MAE values for each pair type along with the corresponding value of $\alpha$. The two methods perform quite similarly (except for O-O pairs in the fixed-height dataset on which the geometry-based method performs better). For example, for the fixed-height dataset in Table 4 MAE for the best $\alpha$ for V-V pairs drops to about 9 inches for both algorithms compared to 12-18 inches seen in Table 2. Overall, in both datasets, with the right choice of $\alpha$, the MAE is well below 2 ft, which can be argued to be a reasonable result considering that the inter-people distances in our dataset are up to 58.5 ft.

Looking at Fig. 5 and Tables 4-5, one notes that MAE is minimized for much smaller values of $\alpha$ for V-V pairs ($\alpha=$ 0.08-0.17) than for O-O pairs ($\alpha=$ 0.32-0.51). This is because the majority of occlusions happen in the lower half of people’s bodies in the testing dataset. When a person is blocked in the bottom half, the bounding-box center radially shifts away from the image center. An example of this can be seen in Fig. 1, where the person delineated by the yellow bounding box would have been delineated by the blue bounding box had there been no occlusion. Due to the occlusion, however, the bounding-box center shifts from the blue point to the yellow point. Therefore, the O-O pairs need to be compensated more than the V-V pairs, i.e., a higher value of $\alpha$ is needed.

In results reported thus far, the same value of $\alpha$ was used for both people in every pair. In the V-O and ‘All’ categories, however, it could be advantageous to use different values of $\alpha$ for the visible and occluded person. To verify this hypothesis, we applied $\alpha=0.1$ to all visible bounding boxes and $\alpha=0.5$ to all occluded bounding boxes in the fixed-height dataset. This $\alpha$ adjustment per person gave an MAE of 12.80 inches (down from 18.10 inches) for the geometry-based algorithm and 11.06 inches (down from 19.27 inches) for the neural-network approach. The corresponding MAE values for the varying-height dataset were: 13.97 inches (down from 21.48 inches) and 13.14 inches (down from 21.47 inches). Clearly, an automatic detection of body occlusions and a suitable adjustment of parameter $\alpha$ can further improve the distance estimation accuracy. This could be a fruitful direction for future work.

One very practical application of the proposed methods is to detect situations when social-distancing recommendations (typically 6 ft) are being violated. This problem can be cast as binary classification: two people closer to each other than 6 ft are considered to be a “positive” case (violation takes place) whereas two people more than 6 ft apart are considered to be a “negative” case (no violation). To measure performance, we use Correct Classification Rate (CCR) and F1-score. Table 6 shows their values for both algorithms applied in this context to “All” pairs. We report results for $\alpha=$ 0.5 which gives the lowest MAE for pairs with occlusions on the varying-height dataset (Table 5).

On the fixed-height dataset, the neural-network approach slightly outperforms the geometry-based algorithm: by 1.37% points in terms of CCR and by 0.63% points in terms of F1-score. The methods perform identically on the varying-height dataset, achieving CCR value close to 95% and F1-score close to 80%. These results suggest that despite the presence of people of different heights both approaches achieve high enough CCR and F1-score values to be potentially useful in practice for the detection of social-distance violations in the wild.