Multi-view data capture for dynamic object reconstruction using handheld augmented reality mobiles

To reduce the network traffic generated by trigger counts and to promote deployment (e.g. 802.11, 5G), we design a Relay Server to forward triggers received by the host to the clients. Relay-based architectures have the disadvantage of increased latency compared to peer-to-peer ones Hu2016 , but we practically observed that they ease deployment, especially in networks that use Network Address Translation. For timely delivery of the triggers, we employ a relay mechanism that is typically used for collaborative virtual environments Pretto2017 and teleoperation Yin2019 , namely MLAPI mlapi_relay . MLAPI uses UDP to avoid delays caused by the flow-control system of TCP. Our Relay Server is based on the concept of Multi-access Edge Computing and exploits the design of MLAPI. Its architecture and synchronisation protocols are detailed in Bortolon2020 .

Throughput linearly increases with the number of mobiles until network congestion is reached. We observed that four mobiles transmitting at 10fps on a conventional 2.4GHz WiFi suffice to reach maximum bandwidth. Although HTTP-based applications are network-agnostic, they are unreliable with large throughputs. When the maximum bandwidth capacity is reached, numerous HTTP-connections undergo time-outs due to delayed or lost TCP packets. This leads to re-transmissions: more HTTP connections are created and more packets are dropped due to Socket buffer overflows. Hence, triggers are not delivered (as they are UDP packets), leading to partial data captures. To mitigate these problems, we design a Socket-based architecture that establishes a single connection between each mobile and the Data Manager (Fig. 3). This allows us to increase the percentage of successfully delivered packets. Specifically, after $D_{i,c}$ is captured, the frame $f_{i,c}$ is encoded in JPEG, as it provides the best trade-off between compression and encoding/decoding speed. $D_{i,c}$ is serialised protobuf , split into packets and transmitted to the Data Manager via TCP. Mobiles can eventually buffer packets if they detect network congestions. Frames can also be video encoded to reduce the throughput, but this would require additional on-board processing. Moreover, to our knowledge, there are no video encoders that can embed metadata (like mobile poses) positionable with frame precision within a video stream. The Data Manager receives the packets, merges them to restore the (original) serialised message and de-serialises the message. De-serialisation can ingest up to 5282fps on a Xeon 3.1Ghz. We associate each de-serialisation instance to a Socket. Frames are processed at 200fps by a verification module that checks their integrity. We use a queue for each socket and perform verification using multiple threads. Because frames are often disordered after verification, we re-organise them together with their meta-data (merging) before storing them in a database.

We extract the body pose of each person in $f_{i,c}\forall c\in\mathcal{C}$ Cao2017 ; wu2019detectron2 . OpenPose Cao2017 can be used to estimate the 2D skeleton of each person using up to 25 joints per skeleton. Because $\mathcal{O}$ is typically viewed in close-up on each mobile and in the centre of the frame, for each $f_{i,c}$ we score each detection as the product between the area of the bounding box enclosing the skeleton, and the inverse of the distance between the bounding box centre and the frame centre. We select the skeleton that is inclosed in the bounding box with max score. Let $H_{i,c}=\{h_{i,c}^{m}:h_{i,c}^{m}\in\mathbb{R}^{2}\}_{m=1}^{M}$ be the set of $M$ ordered skeleton joints (e.g. $M$=25) of $\mathcal{O}$ estimated in $f_{i,c}$. $h_{i,c}^{m}$ is the position of the $m$th joint. We find the 3D joints by triangulating the corresponding 2D joints from camera pairs. To compute reliable 3D joints, we use the camera pairs that have a predefined angle between their image plane normals. Specifically, given a camera pair $c,c^{\prime}\in\mathcal{C}$ such that $c\neq c^{\prime}$, the angle between their image plane normals is defined as

where $\hat{n}_{i,c}$ and $\hat{n}_{i,c^{\prime}}$ are the image plane normals of camera $c$ and $c^{\prime}$, respectively. We deem $c,c^{\prime}$ a valid pair if $\tau_{\alpha}\leq\alpha_{i,c,c^{\prime}}\leq\tau_{\alpha}^{\prime}$, where $\tau_{\alpha}$ and $\tau_{\alpha}^{\prime}$ are parameters. Let $\Gamma_{i}=\{\{c,c^{\prime}\}:c,c^{\prime}\in\mathcal{C},c\neq c^{\prime}\}$ be the set of valid camera pairs, $\forall\,\{c,c^{\prime}\}\in\Gamma_{i}$ we compute the set of visible 3D joints. We denote the $m^{th}$ 3D joint computed from $\{c,c^{\prime}\}$ as

where $\mathrm{t}(\cdot)$ is a triangulation algorithm, e.g. the Direct Linear Transform algorithm (Sec. 12.2 in Hartley ).

Because each camera pair produces a 3D point for each joint, we typically have multiple 3D points for the same joint. Let $\Omega_{i}^{m}=\{\omega_{i,c,c^{\prime}}^{m}:\omega_{i,c,c^{\prime}}^{m}\in\mathbb{R}^{3},\{c,c^{\prime}\}\in\Gamma_{i}\}$ be the set of 3D points of the $m^{th}$ joint triangulated from $\{c,c^{\prime}\}$ and let $\Omega_{i}=\{\Omega_{i}^{m}\}_{m=1}^{M}$ be the set of all triangulated 3D points. We seek a 3D point for each joint to reconstruct the 3D skeleton. Let $\mathcal{S}_{i}=\{s_{i}^{m}:s_{i}^{m}\in\mathbb{R}^{3}\}_{m=1}^{M}$ be the set of the skeleton’s 3D joints. We design a function $g:\Omega_{i}\rightarrow\mathcal{S}_{i}$ that transforms a set of 3D points into a single 3D point for each joint as follows. A 3D point seen from multiple viewpoints can be reliably computed by minimising the re-projection error through Bundle Adjustment (BA) Lourakis2009 as

where $\mathcal{A}_{i}=\{a_{i,c}:c\in\mathcal{C}\}$ and $a_{i,c}$ is a vector whose elements are the intrinsic and extrinsic parameters of camera $c$ at frame $i$, i.e. $K_{c}$, $P^{\prime}_{i,c}$, respectively. $Q(a_{i,c},s_{i}^{m})$ is the predicted projection of the joint $m$ on view $c$, and $d(x,y)$ is the Euclidean distance between the image points represented by the inhomogeneous vectors $x$ and $y$. The solution of Eq. 6 is a new set of camera parameters and 3D points. Without loss of generality we keep the camera parameters fixed and let BA determine only the new 3D points. The non-linear minimisation of Eq. 6 requires initialisation. Snavely et al. Snavely2007 use the camera parameters and the 3D points triangulated from an initial camera pair as initialisation, then BA is executed incrementally each time a new camera is added, using the new solution as initialisation for the next step (see Sec. 4.2 of Snavely2007 ). This requires multiple iterations of BA at each $i$. To reduce the processing time, we execute BA globally, once for each $i$, using the centre of mass of each joint as initialisation. The $m^{th}$ centre of mass is

where $|\cdot|$ is the cardinality of a set. If $\mu_{i}^{m}=0$, it will not be processed by BA. The solution of Eq. 6 provides $\mathcal{S}_{i}$. We will empirically show in Sec. 6.5 that initialising with the centres of mass can reduce the computational time and that BA converges to the same solution as using the incremental approach of Snavely2007 .

We extract the silhouette of each person in $f_{i,c}\forall c\in\mathcal{C}$ using instance-based segmentation He2017 ; Kirillov2020 . We use PointRend Kirillov2020 as it provides superior performance in terms of silhouette segmentation to Mask R-CNN He2017 . Let $\Psi_{i,c}$ be the silhouette of $\mathcal{O}$, which is selected at each frame as in Sec. 5.1. We compute the volume of $\mathcal{O}$ as Visual Hull (VH) Laurentini1994 , i.e. the intersection of $C$ visual cones, each formed by projecting each $\Psi_{i,c}$ in 3D through the camera centre of $c$ Cheung2003 . Because finding visual cone intersections of generic 3D objects is not trivial, VH can be approximated by a set of voxels defining a 3D volume Mikhnevich2014 . Each voxel is projected on each camera image plane and if its 2D projection is within $\Psi_{i,c}$ for all $c\in\mathcal{C}$ then this voxel is considered to belong to $\mathcal{O}$’s VH. In practice, due to noisy silhouette segmentation and/or noisy camera poses, one seeks if the voxel projection is within a subset of silhouettes rather than all. Let $\mathcal{V}_{i}=\{\nu_{i,\theta}:\theta=1,\dots,\Theta\}$ be a set of voxels, where $\nu_{i,\theta}\in\mathbb{R}^{3}$ is the centre of the $\theta$th voxel approximating a volume of size $\Delta_{\nu}\times\Delta_{\nu}\times\Delta_{\nu}$ and $\Theta$ is the total number of voxels. We fix $\Delta_{\nu}$ to keep the same spatial resolution over time. Each $\nu_{i,\theta}$ is projected on each camera’s image plane using the camera parameters $K_{c}$ and $P^{\prime}_{i,c}$ for each $c$. In our experiments we define a bounding volume composed of 160$\times$160$\times$160 voxels occupying a space of 1.8$\times$1.8$\times$1.9 meters centred in the position of the 3D skeleton’s hip joint. Let $x_{i,c,\theta}$ be the 2D projection of $\nu_{i,\theta}$ on $c$. If $\lfloor x_{i,c,\theta}\rfloor\in\Psi_{i,c}$, where $\lfloor\cdot\rfloor$ is the rounding to lower integer operation, then $\nu_{i,\theta}$ is within the visual cone of $c$. If $\nu_{i,\theta}$ is within the visual cone of at least $n_{\Psi}$ cameras, then we deem $\nu_{i,\theta}$ to belong to $\mathcal{O}$’s VH. We post-process VH with Marching Cubes to find the iso-surfaces and to produce the final mesh Lorensen1987_1 .

The mobile app is developed in Unity3D unity3d , using ARCore for SLAM arcore and the Cloud Anchor to find $T_{c}$ cloudanchors . Capture sessions are managed by Unity3D Match Making matchmaking . The Relay Server and the Data Manager are embedded in separate Docker containers, running on a laptop within the local network. In Bortolon2020 we describe our Relay Server architecture/protocols in detail. JPEG frames are serialised/deserialised using Protobuf protobuf . We use the Sparse Bundle Adjustment C-library Lourakis2009 . To collect data, we used two Huawei P20Pro, one OnePlus 5, one Samsung S9, one Xiaomi Pocophone F1 and one Xiaomi Mi8. Mobiles were connected to a 5GHz WiFi 802.11 access point. We used a Raspberry Pi 2 to route packets to the internet. Relay Server and Data Manager are deployed on an Edge computer within the same local network as the mobiles.

We collected a dataset, namely 4DM, which involves six people recording a person acting table tennis (Fig. 1). We recorded the 4DM dataset outdoors, with cluttered backgrounds, cast shadows and with people appearing in each other’s view, thus becoming likely distractors for object detection and human pose estimation. We recorded three sequences: 4DM-Easy (312 frames, all recording people were static), 4DM-Medium (291 frames, three recording people were moving) and 4DM-Hard (329 frames, all recording people were moving and we added occluders to the scene). The host generated triggers at 10Hz. Frames have a resolution of 640$\times$480 and an average size of 160KB. The latency between mobiles and the Relay Server was $\sim$5ms.

We compare our Edge solution with a Cloud-based one, i.e. AWS aws . Tab. 1 shows the percentage of successfully received data in the case of Socket and HTTP based transmissions to the Cloud and to the Edge using 2.4GHz and 5GHz WiFi. The host generated triggers at 20Hz for one minute. HTTP is as effective as Socket when the bandwidth has enough capacity, but unreliable when there is limited capacity. We observed that Socket-level data management is key to effectively manage large throughputs.

Fig. 4 shows the statistics of the time difference between consecutive frames received on the Data Manager as a function of the trigger frequency and the number of mobiles. We used up to four mobiles connected to a 2.4GHz WiFi. The host transmitted triggers at 5Hz, 10Hz and 15Hz for 30 seconds. We associated a reference time difference for each capture showing the ideal case, e.g. 10Hz$\rightarrow$0.10s. Experiments were repeated three times for each capture. We can observe that when the network bandwidth is enough, the Data Manager can receive frames at the same trigger frequency as that set by the host mobile. When the throughput exceeds the bandwidth capacity, the system starts buffering packets due to TCP re-transmissions.

To assess the end-to-end delay we used two mobiles configured in the same way as in 4DM. We adopted the same procedure as Bortolon2020 to quantify the frame capture delay between mobiles, i.e. two mobiles captured the time ticked by a stopwatch (up to millisecond precision) displayed on a 60Hz computer screen (16ms refresh rate). We extracted the time digits from each pair of frames captured by the two mobiles using OCR ocr and computed their time difference. Frames that were likely to provide incorrect OCR digits were manually corrected. For each experiment, we used the first 100 captured frames and calculated the average and standard deviation of the difference between consecutive frames. For comparison, we implemented the approach proposed in Sec. IV of Ansari2019 , i.e. the NTP is requested 50 times by each mobile and the average NTP over these requests is used as a global synchronisation between mobiles. The clock of the mobiles is aligned to this estimated global time and used to synchronise the capture of each frame. The host informs the other mobiles when to start the acquisition with respect to this estimated global time through a trigger. As we explained in Sec. 1, we cannot achieve a camera-sensor synchronisation as in Ansari2019 , because the capture of frames and the associated mobile poses depends on the SLAM process and the Unity3D rendering engine. We name this approach NTP-Sec. IV Ansari2019 . We also implemented a NTP baseline version of NTP-Sec. IV Ansari2019 , namely NTP-Baseline, where a single NTP request is used for the global time.

Tab. 2 shows the comparison between our trigger-based synchronisation method and NTP-based synchronisation approaches. We empirically measured a standard deviation of about 16ms in all cases, corresponding to the monitor refresh rate. The results show that the synchronisation using our trigger-based method achieves the same accuracy as that of NTP-based approaches. Because triggers are generated by the host, we can inherently vary their frequency online based on the application requirements. For completeness, we also include the results reported in Ansari2019 in the case of mobiles synchronised without exploiting the prioritised camera sensor access. The three “Naive” experiments used a single button press to trigger both phones to capture a frame each. The connectivity amongst mobiles was established through an audio wire, a Bluetooth connection and peer-to-peer WiFi. Note that their measured delays are much higher than ours.

Fig. 5 shows a sequence of 3D skeletons reconstructed from 4DM-Easy, while Fig. 6 reports the re-projection error using Eq. 7.

The average re-projection error (over time and across joints) is 2.86 pixels for 4DM-Easy, 2.64 pixels for 4DM-Medium and 3.11 pixels for 4DM-Hard. We can observe that BA reaches the same solution as the strategy in Snavely2007 . However, as shown in the legends of Fig. 6, the global minimisation allows us to reduce by more than thrice the computational time for each $i$. Fig. 7 shows the 3D skeleton and the mobile poses in 4DM-Hard at $i$=164, and three zoomed-in frames of the person captured from mobiles 1, 3 and 5, overlaying the re-projected 3D joints (red dots) and the 2D joints computed with OpenPose (green dots). Some of the re-projected 3D points are shifted from the 2D locations estimated with OpenPose. We attribute these errors to the mobile poses that may have been inaccurately estimated by SLAM due to the high dynamicity of the scene (moving object and moving cameras), and to synchronisation delays.

To assess the robustness of our system, we simulated different miss-detection rates affecting OpenPose output on different numbers of cameras. Miss-detection implies that the actor is not detected, hence the affected camera cannot contribute to triangulation. Fig. 8 shows the average (line) and standard deviation (coloured area) of the re-projection error. We can observe that miss-detections lead to an expected steady increase in re-projection error. We did not encounter breaking points at which the system completely fails triangulation. As long as a valid camera pair (Sec. 5.1) does not undergo miss-detections, the system can triangulate the skeleton’s joints.

BA can also jointly optimise camera parameters (i.e. mobile poses) and 3D points. Fig. 9 shows results (at $i$=143 in 4DM-Hard) by applying BA on 3D points only, and on both 3D points and mobile poses. The average re-projection error in this example is 5.35 pixels for the former and 2.23 pixels for the latter. The full sequence is available in the supplementary video. Here we can observe that the 3D skeleton is estimated similarly in the two cases, but camera poses are different: they are often positioned in implausible locations. To further investigate this issue, we fixed one mobile as world reference and optimised the others in order to enforce BA to anchor the optimisation with respect to this mobile’s pose (reference), without seeking a new coordinate re-arrangement. However, we still experienced this undesired behaviour. We believe that this happens for three reasons. First, there are similar numbers of 3D points and camera parameters to optimise. This may hinder BA in converging to the correct solution, as the structure of the environment has few 3D points. Second, mobiles may receive triggers with different delays. Delays may vary based on the quality of the wireless connection: e.g. if an external user, within the same wireless network, generates traffic (e.g. downloads a file), they will affect the network latency. Third, BA assumes 2D joints to be correct. Hence, if OpenPose fails to estimate some joints, this will then affect the minimisation process. An example is in mobile 4 of Fig. 9, where the person behind the actor has affected OpenPose estimation.

We compare the quality of our reconstructed 3D skeleton with the 3D skeleton estimated with a deep-learning-based method, namely VideoPose3D Pavllo2019 , that infers 3D skeletons directly from 2D images. VideoPose3D processes the 2D joints computed on the images of the whole sequence and outputs the respective 3D joints for each frame with respect to the mobile coordinate system. VideoPose3D exploits the temporal information to generate 3D joint locations over time. The input 2D joints are computed with Detectron2 wu2019detectron2 . Differently from OpenPose, the skeleton model of VideoPose3D uses $M$=15 joints. We use a model pretrained on Human3.6M Ionescu2014 . To make the results comparable, we input the same 2D joints estimated with Detectron2 to our system.

Fig. 10 shows the overlay between the 3D skeleton triangulated with our method (black), (a) the six 3D skeletons estimated from each mobile with VideoPose3D (green), and (b) the skeleton whose joints are computed as the centres of mass amongst the corresponding joints inferred from all the mobiles with VideoPose3D (light blue). VideoPose3D’s output skeleton is up to a scale. To overlay the skeleton obtained via triangulation with our system with that of VideoPose3D, we scaled VideoPose3D’s skeleton by making the hip centres of the two skeletons coincide (see the yellow spheres). VideoPose3D outputs a different torso and head’s representations of the input skeleton. Therefore we compute the re-projection error only for the skeletons’ joints that exist in both input and output. VideoPose3D is an offline approach as it uses temporal information: the skeleton pose in a frame is computed using past and future information. Although the skeletons that are estimated with the two methods appear with a similar pose, the joints do not coincide. In this regard Tab. 3 reports the re-projection error. The re-projection error of VideoPose3D is consistently higher than that of our system in all three setups.

Fig. 11 shows examples of volumetric reconstructions extracted from 4DM-Easy, 4DM-Medium and 4DM-Hard. The supplementary video contains the full sequences. Although we are using a fully automated volumetric reconstruction baseline, we can observe that we can achieve promising reconstruction results even when cameras are moving. Note that the quality of the reconstructed mesh is limited by the number of cameras: in fact, we can distinguish the intersection of the projected cones. This is a known problem of shape-from-silhouette based approaches. To obtain reconstructions with higher geometric details there should be a larger number of cameras or more sophisticated approaches, as those proposed in Pages2018 ; Mustafa2019 , should be used.