Superluminal Motion-Assisted 4-Dimensional Light-in-Flight Imaging

Progress in high speed imaging techniques has enabled observation and recording of light propagation dynamics in free space as well as in transparent, translucent and scattering media. Various approaches for capturing the light-in-flight have been demonstrated: holographic techniques [1, 2, 3], photonic mixers [4], time-encoded amplified imaging [5], streak cameras [6, 7], intensified CCDs [8] and silicon CMOS image sensors [9]. In recent years, 1- and 2-dimensional arrays of single-photon avalanche diode (SPAD) have been adopted for the light-in-flight imaging systems [10, 11, 12, 13, 14, 15]. These sensors boost data acquisition speed by employing pixel-parallel detection of time stamping with picosecond time resolution and single-photon sensitivity. However, the sensors are capable of sampling only 3-dimensional spatio-temporal information ($x$, $y$, $t$) of the target light, and hence the tracking of light-in-flight breaking outside a $xy$-plane is a challenge.
A method to reconstruct an extra-dimensional position information, $z$, from measured spatio-temporal information has been recently investigated by researchers [16]. The authors remark that different propagation angle with respect to $xy$-plane results in the different apparent velocity of light. Thus, comparing the measured spatio-temporal data set with this theory could give an estimation of $z$-component in the light propagation vector. Yet, the analysis is limited to a simplified case with single straight light paths within $xz$-plane passing a fixed point, and complete 4-dimensional reconstruction of light-in-flight in arbitrary paths remains to be verified. In addition, spatial resolution of the detector array is critical for accurate reconstruction of the 4-dimensional trace; the reported SPAD sensor resolutions, e.g. 64$\times$32 [15], 32$\times$32 [10] and 256$\times$1 [13] pixels, are not sufficient, whereas significant improvement of the estimation error for ($x$, $y$, $z$, $t$) is expected with orders of magnitude larger arrays, typically in the megapixels. CMOS sensors routinely achieve megapixel resolutions, e.g. 0.3 Mpixel resolution is reported for high speed imaging [9], however the necessary timing resolution is not reached and it is limited to 10 nanoseconds.
In this paper we demonstrate the 4-dimensional light-in-flight imaging based on the first time-gated megapixel SPAD camera [17]. In contrast to conventional time-correlated single-photon counting (TCSPC) approaches, where power- and area-consuming time-to-digital converter (TDC) circuits restrict the scaling of the pixel array [18, 19, 20], our time-gating approach [17] achieves a much more compact pixel circuit, suitable for large-scale arrays. Our camera achieves a megapixel format, while, at the same time ensuring a time resolution comparable to that of a TDC. Owing to an unprecedented high spatio-temporal resolution, a superluminal motion of laser pulse was successfully captured in a frame sequence. We then introduced a theoretical equation to fit the measured data set in 3-dimensional space. A 4-dimensional point cloud is computationally reproduced without prior knowledge, exhibiting a good agreement with the actual light propagation vectors.

To illustrate our method, we introduce a Minkowski space that is describing the spatio-temporal propagation of a laser pulse and scattered photons towards fixed observation points. Fig. 1(a) shows a schematic illustration of light-in-flight observation in the Minkowski space at $y$ = 0. A straight light propagation along $x$-axis is depicted as ($x$, $y$, $z$) = ($ct$, 0, 0), where $c$ is the speed of light. In the Minkowski space, this equation corresponds to a single tilted line crossing the origin (purple arrow in Figure 1(a)). A camera is located at the position $C_{i}$ ($i$ = 1,2,…,5). The observation of the light-in-flight at the camera is mediated by scattered light at each point on the path (colored dots on the purple arrow). The propagation of the scattered light from the points are described as light cones. Here we define t as the time at which the light propagating from a laser source reaches a point ($x$, $y$, $z$), called scattering point on the light path, and $t^{\prime}$ as a time at which the light scattered by the medium at ($x$, $y$, $z$) reaches the camera. Due to the finite speed of light, a time difference from $t$ to $t^{\prime}$ is proportional to the spatial distance between the scattering point and the camera location. In Minkowski space, the observation time $t^{\prime}$ can be visualized as an intersection point of the light cone and a vertical line $L$ passing $C_{i}$ (dotted line). The relation between the light position $x$ and the corresponding observation time $t^{\prime}$ is shown in Figure 1(b). The behavior is highly dependent on the camera position with respect to the scattering point, i.e. at certain points it appears to be faster than in others. For example, at $C_{1}$ where the light is coming towards the camera, the scattering light from all the points on the light path reaches the camera at the same time. The corresponding apparent velocity, defined as $v_{ap}$ = $dx/dt^{\prime}$, is infinite, or $t^{\prime}$ appears not to change irrespective of the distance of the scattering point ($x$, $y$, $z$) from the camera. At $C_{5}$, in contrast, the light propagating away from the camera is observed with the apparent velocity of $c$/2. For other scattering points corresponding to relative camera positions $C_{2}$, $C_{3}$, and $C_{4}$, one observes intermediate apparent velocities, depending on the relative position and angle of the line of sight between scattering point and camera. Hence, analyzing the apparent velocity provides an estimated light propagation vector with extra-dimensional information. The discussion can be generalized for higher dimensions; the detailed analysis of measured ($x$, $y$, $t$) data set and its local apparent velocity would enable a 4-dimensional reconstruction of the light-in-flight history. A 2D-projected intuitive visualization for the propagation of scattered light is provided in Supplementary Video 1. Note that angle-dependent apparent velocity of light can be seen in nature, for instance in an astronomical phenomenon called superluminal motion. In this case, relativistic jets of matter are emitted from radio galaxies, quasars, and other celestial objects, appearing to travel faster than light to the observer in a time scale of months to years. Light echoes are also known to produce superluminal motion [21, 22].
Fig. 1(c) is the experimental setup to verify the 4-dimensional light-in-flight imaging. A 510 nm-laser (average power: 2 mW, frequency: 40 MHz, optical pulse width: 130 ps, PicoQuant GmbH, Berlin, Germany) and the megapixel SPAD camera are synchronized by a pulse generator. The emitted laser pulses are reflected by four mirrors to construct a 3-dimensional trajectory. The megapixel SPAD camera can be operated in intensity imaging mode to capture the background scene and in time gating mode to capture laser propagation [17].

Fig. 2(a) describes the principle of time gate scanning method. A 3.8 ns global gate window is synchronized to a pulsed laser source. The pixels detect only photons impinging during the gate window. The gate position can be scanned with a gate shift of 36 ps relative to the laser pulse. At each gate position, 255 binary photon-counting frames are summed to form an 8-bit image. 250 slices of 8-bit images with the scanned time-gating window position over 9 ns range are used to extract the timing information of impinging photons. The gate window is defined by sending a short electrical pulse to the gate switch in the pixels. The data acquisition time for full gate scanning (250 slices) is 15 to 30 seconds, much shorter than in previous works (e.g. hours [6, 7] to 10 minutes [10]). This can be potentially reduced further by introducing a higher power laser.
Fig. 2(b) shows the procedure to generate a 3-dimensional light-in-flight video. The captured 250 slices of time-gating images form a photon intensity distribution for each pixel as a function of gate position. A rising edge position $\tau^{\prime}$ and “high” level of photon intensity $I$ in the intensity profile can be extracted for each pixel. Time-of-arrival information $t^{\prime}$ over the array is obtained by subtracting independently measured pixel-position-dependent timing skew from $\tau^{\prime}$. The intensity profile is given as a convolution of the arriving photon distribution and the gating profile, and the laser pulse width appears to be elongated by 3.8 ns in the raw intensity profile. Analogously to a previously reported temporal deconvolution technique [10], the laser pulse width can be narrowed to the picosecond scale by replacing the intensity profile with a Gaussian distribution having a mean value of $t^{\prime}$, standard deviation of 150 ps (corresponding to the combined jitter of laser and detector), and integrated photon counts of $I$. Fig. 2(c) shows the selected frames of a reconstructed 3-dimensional light-in-flight video, where each frame is superimposed with a 2-dimensional intensity image independently captured by the same SPAD camera. A laser pulse starts to be observed around $t^{\prime}$ = 0 ns, is reflected by multiple mirrors, and goes out of sight around $t^{\prime}$ = 5.832 ns. The laser beam at $t^{\prime}$ = 5.112 ns appears to be stretched compared to the beam at $t^{\prime}$ = 2.232 ns. This implies an enhanced apparent velocity of the beam coming towards the camera at $t^{\prime}$ = 5.112 ns. Note that our system requires no mechanical laser scanning, spatial interpolation and dark noise image subtraction owing to its high spatial resolution array and low dark count rate [17]. A full 3-dimensional light-in-flight video with the frame interval of 36 ps is available in Supplementary Video 2.

Fig. 3(a) shows a color-coded plot of the observation time $t^{\prime}$ for the propagating light. To reconstruct 4-dimensional light path, the data set needs to be subdivided into straight paths. We adopted the 2D Gaussian mixture model (GMM) fitting for data clustering, which is a common unsupervised machine learning technique. Fig. 3(b) shows the data points separated into five clusters. Note that the data clustering is performed based on 2-dimensional spatial information ($x$ and $y$), whereas the time information $t^{\prime}$ is not taken into account.
Fig. 3(c) depicts the coordinate system for the light-in-flight analysis. Here we derive a theoretical formula to perform the least square regression for estimation of 4-dimensional light trajectory:

where $N$ is the total number of measurement data points, ($x_{p}^{i}$, $y_{p}^{i}$) the corresponding pixel position on the focal plane for $i$-th data point, $-z_{p}$ the focal length (12.5mm), $t^{\prime i}$ the measured observation time for $i$-th data point, $\vec{r}_{0}=(x_{0},y_{0},z_{0})$ the position of the beam at $t$ = 0, and the normalized light propagation vector is represented as $\vec{n}=(\sin{\theta}\cos{\phi},\sin{\theta}\sin{\phi},\cos{\theta})$ using a polar coordinate system. A single straight light trajectory can be expressed as $\vec{r}=\vec{r_{0}}+ct\cdot\vec{n}$, defined by five fitting parameters, $x_{0}$, $y_{0}$, $z_{0}$, $\theta$, $\phi$. Those parameters are estimated by solving the above optimization problem for each data cluster.
Fig. 3(d) and 3(e) show the measured observation time $t^{\prime}$ as a function of $x_{p}$ and $y_{p}$, where ($x_{p}$, $y_{p}$) is the position of the pixel on the focal plane. Different colors of the data points correspond to the different data clusters. The result of fitting for each data cluster is shown as a dashed line, exhibiting a good agreement with the measured points for both $x_{p}$ and $y_{p}$. Note that the fitting involves five independent parameters and is sensitive to the variation of the data. Hence, the spatio-temporal resolution of the camera is a critical factor determining the accuracy of the 4-dimensional reconstruction.
As shown in Fig. 4(a), the 4-dimensional point cloud is reconstructed without prior knowledge. The time evolution t of the beam is shown as colors of points, having a certain offset from the observation time $t^{\prime}$ due to the finite traveling time from the scattering point to the camera. The fully-recovered 4-dimensional information for each point enables the visualization of the point cloud from arbitrary viewpoint apart from the actual location of the SPAD camera. In contrast to the 3-dimensional light-in-flight video where the apparent velocity of light changes as a function of propagation angle, the 4-dimensional point cloud indicates the uniform speed of light irrespective of the beam position and angle. Relatively larger variation of the data points is observed for cluster 4 and 5. This originates from the enhanced apparent velocity of light coming towards the camera, leading to the reduced fitting accuracy and increased deviation.
The accuracy of the reconstruction is evaluated in Fig. 4(b) and  4(c). The figures show the fitted light propagation angles $\theta$ and $\phi$ as a function of the actual geometrical angles. Five dots correspond to the fitting results of the five data clusters. All the dots are along with the ideal trend (dashed line), indicating that the fitting results are in good agreement with the actual light propagation vectors.
Detailed analysis of the reconstruction performance for various parameter combinations is investigated based on Monte Carlo simulation in the Supplementary Note. The simulation shows that with the horizontal resolution of 1024 pixels, the estimation errors of the fitting parameters improve by a factor of 6 to 9 with respect to previously reported 32$\times$32 pixel SPAD cameras, thereby justifying the necessity of megapixel resolution for high-precision 4-dimensional light-in-flight imaging. The results also suggest that a further increase of the pixel array size could have a non-negligible impact on the stability and accuracy of light-in-flight measurements, along with the potential of further expansion of the measurement in distance and field-of-view.

Our approach of reconstructing extra-dimensional light-in-flight information with high spatio-temporal resolution single-photon camera can be applied to a wide range of high-speed imaging techniques. One of the promising applications is the non-line-of-sight imaging [23, 24, 25, 26, 27]; by extending our approach for concentrated linear beam towards a diffused light, hidden objects can potentially be imaged with more simplified setup. The technique could also be used for imaging around and behind obstructing objects. Another potential application is the combination with optical tomography technique [28, 29, 30]; generalization of our theory can be useful for non-invasive 3-dimensional monitoring inside a target object structure as well as non-destructive measurement of 3-dimensional distribution of physical parameters such as refractive index and transmittance.