High-speed multiview imaging approaching 4pi steradians using conic section mirrors: theoretical and practical considerations

Angular diversity is important in a wide variety of imaging techniques as a means for enhancing not only lateral resolution, but also axial resolution to enable high-resolution 3D imaging. Perhaps the most familiar examples are standard wide-field or point-scanning microscopy, whose 3D spatial resolutions improve when using objectives with increasing numerical aperture (NA). Whether detecting coherent scattering or incoherent fluorescence, higher NA objectives provide access to higher illumination or collection angles, which contain higher-spatial-frequency information. In these cases, the information from all angles is present simultaneously. A variation of this is imaging based on speckled illumination, whose theoretical resolution is proportional to speckle grain size, which in turn depends on angular diversity. Other techniques computationally combine images sequentially acquired from multiple illumination angles to create a synthetic aperture, thereby reconstructing a 3D or resolution-enhanced image. Examples include optical diffraction tomography (ODT) [1, 2, 3, 4], Fourier ptychographic microscopy (FPM) [5, 6, 7], and structured illumination microscopy (SIM) [8, 9], including with speckled illumination [10, 11, 12]. Both the illumination and collection angles may also be varied, as in optical projection tomography (OPT) [13], optical coherence projection tomography (OCPT) [14], and optical coherence refraction tomography (OCRT) [15, 16], which can be achieved through rotation of the sample or source-detector pair. While OPT and OCPT are transmissive modalities, OCRT is reflective and thus illumination steering automatically steers the detector. Finally, yet another class of related techniques, including photogrammetry, structure-from-motion (SfM), and multi-view stereo (MVS) [17, 18, 19, 20, 21], reconstructs a 3D surface using a sequence of non-telecentric images (e.g., from standard consumer-grade cameras) acquired from multiple angles.

In all of these techniques, the wider the angular coverage, whether across a physical or synthetic aperture, the better the theoretical resolution. At the same time, however, the wider the angular coverage, the more difficult it is to achieve, especially when approaching $\pm$90^∘. While high-NA objectives are commonly used, they are in practice limited to approximately $\pm$70^∘ (i.e., NA=0.95 in air objectives) over small fields of view (FOVs) and only allow multi-angle illumination of samples from one side. As such, techniques like OPT and photogrammetry typically employ mechanical rotation of the sample or detector, which sacrifices temporal resolution or require contrived sample preparation.

Here, we propose several different designs that can achieve multi-angle imaging approaching 4pi steradians, without the need for rotating the sample or source-detector pair. Our proposed designs achieve this by employing low rotational inertial scanning (e.g., of galvanometers) across the apertures of conic section mirrors, such as parabolic, ellipsoidal, or axicon mirrors. Conceptually equivalently, we can altogether avoid mechanical scanning by using a multi-camera array design [22, 23, 24]. Parabolic and ellipsoidal mirrors have been previously incorporated in a wide variety of imaging systems, such as confocal microscopy [25, 26], multiphoton microscopy [27], total internal reflection fluorescence (TIRF) microscopy [28, 29], Raman imaging [30], photoacoustic imaging [31], FPM [32], and the recently proposed random access microscopy [33]. While the primary attractive feature of these conic section mirrors is their access to very wide angular ranges and their ability to focus “perfectly” at very high NAs, they exhibit well known tilt aberrations that restrict high-NA focusing to very small regions. As a result, many of these imaging techniques require physically translating the sample to form an image [25, 26, 28, 30] or use conic section mirrors only for illumination [31, 29, 32]. Few techniques use these mirrors for 2D image formation as an imaging objective, which requires using tilted rays that lead to aberrations. This usage is of primary interest and explored in detail in this paper.

To this end, building upon previous works that study the effects of input illumination tilt on focus quality [34, 35, 36, 37, 38], we carefully characterize the achievable FOVs over which near-diffraction-limited performance is achievable as a function of various experimental parameters (key equations: Eqs. 24 and 27). These parameters include NA, mirror parameters (e.g., focal length, major and minor axes), sample-incident angle, wavelength, and chief ray geometry (i.e., whether a telecentric design is used). Crucially, we will argue that for 3D imaging applications, the well-known tilt aberrations do not significantly limit the lateral FOV, any more than the depth of field limits the axial FOV. This work lays the theoretical foundation that generalizes the parabolic-mirror-based 3D imaging method for which we recently presented preliminary results [39].

We first describe the general strategies for imaging over wide angular ranges, common to all conic section mirror shapes considered here, which achieve similar results slightly differently (Fig. 1). Specifically, our goal is to acquire 2D (or 3D) images from multiple views over a very wide angular range, whether with active illumination via point-scanning or with full-field detection with a 2D camera. Our explanations proceed based on a point-scanning system, with the understanding that similar rays govern a 2D-camera-based imaging system (Sec. 7.2). Further, this concept is applicable to both reflection and transmissive imaging geometries (Sec. 7.3).

To obtain multi-view imaging over 360^∘ about a single axis, we propose using a rotationally symmetric, concave mirror as a focusing element that reflects incident beams and weakly focuses them at a focal point, where the sample is placed (Fig. 1). Since the mirror is rotationally symmetric, varying the incidence position along a circle concentric to the mirror aperture allows varying the incidence angle to the sample to cover the full 360^∘. An image is formed by independently scanning the focus laterally, the mechanism of which depends on the mirror properties.

This idea can be straightforwardly extended to two rotational axes by simply allowing the entry position across the mirror aperture to vary independently over two dimensions. The angular range of the second rotation axis is asymmetric and slightly more limited, depending on the aperture size relative to the focal length. In particular, as the entry position approaches the optic axis, the sample-incident angle approaches 90^∘, which may be clipped if the incident beam is blocked by the sample before hitting the mirror. Similarly, as the entry position moves away from the optic axis, the incidence angle increases in the opposite direction, and achieves a maximum angle limited by the mirror aperture size or input scanning mechanism. Thus, if the sample is small and the aperture is very large, nearly 4pi steradians of coverage is theoretically possible. In practice, sample mounting constraints may restrict the range to, for instance, $\pm$90^∘ or 2pi steradians, if we strictly only have access to one side of the sample (Fig. 1c).

A number of system designs can achieve these aforementioned multi-angle imaging properties, but we focus on two concave conic section mirrors, parabolic and ellipsoidal mirrors, due to their commercial availability and conceptual simplicity. In the following sections, we discuss in detail the properties of these two mirrors and how they can achieve multi-view imaging over wide angular ranges.

Parabolic (technically, paraboloidal) mirrors are often thought of as “ideal” focusing elements for collimated beams. Unlike the more common spherical optics, they can achieve aberration-free, diffraction-limited focusing of untilted (parallel to the optic axis) beams, regardless of the input beam diameter or position. A parabolic mirror surface $P$ can be parameterized by a single value, the focal length $f$, as

$$P(r)=\frac{r^{2}}{4f},$$

(1)

where $r$ is the radial entry position along the mirror aperture. By definition, $f$ is the distance from the mirror apex to the nominal focus, where all collimated, untilted incident rays converge and where the sample is placed. In particular, the sample-incident angle in the $rz$-plane, relative to the $z$-axis (the mirror optic axis), after reflection off of the mirror (Fig. 2a), is given by

$$\theta_{\mathit{rz}}(r)=2\tan^{-1}\left(\frac{r}{2f}\right).$$

(2)

Thus, the maximum inclination angular range is restricted by the mirror aperture size (relative to its focal length). Note that the distance between the focus and where a ray intersects with the mirror surface also varies as a function of $r$, meaning that the lateral resolution varies with $r$. Thus, it is useful to define this distance as the effective focal length,

$$f_{\mathit{eff}}(r)=f+\frac{r^{2}}{4f}.$$

(3)

The effective output NA in air is thus given by

$$\mathit{NA}(r)=\sin\left(\frac{\theta_{\mathit{rz}}(r+w/2)-\theta_{\mathit{rz}}(r-w/2)}{2}\right)\approx\sin\left(\frac{w}{2f_{\mathit{eff}}(r)}\right)\approx\frac{w}{2f_{\mathit{eff}}(r)},$$

(4)

where $w$ is the input beam diameter. In both approximations, we assume that $w\ll f$, first in the argument of the sine, and then in the overall expression. In this small-angle regime, the $\mathit{NA}$ of the parabolic mirror is a simple function of $f_{\mathit{eff}}$. For the single-axis 360^∘ imaging configuration described in Sec. 2, the incidence position is varied azimuthally with a fixed $r$ and therefore fixed $f_{\mathit{eff}}$ and $\mathit{NA}$. In particular, when the special case of

$$f_{\mathit{eff}}^{90^{\circ}}=2f$$

(5)

is met, the sample-incidence angle $\theta_{\mathit{rz}}=90^{\circ}$, meaning the multi-angle illumination central chief rays sweep out a plane. However, as the second rotation axis is achieved by varying $r$, the resolution necessarily varies as well, assuming $w$ is not dynamically adjusted.

So far, we have only described focusing of incident beams parallel to the optic axis, such that the beams always converge at the mirror’s nominal focus. To scan the focus laterally and form a 2D image, we scan the beam’s incident angle to the mirror aperture. The pivot point of this beam angle scanning determines whether the lateral scanning is telecentric (parallel chief rays) or non-telecentric (fanned chief rays, as in the human visual systems and most consumer cameras). To understand this point, it’s instructive to think of the parabolic mirror as part of a 4f imaging relay that includes a second focusing element, a tube lens with focal length $f_{\mathit{tube}}$, to assist in input beam angular scanning (Fig. 2b). The 4 focal lengths are thus $f_{\mathit{eff}}+f_{\mathit{eff}}+f_{\mathit{tube}}+f_{\mathit{tube}}$, meaning a spacing of

$$d(r)=f_{\mathit{eff}}(r)+P(r)+f_{\mathit{tube}}$$

(6)

between the parabolic mirror apex and the tube lens’ principal plane. Note that telecentricity can only be achieved for one value of $r$, and therefore one sample-incident angle $\theta_{\mathit{rz}}$ and $f_{\mathit{eff}}$, without adjusting element spacing. A pair of beam-steering elements can be placed at image and Fourier planes (conjugate and anti-conjugate planes) of the sample to independently vary sample-incident position and angle (Fig. 2b), as described previously [40]. For example, one might use 2D MEMS scanners or pairs of galvanometers, either imaged onto each other or slightly offset around the same focal plane. Another approach is to use a lens array, which achieves the same effect as scanning across the back aperture.

We can specify this simple imaging system’s incident-angle-dependent theoretical magnification, as dictated by its 4f configuration, as

$$M(r)\approx\frac{f_{\mathit{tube}}}{f_{\mathit{eff}}(r)},$$

(7)

which is important for choosing scanning mirror sizes or sensor sizes in the case of non-point-scanning systems. However, the maximum achievable lateral FOV is much more challenging to analyze theoretically through basic optics principles, which is also generally the case for high-NA objectives. This is because the lateral FOV depends on aberrations, which are not present for the perfect focusing of untilted beams, but get progressively worse as the beam is tilted and scanned laterally. As we will show in a full analysis below (Sec. 5), this fall-off in focusing quality depends on a variety of factors, including NA, the wavelength, telecentricity, and both $f$ and $f_{\mathit{eff}}$, independently.

Ellipsoidal mirrors are also considered “perfect” focusing elements for diverging fields originating from the other focus of the mirror. Specifically, a symmetric ellipsoidal mirror $E$ can be parameterized by two values, the semi-major and semi-minor axes, $a$ and $b$ (Fig. 3a), as

$$E(r)=\pm a\sqrt{1-\frac{r^{2}}{b^{2}}},$$

(8)

where $r$ is the coordinate along the minor axis, and the two halves (from the $\pm$) each envelope one of the two foci, which are located at

$$f_{\pm}=\pm\sqrt{a^{2}-b^{2}},$$

(9)

relative to the center of the ellipse. All rays originating from one focus intersect with the other focus, where the sample of interest is placed, after reflecting off of the mirror, with a fixed propagation distance of $2a$. If a ray originating from one focus is launched at an angle $\theta_{\mathit{rz}}^{\mathit{in}}$, defined relative to the $z$-axis, the distance the ray travels to the mirror surface is given by

$$d_{\mathit{in}}(\theta_{\mathit{rz}}^{\mathit{in}})=\frac{a(1-e^{2})}{1-e\cos(\theta_{\mathit{rz}}^{\mathit{in}})},$$

(10)

where $e=f_{+}/a$ is the eccentricity of the ellipsoid. Thus, it is straightforward to calculate the distance from the mirror to the other focus as

$$d_{\mathit{out}}(\theta_{\mathit{rz}}^{\mathit{in}})=2a-d_{\mathit{in}}(\theta_{\mathit{rz}}^{\mathit{in}}).$$

(11)

Thus, an ellisoidal mirror can be thought of as a non-telecentric, finite-conjugate imaging system, with an effective focal length governed by the lens equation,

$$\frac{1}{f_{\mathit{eff}}(\theta_{\mathit{rz}}^{\mathit{in}})}=\frac{1}{d_{\mathit{in}}(\theta_{\mathit{rz}}^{\mathit{in}})}+\frac{1}{d_{\mathit{out}}(\theta_{\mathit{rz}}^{\mathit{in}})}$$

(12)

Further, the output angle $\theta_{\mathit{rz}}^{\mathit{out}}$ can be calculated using the law of sines as

$$\theta_{\mathit{rz}}^{\mathit{out}}(\theta_{\mathit{rz}}^{\mathit{in}})=\begin{cases}\pi-\sin^{-1}\left(d_{\mathit{in}}/d_{\mathit{out}}\sin\left(\theta_{\mathit{rz}}^{\mathit{in}}\right)\right)&d_{\mathit{out}}\leq\frac{b^{2}}{a}\\ \sin^{-1}\left(d_{\mathit{in}}/d_{\mathit{out}}\sin\left(\theta_{\mathit{rz}}^{\mathit{in}}\right)\right)&d_{\mathit{out}}>\frac{b^{2}}{a}\end{cases},$$

(13)

where the switching condition of $d_{\mathit{out}}=b^{2}/a$ corresponds to a $\theta_{\mathit{rz}}^{\mathit{out}}=90^{\circ}$ incidence to the sample at the second focus (under this condition, $d_{\mathit{out}}$ is known as the semilatus rectum). This occurs when

$$\theta_{\mathit{rz}}^{\mathit{in,90^{\circ}}}=\sin^{-1}\left(\frac{b^{2}}{2a^{2}-b^{2}}\right)$$

(14)

For a given mirror, the maximum inclination angular range is theoretically unbounded, but in practice restricted by the angular scan range of the galvanometers at the other focus. By Eq. 12, we have

$$f_{\mathit{eff}}^{90^{\circ}}=\frac{b^{2}(2a^{2}-b^{2})}{2a^{3}}.$$

(15)

In other words, this configuration corresponds to the 360^∘ single-rotation-axis imaging configuration analogous to Eq. 5 for parabolic mirrors.

To find how input $\mathit{NA}_{\mathit{in}}=\sin(\alpha)$ at the first focus maps to the output $\mathit{NA}$ at the second focus, we can use Eq. 13 to propagate the marginal rays, yielding

$$\mathit{NA}(\theta_{\mathit{rz}}^{\mathit{in}})=\sin\left(\frac{|\theta_{\mathit{rz}}^{\mathit{out}}(\theta_{\mathit{rz}}^{\mathit{in}}-\alpha)-\theta_{\mathit{rz}}^{\mathit{out}}(\theta_{\mathit{rz}}^{\mathit{in}}+\alpha)|}{2}\right)\approx\sin\left(\tan^{-1}\left(\frac{d_{\mathit{in}}}{d_{\mathit{out}}}\tan(\alpha)\right)\right)\approx\frac{d_{\mathit{in}}}{d_{\mathit{out}}}\alpha,$$

(16)

where the first approximation assumes that the left and right marginal rays travel the same distance to the mirror (and the same distance from the mirror to the second focus), and the second approximation assumes a small $\alpha$.

Thus, just like parabolic mirrors, ellipsoidal mirrors can also achieve multi-view imaging over 360^∘ single-axis or over two axes with the same physical constraints. One practical advantage of ellipsoidal mirrors is that they behave like finite-conjugate imaging systems and thus do not require a tube lens, while parabolic mirrors behave more like 4f imaging systems and thus require an extra tube lens. Nevertheless, it is possible to achieve object-side telecentricity through a judicious choice of lenses and their spacings (Fig. 3b). In particular, we can use two lenses (lenses a and b) in an almost-4f configuration, varying the third f ($d_{t}$) to achieve two intermediate focal planes: one where the marginal rays converge and the other where the central chief rays converge. The former corresponds to the other focal point of the ellipsoidal mirror (the one without the sample), where the sample-incident-angle-scanning galvanometers ($G_{\mathit{\theta\phi}}$) are placed. The latter focal plane is $f_{\mathit{eff}}$ (Eq. 12) away from the ellipsoidal mirror surface for object-side telecentric scanning. The $d_{t}$ that achieves this by satisfying all the imaging conditions can be shown to be

$$d_{t}=\frac{f_{b}(d_{\mathit{in}}^{2}+d_{\mathit{in}}f_{b}+d_{\mathit{out}}f_{b})}{d_{\mathit{in}}^{2}}.$$

(17)

Finally, we can specify the approximate magnification onto the galvo placed at the other ellipsoidal focus, based on the thin-lens model:

$$M(\theta_{\mathit{rz}}^{\mathit{in}})\approx\frac{d_{\mathit{in}}(\theta_{\mathit{rz}}^{\mathit{in}})}{d_{\mathit{out}}(\theta_{\mathit{rz}}^{\mathit{in}})}.$$

(18)

As with the parabolic mirror case, we will defer discussion of FOV to Sec. 5.

The FOV can be defined as the spatial lateral (or axial) range over which aberrations are “acceptably” small. We adopt the criterion where the Strehl ratio is greater than 0.8 (1 = aberrationless), though in practice lower values may still be usable at the cost of signal-to-noise ratio (SNR) and resolution. While FOV cutoffs are in general arbitrary (just like for resolution), their dependence on other system parameters remain well defined. Perhaps a familiar example of this is arbitrarily defining the axial FOV by the depth of field of a camera or depth of focus of a point-scanned Gaussian beam to be when the defocus, the simplest form of aberration, causes the spot size to expand by a factor of $\sqrt{2}$ (i.e., the Rayleigh range or confocal parameter). Regardless of the definition however, the axial FOV for Gaussian optics scales quadratically with the lateral resolution or output NA:

$$\mathit{FOV}_{z}\propto\frac{\delta_{\mathit{x}}^{2}}{\lambda}\propto\frac{\lambda}{\mathit{NA}^{2}},$$

(19)

where $\lambda$ is the wavelength. While axial FOV is restricted by defocus, the lateral FOVs for conic section mirrors are restricted by higher-order aberrations, notably astigmatism and coma from “misaligning” the mirror from its perfect, untilted focusing conditions. Here, as described in the preceding sections, we are essentially using misalignment to perform lateral scanning. Extending previous studies on characterizing the effects of such aberrations in misaligned parabolic mirrors on focus quality [34, 35, 36, 37, 38], here we provide a thorough investigation of the dependence of the lateral FOV, $\mathit{FOV}_{\mathit{x}}$ (or $\mathit{FOV}_{\mathit{y}}$, which is approximately the same), on a variety of relevant system design parameters: lateral resolution ($\delta_{\mathit{x}}$ or $\delta_{\mathit{y}}$) or $\mathit{NA}$, wavelength ($\lambda$), mirror size (via $f$, or $a$ and $b$), entry position or sample incidence angle, and telecentric vs. non-telecentric designs. Unless otherwise noted, our results for $\mathit{FOV}_{\mathit{x}}$ and $\delta_{\mathit{x}}$ also apply identically to $\mathit{FOV}_{\mathit{y}}$ and $\delta_{\mathit{y}}$, respectively. Note that “lateral” here is defined relatively along the plane orthogonal to the view direction (i.e., the central chief ray), which may be oblique relative to the mirror optic axis after reflecting off of the mirror surface.

Depending on the application and the way the sample is mounted, the angularly varying incident beam may encounter interfaces that can cause aberrations. For example, spherical aberrations can be serious if there is a flat refractive index (RI) discontinuity before the sample (or at the surface of the sample itself). In this situation, although near-normal incidence angles may be negligibly aberrated if the NA is sufficiently small, the more oblique the incidence angle and the higher the NA, the worse the aberrations (Figs. 8 and 9a). Another disadvantage is that the maximum sample-incident angle becomes more restricted due to Snell’s law (e.g., when switching from medium 1 to 2, the maximum angle in medium 2 is $\sin^{-1}(n_{1}/n_{2})$). This can be problematic for techniques that aim to reconstruct sub-surface information (e.g., OPT, ODT, OCPT, OCRT) if the sample itself exhibits a flat surface or if it is submerged in water with flat air-water or air-glass interface.

A workaround is to fill the entire mirror with water or another RI-matching medium to eliminate all RI boundaries, akin to water- and oil-immersion objectives. Such an approach may be practical if the mirror were shrunken, which does not affect the theoretical FOV for certain inclination angles (Sec. 5.6). Here, we propose two additional solutions that introduce additional optical elements to overcome these spherical aberrations at oblique illumination angles: 1) an optical dome (Fig. 9c-d), and 2) a cylindrical Petri dish or tube with a toroidal lens (Fig. 9e-f).

We have presented a general set of strategies employing conic section mirrors, particularly parabolic and ellipsoidal mirrors, for performing multi-angle imaging over very wide angular ranges over one or two axes without requiring sample rotation. Thus, high-speed 3D imaging is possible through fast optomechanical scanning or multiple cameras. Through derivations and comprehensive, but not exhaustive, simulations, we have established key scaling relationships between FOV/SBP and imaging system parameters, notably the resolution or NA, wavelength, mirror size, effective focal length, and telecentricity (Sec. 5; Eqs. 24, 27). Importantly, we have argued that for multi-angle 3D imaging applications, the well-known tilt aberrations of parabolic and ellipsoidal mirrors do not restrict FOV any more than depth of field limits axial FOV. In particular, the scaling is identical (i.e., quadratic) for most practical sample-incident inclination angles. Interestingly, the FOV dependencies are similar for both parabolic and ellipsoidal mirrors, and thus both are in theory viable options for multi-view imaging applications.

We have left out discussions on other conic section mirrors, such as rotationally symmetric hyperbolic (hyperboloidal), spherical, toroidal, and axicon mirrors, which could also be able to achieve single-axis 360^∘ imaging performance when combined with other elements. For example, an axicon mirror could be combined with a toroidal lens in a similar manner discussed in Sec. 6.2 to provide focusing power in the unfocused dimension (e.g., see Fig. 3 of [43]).

Our proposed approaches allow for very wide angular ranges, which would lead to spherical aberrations at obliquely illuminated flat RI discontinuities (e.g., glass coverslip) if uncorrected. We have proposed multiple strategies for compensating for these aberrations (Sec. 6). Note that this problem is not intrinsic to our approaches, but rather a general one for high-angle imaging applications – this is why some high-NA objectives have correction collars for glass coverslips.

As many imaging techniques benefit from imaging over wide angular ranges, the applicability and generality of our approach is broad. For example, a symmetric parabolic or ellipsoidal mirror can be used to collect 2D transmissive projection images over 180^∘ for OPT or OCPT. Similarly, these approaches can be applied to epi-mode multi-angle imaging techniques such as OCRT and photogrammetry.

We hope that our theoretical analyses of conic section mirrors will prove useful to researchers designing multi-angle imaging systems. To this end, we have provided Zemax files (Code 1 [42]) as generic templates for parabolic- and ellipsoidal-mirror-based system designs, including configurations for spherical aberration compensation, to facilitate adoption.