Wavelet Transparency

The ultimate solution for transparency is path tracing, as has been demonstrated by the quality of offline rendering and relative elegance of its implementation [PJH16]. We expect that algorithmic and hardware improvements some day will enable real-time path tracing for complicated transparent objects. Today’s fastest GPUs can trace and shade 1-10 million rays per millisecond, which enables ray-traced reflections and shadows in currently available video games. However, path tracing and even Whitted ray tracing currently are too expensive for complex transparent materials, which require many more rays per pixel than opaque materials.

For example, consider a simple bottle of wine in an otherwise empty room. At least five ray intersections per pixel are required to render this scene: air to glass, glass to wine, wine to glass, glass to air, and then air to the wall behind the bottle. If each of those intersections spawns recursive shadow and reflection rays, then at least 15 rays per pixel are required and the ray cast and shade cost quickly consumes the entire frame budget at 1080p 60 Hz. Furthermore, this is the simplest case of path tracing for perfect specular reflection and refraction. For the more difficult case of rough, diffusing surfaces such as ground glass and participating media such as fog, path tracing requires many stochastic samples to converge. For opaque surfaces, it is common to take fewer samples, which leave noise, and then apply spatio-temporal denoising methods (e.g., [MMBJ17, SKW^∗17]). However, those denoisers do not work well for transparent materials because the noise is at varying depths and temporal reprojection is impractical.

The transparency visibility function maps distance along a view ray to net transmittance from that distance back to the ray origin. For example, in a scene with a single, 25% opaque plane 1m from the camera, the visibility function is $v(x)=1$ on $x=[0\mathrm{m},1\mathrm{m})$ and then $v(x)=0.75$ for $x>1\mathrm{m}$ (Figure 2b). Opaque rendering is the subcase in which the visibility function is binary. This compositing notion of transparency of grew out of Deep Shadow Maps [LV00] in which visibility is relative to a light source, but is now applied to all visibility problems. The visibility function also can be computed separately for discrete light frequencies to model colored transmission and absorption, and the “ray” can be bent into a path or extended into a cone to model specular and diffuse refraction.

In contrast to earlier approaches (see the survey in [AMHH^∗18]), the order-independent transparency (OIT) methods preferred for real-time transparency today do not require storing or sorting all surfaces at a pixel. This enables constant space per pixel and linear time in the number of pixels covered collectively by the primitives. Grouped roughly by their representation of the visibility function, the main OIT methods include:

Depth peeling

Exact, by exhaustive evaluation [Eve01]

Stochastic

Binary visibility, with depth chosen by importance sampling [ESSL10a, ESSL10b, ME11]

Occupancy map

Samples at uniform depths [KN01, SA09]

$k$-buffer

Samples at $k$ arbitrary depths, with varying strategies for choosing and merging samples [JC99, BCL^∗07, VF14, MCTB13, SML11, SV14, CMFL15, CWML18]

Fourier opacity map

Low-order Fourier basis [JB10]

Weighted, blended OIT (WBOIT)

Constant-absorption exponential or polynomial [Mes07, BM08, MB13]

Moment

Statistical moments [DL06, PK15, PMWK17, Sha18]

These all can be extended to perform not just partial coverage but also correct modulation of color during transmission by processing color channels coverage independently; efficient solutions have been demonstrated for stochastic [ME11] and WBOIT [MB13].

Although developed for rasterization systems, these OIT methods even allow limited ray tracing of partial coverage transparency along a straight line. For example, Battlefield V uses WBOIT to accumulate shading along particles using an “any hit” ray cast, rather than performing a true “closest hit” cast or stochastic path tracing which, as discussed in the introduction, would be prohibitively expensive today [SUD19].

There are several methods for approximating refraction in real-time rendering, which the OIT methods from the previous section do not address [Wym05, Sou05, dRBS^∗12, GD15].

Chromatic aberration occurs due to the index of refraction varying with the frequency of light. This produces the phenomena of colored fringes at edges in strong refractions (photographed in Figure 1), rainbows, and the spectral spread observed with a prism. It is trivial to implement in a ray tracer that samples light frequencies separately, and very approximate chromatic aberration due to the camera lens is a common post-processed effect in video games [GKK15].

There are many targeted solutions for transparency in specific contexts, for example, fog [NSTN93, Nis98, NN03, NRN04, PAT^∗04, Hil15], caustics [WD06, SKP07, Wym08, WD08, YK09], skin [JWSG10], and hair [SA09, YT10]. These outperform general transparency methods for their specific case.

Phenomenological Transparency [MM17] attempts to present a unified solution for real-time transparency by combining weighted, blended OIT and colored stochastic shadow maps [ME11] with new order-independent approximations for refraction, diffusion, ambient occlusion, shadowing, and caustics. It gains efficiency from approximating specific phenomena important to perception of transparency rather than performing physically-accurate simulation of all light rays.

We extend Phenomenological Transparency with (1) an improved visibility representation and (2) a minor phenomenological approximation of chromatic aberration at transparent surfaces.

The new visibility representation uses wavelets [Mal08], a hierarchical basis transformation combining the compressibility of frequency methods with the locality of spatial methods. This suits the visibility function, which exhibits sudden local changes for surfaces such as glass as well as continuous variation for materials such as fog. As with Fourier Opacity Maps, Deep Shadow Maps, and Moment Transparency and unlike other OIT methods, our representation requires separate passes over geometry to build the visibility approximation and then perform shading using it. We suggest some ways to eliminate this limitation as future work.

The new chromatic aberration term adjusts the gather samples for each color channel based on the strength of refraction and heuristics for identifying the “front” of objects. This is similar to previous ad hoc methods observed in video games for specific objects. It is the first addition of chromatic aberration into a general real-time OIT method that we are aware of in the literature.

Our results section shows that these new ideas combine with the previous strengths of Phenomenological and Moment Transparency to render more robustly than alternative OIT algorithms while also creating improved perception of transparency phenomena in many cases. The performance of this new Wavelet OIT depends on a design that caters to human perception rather than radiometric error, so it is not appropriate for predictive rendering and we make no claim of a direct relationship or objective comparison to physically-based simulation. Instead, we evaluate the method based on objective space and time efficiency and present visual comparison to real-time alternatives for the reader to consider for their application.

The wavelet model of compositing has three steps, similar to MLAB and Moment Transparency. The first step is two unordered passes over all transparent geometry or their bounding boxes to build the visibility approximation, which is stored as wavelet coefficients. The second step is another unordered pass over all transparent geometry to shade it and weigh it by the visibility function. The third step composites the accumulated transparent results over the opaque parts of the scene, which is where the other effects from Phenomenological Transparency are applied.

The algorithm is:

1. 1.

   Compute tight per-pixel depth bounds by drawing transparents (or bounds) into a depth buffer.

2. 2.

   Construct the wavelet coefficient buffer for transparents.

3. 3.

   Accumulate transparents by drawing modulated by the visibilty function, as with previous methods.

4. 4.

   Composite over opaques and apply refraction, diffusion, etc.

For wavelets of rank $N$ we store $2^{N+1}$ coefficients per channel, packed into 32 bits with a shared exponent in the form E5B9G9R9. See our code supplement for the bitpacking implementation. For example, rank $N=3$ wavelets encode 48 coefficients in 64 bytes per pixel total.

The radiance compositing equation is

where $L_{i}$ is the incoming radiance at $i$th interface, $r_{i}$ is the reflectance, and $v\left(x_{i}\right)$ is the net transmittance visibility function at depth $x_{i}$. Because transmittance decays exponentially with successive surfaces, for numerical stability convert to the net absorbance function:

For discrete interfaces such as glass, the absorbance function is:

where $\theta\left(x\right)$ is Heaviside theta function, and it is smooth for continuous media such as fog. We observe that Haar wavelets are a good representation of $\theta$.

We now derive a wavelet transformation for the visibility function, and then provide an implementation and results. The code extends previous work from [MM17] and [MMM17].

The purpose of this section is to approximate $A$ with a linear interpolation of Haar wavelets. First we start with a direct approximation of $A$ with Haar wavelets.

The approach will be based on two algorithms. One for efficient evaluation of the coefficients, and another one for efficient evaluation of the approximation at a point.

To understand the nature of the approximations, we graph the visibility function along the central ray for some everyday scenes with transparency in Figure 2. The figures in this section display the scenes as cartoon schematics and the graphs as computed from the equivalent real 3D scene.

In the figure, ground truth visibility is a dashed black line, Moment Transparency is yellow, and Wavelet Transparency is a thick green line. We compare at parameters that yield approximately equal performance for them.

Figure 2a shows the view along a ray through the windows of a car that is surrounded by fog. The ground truth visibility function decreases in a shallow exponential within fog, is constant in air, and suddenly drops at the surface of the glass windows. Although both approximations slightly overestimate visibility within fog near the camera, Wavelet is a strictly better approximation than Moment across the entire scene. Figure 2b shows visibility in the bottle of winebottle scene described in Section 1.2, modeled without extinction within the wine. Here, both approximations oversmooth and Moment gives a tighter fit near the first interface. Figure 2c shows separate smoke and fire particle systems. Wavelet is a better fit overall and captures the sharp transitions slightly better.

The grid in Figure 3 compares three recent OIT strategies to the new Wavelet Transparency and to a non-realtime A-buffer as a reference. We implemented each of these methods as a replacement for the visibility strategy for Phenomenological Transparency but retained the full suite of other effects that algorithm produces. In the first two rows, Moment8 and Wave4 are used because Moment6 underflowed on this scene. All other rows use Moment6 and Wave3. We use single-precision Moment instead of quantized to maximize its image quality (for performance comparisons we used 16-bit quantization). MLAB uses four layers in all rows.

Row 1 shows an opaque sphere in a particle system against an orange background. Row 2 repeats this scene with a transparent sphere. In each case MLAB4 underperforms on coverage and WBOIT fails badly, revealing too much of the sphere. Moment and wavelet transparency each give similar results, which are close to the ideal A-buffer result.

Rows 3-5 are different views of the Bistro scene with many layers of drinking glasses. Where the number of layers is small, all algorithms are similar. When the number of layers becomes high it exceeds the MLAB’s buffer, as seen at the reflection on the first glass just left of center in row 5. In these cases WBOIT fails more gradually but presents an ambiguous result, where the ordering of glasses in depth becomes less obvious. Moment is much better, but oversmooths the visibility function and tends to excessively dim the highlights from glasses beyond the first layer. Wavelets are indistinguishable from the A-buffer, as the step functions due to glass are the ideal case for the Haar basis.

Rows 6-7 show alpha-masked leaves from two viewpoints. Examining the coverage transition at the edges of leaves, especially where those edges overlap in depth, we observe that MLAB underestimates coverage, WBOIT and moment have some noise and discontinuity, and Wavelet most closely (but not perfectly) matches the reference A-buffer.

We conclude that across all results, Wavelet is the most robust because it is consistently among the best approximations, even though in specific cases others may also perform well.

Figure 5 shows a simplified scene to highlight the new chromatic aberration approximation (all other figures in this paper used it with refraction as well). On the left is the scene without refraction simulation, in the center is the method of McGuire and Mara [MM17], and in the right is the new chromatic aberration, which is most noticable at high-contrast edges. Compare this to real chromatic aberration photographed in Figure 1. Despite a tenuous relationship to the physics of the real effect, we suggest that the simple simulation in (c) yields a more convincing image of the refraction phenomenon than without it in (b).

Figure 6 is inspired by a famous path traced figure by Walter et al. [WMLT07]. At low cost in a rasterization pixel shader pass, our method demonstrates the phenomena of refraction, diffusion, and chromatic aberration of a frosted glass globe. The refraction pattern differs because they modeled a solid ball and ours is a hollow sphere with thick walls.

Table 1 shows the time in milliseconds for steps 2 and 3: building the visibility approximation and then evaluating it and shading, for the $1920\times 1080$ rendering of the view of drinking glasses on the Bistro bar in row 5 of Figure 3 on GeForce RTX 3090. The composite time for step 4 is the same for all algorithms: 0.1ms. Performance of Moment Transparency is done here in 16-bit quantized precision mode.