Controlling Material Appearance by Examples

Control and editing of material appearance is a long standing challenge in computer graphics. Different solutions have been proposed, depending on the targeted material representation. Tabulated materials are represented in a very high dimension, unintuitive space [WAKB09], making their manipulation difficult. Lawrence et al. [LBAD^∗06] represent a spatially-varying (SV) measured materials as an inverse shade tree, decomposing it into spatial structure and basis BRDFs to facilitate editing through extracted "1D curves" representing physical directions. Ben-Artzi et al. \shortciteArtzi2006, Artzi2008 proposed a fast iteration editing framework of material "in situ", allowing to efficiently visualize global illumination effects. Lepage et al [LL11] proposed material matting to decompose measured SV-BRDF into layers for spatial editing. More recently, Serrano et al. \shortciteSerrano16 and Shi et al \shortciteshi2021 proposed designing perceptual spaces for more intuitive BRDF editing and Hu et al. \shortciteHu2020 proposed encoding the BRDF in a deep network, reducing its dimensionality and simplifying editing. These methods target measured BRDF editing and focus on extracting relevant dimensions (perceptual, spatial) along which globally uniform edits can be made. As opposed to these methods, our approach targets spatially varying analytical model editing, enabling complex spatial detail transfer.

Analytical BRDFs represent materials based on pre-determined models (e.g Cook-Torrance \shortciteCookTorrance, Phong \shortcitePhong, GGX \shortciteGGX, etc.) and their parameters. To explore this parameter space, Ngan et al. \shortciteNgan2006 propose an interface to navigate different BRDF properties with perceptually uniform steps and Talton et al. \shortciteTalton2009 leverage a collaborative space to define a good modeling space for users to explore. Image-space editing was also explored with gloss editing in lightfields [GMD^∗16] and material properties modification [ZFWW20]. Recently, different methods were proposed to optimize or create procedural materials \shortcitehu2019, Shi20, hu2022. While procedural materials are inherently editable, they are limited to the expressivity of their node graphs and are difficult to design. More closely related to our approach are image-guided material properties and style transfer. Fiser et al. \shortcitefiser16 used drawing on a known shape (sphere) to transfer the style and texture to a more complex drawing. Deschaintre et al. \shortciteDeschaintre20 proposed using a surface picture and material exemplars to create a material with the surface picture structure and exemplar properties. Recently, Rodriguez-Pardo et al. \shortciterodriguezpardo2021transfer proposed leveraging photometric inputs to transfer material properties annotated on a small region to larger samples. As opposed to these methods our approach transfers the appearance and texture micro and meso structures from photo(s) to a pre-existing analytical SVBRDF.

We formulate our by-example control on material maps as a material transfer problem. In recent years, neural style transfer [GEB16] and neural texture synthesis [GEB15] have been used in a variety of contexts (e.g. sketching [TTK^∗21, SJT^∗19], video [JvST^∗19, TFK^∗20], painting style [TFF^∗20]). These methods are based on the matching of the statistics extracted by a pre-trained neural network between output and target images. For example, Gatys et al. \shortcitegatys2015, gatys2016 leverage a pre-trained VGG neural network [SZ15] to guide style transfer, using the Gram Matrix of extracted deep features from the image as their statistical representation. Heitz et al.\shortciteHeitz2021 described an alternative sliced Wasserstein loss as a more complete statistical description of extracted features. Different approaches proposed to train a neural network to transfer style of images or synthesize textures in a single forward operation [JAFF16, ULVL16, HB17, ZZB^∗18]. These methods however focus on the transfer of the overall style of images and do not account for details. Domain specific (faces) and guided style transfer have been proposed [KSS19, HZL19] to better control the process and transfer style between semantically compatible parts of the images.

In the context of materials, Nguyen et al. \shortcitenguyen2012 proposed transferring the style or mood of an image to a 3D scene. They pose this problem as a combinatorial optimization of assigning discrete materials, extracted from the source image, to individual objects in the target 3D scene. Mazlov et al. [MMTK19] proposed directly applying neural style transfer on material maps in a cloth dataset using a variant of RGB neural style transfer [GEB16].

In contrast to this previous work, we focus on transferring micro- and meso-scale details from photos onto 2D material maps and enable transfer spatial control. In particular, we differ from Mazlov et al. \shortciteMazlov2019 in our inputs. Our style examples are simple photographs, easily captured or found online. Our approach allows us to deal with the ambiguity of the material properties in the target photo through material priors and differentiable rendering while preserving the input material map structure and eventual tileability.

As mentioned, procedural materials are inherently editable as they rely on parametrically controllable procedures. Recent work on inverse procedural material modeling [HDR19, GHYZ20, SLH^∗20, HHD^∗22] aim at reproducing material photographs, but are still limited to simple or existing procedural materials. While editable, these models remain challenging to create and their optimization relies on the expressiveness of the available procedural materials, with no guarantee that a realistic model exists [HDR19, SLH^∗20]. Often procedural materials can look synthetic because they are not complex enough. In such cases, our method can augment the material appearance by transferring detailed real materials onto the synthetically generated material maps, while preserving their tileability. In this context, Hu et al. \shortcitehu2019 proposed to improve the quality of fitted materials through a style augmentation step, but style was only transferred to the final rendered images and did not impact the material maps themselves.

As an alternative to material transfer, one could directly capture the target material. Classical material acquisition methods require dozens to thousands of material samples to be captured under controlled illuminations [GGG^∗16]. Aittala et al. \shortciteaittala2016 proposed using a single flash image of a stationary material to reconstruct a patch of it through neural guided optimization. Recently, deep learning was used to improve single [LDPT17, DAD^∗18, HDMR21, GLT^∗21, ZK21] and few-images [DAD^∗19, GSH^∗20, GLD^∗19, YDPG21] material acquisition. These methods recover 2D material maps based on an analytical BRDF model e.g. GGX [WMLT07b]. As they mainly focus on acquisition, they do not enable control. As opposed to these methods, our approach provides a convenient way to control the appearance of material maps, leveraging the input material to retain high quality, allowing for multi-target control and preserving important properties such as tileability. Additionally, our method does not require a flash photograph as target, allowing the use of internet-searched references.

MaterialGAN [GSH^∗20] has been shown to be a good prior for lightweight material acquisition. However, the original MaterialGAN, based on StyleGAN2 [KLA^∗20], is not designed to produce tileable outputs. It has slight artifacts on the borders, even if the training data is perfectly tileable. Moreover, material maps in most of publicly available material datasets [DAD^∗18, DDB20] are not tileable. To address this limitation, we modify StyleGAN’s architecture following recent insights in GAN designs [KAL^∗21, ZHD^∗22] to ensure tileability of the synthesized material maps, even if the training data is not tileable. Specifically, we prevent the network from processing the image borders differently from the rest of the image, by modifying all convolutional and upsampling operations with circular padding.

Once we enforce the generator network to always produce tileable outputs, we cannot show tileable synthesized and non-tileable real data to the discriminator, as it would have a clear signal to differentiate them. Instead, as suggested by [ZHD^∗22], we randomly crop both real and synthesized material maps. The discriminator cannot identify whether the crop comes from a tileable source or not, and instead has to identify whether the crop content looks like a real or fake material.

We train our model with the same loss functions as StyleGAN2 [KLA^∗20] including cross-entropy loss with R1 regularization and path regularization loss for generators. See Sec. 4 for training details. With the trained tileable material prior, our material transfer method overcomes the local minimum problems caused by simple per-pixel optimization (Fig. 1) and reconstructs high-quality material maps compared to an adaption of Deep Image Prior (Fig. 11). In Fig. 2, we show our modified network architecture can successfully preserve tileability after transfer compared to the original MaterialGAN. Importantly, the preserved tileability allows us to directly apply our transfer materials on different objects seamlessly in a large-scale scene as shown in Fig. Controlling Material Appearance by Examples.

In previous work, Hu et al. \shortcitehu2019 proposed an extra style augmentation step to add realistic details on their fitted procedural materials given a photo as target. However, their transfer is only applied to the rendered images and does not modify the material maps, limiting the impact of their transfer step (e.g. the material can not be relit). Different from their method, we directly transfer the target photo(s) material appearance onto material maps, allowing the use of the new material in a traditional rendering pipeline.

Given a set of input material maps $M_{0}$ and a user-provided target image $I$, we compute the transferred material maps $M$ as follows:

where $R$ is a differentiable rendering operator, rendering material maps $M$ into an image. $d_{0}(R(M),I)$ measures the statistical similarity between the synthetic image $R(M)$ and target image $I$. $d_{1}(M,M_{0})$ is a regularization term that penalizes the structure difference between transferred material maps $M$ and the original input $M_{0}$. The lighting used in $R$ can easily be adapted if needed to roughly match the lighting conditions of target images. However, we found that a simple co-located point light works well in the examples we tested. Similar to neural style transfer and neural texture synthesis, we apply a statistics-based method to measure the similarity for $d_{0}$ and $d_{1}$. Common choices for $d_{0}$ and $d_{1}$ are style loss and feature loss [GEB15]. However, simply performing per-pixel optimization on material maps $M$ (i) fails to reach the appearance of the target photograph. This is caused by challenging local minima in the optimization and a high sensitivity to the learning rate, requiring careful tuning (see Figure 1). And (ii) the optimization may result in departure from the manifold of realistic materials. Instead, we take advantage of the learned latent space of our pre-trained tileable MaterialGAN to regularize our material transfer, effectively solving these problems.

We tackle the material transfer challenges in two steps. First, we project the input material maps $M_{0}$ into the latent space of the pre-trained MaterialGAN model $f$ by optimizing the latent code $\theta$. The optimization is guided by $L_{1}$ loss and feature loss:

where $F$ is a feature extractor using a pre-trained VGG network [SZ15]. With the projected latent vector, we perform material transfer by optimizing $\theta$ to minimize the statistical difference between rendered material $R(f(\theta))$ and the material target $I$

The statistical descriptor $S$, the style loss, can be implemented in different ways. The Gram Matrix [GEB15] is a commonly used statistical loss. Recently, Heitz et al.  \shortciteHeitz2021 proposed a sliced Wasserstein loss to measure a more complete statistical difference, taking additional statistical moments into consideration. Their implementation however relies on comparing signals (here images) with the same number of samples (here pixels). To remove this limitation, we propose using uniform resampling to balance the number of samples in both distributions to facilitate spatial control and multi-target transfer (Sec. 3.3). We derive in supplemental material a slightly slower but better grounded sliced Cramér loss, comparing the distribution CDFs rather than PDFs.

The sliced Wasserstein loss has been shown to be a good statistical metric [HVCB21] to compare deep feature maps, but does not allow comparing feature maps with different numbers of samples (pixels) trivially. The "tag" trick introduced in the original paper cannot be applied to our goal for multi-target transfer. The sliced Wasserstein loss compares two images by projecting per-pixel VGG feature vectors onto randomly sampled directions in feature space, giving two sets of 1D sample points $u$ and $v$, one for each image. These are compared by taking the difference between the sorted sample points. To allow for different sample counts $|u|<|v|$, we introduce the resampled sliced Wasserstein loss as:

where $U(v)$ is an operator that uniformly random subsamples a vector to obtain $|u|$ samples from $v$. Note here we compute $L_{1}$ error as opposed to squared error suggested in the original paper as we found it to perform better.

Using this resampling approach, we can compute statistical differences between labeled regions of different size. Assume we have label maps associated with material maps and each target photo, we define our transfer rule as Label X: Target Y, Z which means to transfer material appearance from regions labeled by Z in target photo Y onto regions labeled by X in the input material maps. We show examples of how to define the label in Fig. 3. The sliced Wasserstein loss will be computed between deep features on each labeled regions. Without loss of generality, we extract a deep feature $p^{l}$ and $\hat{p}^{l}$ from layer $l$ on the rendered image $R(\theta)$ and one of the material target $\hat{I}$. Similar to [HVCB21], we randomly sample $N$ directions $V\in S^{N_{l}}$ and project features $p^{l}$ and $\hat{p}^{l}$ onto the sampled directions to get projected 1D features $p^{l}_{V}$ and $\hat{p}^{l}_{V}$.

Now suppose we have a transfer rule Label i: Target $\hat{I}$, j, instructing us to transfer materials from regions labeled by $j$ in $\hat{I}$ to regions labeled by $i$ in the rendered image $R(\theta)$. We take samples labeled with $i$ from $p^{l}_{V}$ and samples labeled with $j$ from $\hat{p}^{l}_{V}$ as $p^{l}_{V}\{i\}$ and $\hat{p}^{l}_{V}\{j\}$. Note here $p^{l}_{V}\{i\}$ and $\hat{p}^{l}_{V}\{j\}$ usually contain different number of samples, therefore we compute the sliced Wasserstein loss using our proposed resampling technique (Eq. 4). We compute this loss for each transfer rule separately and take their average as our final loss. For completeness, we also evaluate the Gram Matrix [GEB^∗17] for partial transfer and show that it can lead to artifacts as shown in Fig. 5. We therefore adopt sliced Wasserstein loss with resampling in all our experiments.

A particular case is made of the boundary features, as neurons on the labeled boundary will have a receptive field which crosses the boundary due to the footprint of the deep convolutions, forcing them to consider statistics from irrelevant nearby regions. To prevent transferring unrelated material statistics, similar to Gatys et al \shortcitegatys2017, we perform an erosion operation on the labeled regions, and only evaluate the sliced Wasserstein loss on the eroded regions. Fig. 4 shows an example with and without this erosion step. We note that while an erosion step reduces irrelevant texture transfer, too large an erosion may remove all samples from the distribution at deeper layers. In such case, we do not compute the loss for deeper layers with no valid pixels.

Using our material transfer method, we demonstrate different applications:

Material Augmentation. Our method can augment existing material maps based on input photos. This is particularly useful for augmenting procedural materials which are difficult to design realistically. As shown in Fig. 9, our method can be applied to minimize the gap between an unrealistic procedural material and a realistic photo, which can be used as an complementary method for existing inverse procedural material modeling systems  [HDR19, SLH^∗20].

By-example Scenes Design. As our approach ensures tileability after optimization, the transferred material maps can be directly applied to texture virtual scenes smoothly. Fig. Controlling Material Appearance by Examples shows such an application scenario. Given photographs as exemplars, our method transfers realistic details onto simple-looking materials. With our transferred material maps, we can texture and render the entire scene seamlessly.

As we cannot compare to traditional style transfer methods since they operate on the image to image domain, we evaluate the benefit of the material prior used in our optimization and compare to two alternative optimization approaches, evaluating a simple per-pixel image style transfer combined with differentiable rendering and using a deep image prior [UVL18]. We also evaluate a direct single image material acquisition using MaterialGAN on target images and show that the input material regularizes the result.