EndoPBR: Material and Lighting Estimation for Photorealistic Surgical Simulations via Physically-based Rendering

Estimating geometry from RGB images, or vision-based 3D reconstruction, is a long-standing research problem and holds many applications in minimally invasive surgery (MIS). For instance, a Simultaneous Localization and Mapping (SLAM) system can lead to real-time surgical navigation guidance for endoscopy [13], enable robots to perceive the surgical scene [12], and actualize digital twins for various healthcare applications [6]. However, surgical scenes are challenging to reconstruct with conventional algorithms due to factors like non-stationary lighting, textureless surfaces, and non-Lambertian surfaces. Most importantly, the lack of large labeled datasets for medical data limits 1) the utility of supervised learning of neural networks, and 2) a reliable platform to evaluate 3D reconstruction algorithms. Thus, existing research for vision-based 3D reconstruction in medical images falls into two main categories: 1) synthesizing a labeled dataset with simulated images or phantoms [15, 1, 22], or 2) training a neural network with self-supervision, e.g. joint ego-motion and depth estimation via photometric warping [14, 26, 18]. Despite their impressive results, these methods have limited generalizability to patients and anatomies outside of the training set due to the inherent domain gap; synthetic data has limited photorealism and self-supervised models overfit to the training data. Recently, foundation models have emerged as potential solutions for medical vision tasks [21], but their zero-shot performance in surgical scenes has room for improvement [7].

One promising avenue toward robust solutions for surgical 3D vision is synthetic labeled datasets of photorealistic images. If such a dataset could be generated at large-scale, then researchers could train models in a supervised manner or fine-tune foundation models for their specific anatomical domain. For example, synthetic data has been used for various medical computer vision tasks such as segmentation [24, 25, 27], depth estimation [15, 9], and camera pose estimation [1]. However, approaches that generate image frames independently are not suitable for systems like SLAM due to the lack of temporal and 3D consistency. Although some methods enforce such consistency with photometric warping [23] or learning mesh textures [8], these methods cannot generalize to novel views or lack photorealism. Phantom-based datasets like EndoSLAM [19] and C3VD [3] have been useful for the community, but they are not large enough to reliably train neural networks and limited to a specific anatomical domain. Recent works to generate realistic data use state-of-the-art neural reconstruction methods like Neural Radiance Fields (NeRFs) [16] and 3D Gaussian Splatting [11] to perform novel view synthesis. However, these algorithms typically require comprehensive coverage of the scene at many camera vantage points, which is not typical in endoscopy [20, 4]. Furthermore, vanilla versions of these algorithms assume stationary lighting and can be unstable with sparse views. To address these issues, we propose EndoPBR, a photorealistic simulation platform for endoscopy that robustly generalizes to novel views.

EndoPBR generates photorealistic images by optimizing material properties and lighting conditions from images of known camera poses and geometry. To disambiguate the training process, we formulate domain-specific constraints of the scene. For material properties, we use the simplified Disney bidirectional reflective distribution function (BRDF) model [5], consisting of base albedo, metallic, and roughness for a given surface point. We parameterize the BRDF with a Multi-Layer Perceptron (MLP) neural network, which takes as input a query 3D point and predicts a 5D vector of material properties. For light conditions, previous works use a neural network to learn the surface light field with the given viewing direction and 3D point [29]. However, this formulation assumes stationary lighting and cannot account for varying illumination present in endoscopic scenes. To address this issue, EndoPBR uses a moving spotlight model with learnable parameters to calculate incident light intensity at the surface. This formulation is simple enough for stable training but also allows for flexibility with learnable parameters to model the complexities of surgical scenes. We generate photorealistic outputs with physically-based rendering techniques and leverage differentiable rendering to minimize the difference between predicted and ground truth images.

We evaluate our algorithm on the Colonoscopy 3D Video Dataset (C3VD) [3] on novel view synthesis by comparing generated renders with ground truth images. We show that by estimating material and incident light, we synthesize novel views with comparable performance with previous methods that use NeRF and 3DGS. Furthermore, we demonstrate the utility of photorealistic synthetic data by fine-tuning a depth estimation model with EndoPBR’s generated images. We show that realistic synthetic data can be used to fine-tune a foundation depth estimation model, and that it produces similar results when fine-tuning with real images.

Our aim is to learn the material properties and lighting conditions of the scene given posed images and surface geometry, i.e. depth maps. An overview of our pipeline is shown in Fig. 2. EndoPBR determines color pixel values with the physics of light transport via physically-based rendering. We demonstrate that our method generalizes to camera views outside the training set.

Preliminaries. The Rendering Equation [10] computes the outgoing radiance from a surface point $\bm{x}$ along a camera viewing direction $\bm{\omega_{o}}$:

where $\bm{n}$ is the surface normal calculated from depth, $L_{i}$ is the incident light intensity coming from direction $\bm{\omega_{i}}$, and $f$ is the BRDF. The integration is performed in all incident directions $\bm{\omega_{i}}$ in the upper hemisphere $\Omega$ where $\bm{\omega_{i}}\cdot\bm{n}>0$.

Given a pixel, we calculate its corresponding 3D point by unprojecting its depth value. We query its incident light intensity (Sec. 2.1), BRDF parameters (Sec. 2.2), and evaluate Eqn. 1 to estimate the pixel value (Sec. 2.3). We use differentiable rendering to learn the appropriate BRDF and scene lighting $L_{i}$ from endoscopic images.

We develop and test our system on C3VD, a colonoscopy dataset with ground truth poses and depth maps. We use their provided camera intrinsics to undistort the images and also resize them to $(480,640)$. To ensure valid multiresolution hash encodings, we normalize all 3D points to $[0,1]$.

Implementation Details. We sample $30$k pixels with a batch size of $5$ in each iteration. We train our model for $1500$ epochs using PyTorch on an NVIDIA RTX4090 GPU. The BRDF network’s hidden dimensions are 64 with 2 layers following the multiresolution hash encoding layer [17]. We use the Adam optimizer with learning rate $10^{-4}$ and $\beta=(0.9,0.999)$. We set $\lambda_{m}=10^{-4}$ and $\lambda_{b}=10^{-3}$ in our experiments. Finally, we apply a $8:1$ train-test split to evaluate our method, following previous work [4]. Qualitative results of our method can be seen in Fig. 1.

We design two core experiments to demonstrate the photorealism and robustness of our model. First, we evaluate EndoPBR’s ability to generalize to novel views with image quality metrics like Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS), and Structural Similarity (SSIM). We compare our results with vanilla NeRF [16] as well as two recent works, namely REIM-NeRF [20] and Gaussian Pancakes [4] on the cecum_t4_b, desc_t4_a, trans_t1_a sequences from C3VD. These works leverage state of the art neural reconstruction algorithms to synthesize novel views.

Second, we analyze how realistic synthetic data can be useful for developing 3D vision solutions in minimally invasive surgeries with a simple experiment: fine-tuning a foundation depth estimation model with rendered images. We first train EndoPBR on the cecum_t1_b sequence and apply specific transformations like camera translation and rotation, varied material properties, and adjusted light conditions. Because these components are explicitly disentangled in our model, we are able to adjust them separately for meaningful data augmentation compared with NeRF or 3DGS-based approaches. With the aforementioned transformations, we acquire a synthetic dataset of around $18$k image-depth-pose samples (from $765$ images), which we visualize in Fig. 3. We do not include any training camera views to ensure valid testing.

We fine-tune a depth foundation model, namely ViT-S model of Depth Anything V2 (DAv2) [28], with the generated synthetic dataset for 100 epochs and the Adam optimizer (lr=5e-4). The training set is completely comprised of rendered images, but we use original C3VD images to evaluate the depth predictions. We use conventional metrics like Absolute Relative Error, Root Mean Squared Error, and $\delta<1.25$, the percentage of pixels within $25\%$ of ground truth depth values. We compare our results with baseline (zero-shot DAv2), baseline-scaled (median-scaled baseline), and using the original C3VD images to fine-tune the model. We note that original is the upper bound of depth estimation results.

Experiment 1. Novel View Synthesis. Image quality metrics on the test set are shown in Table 1. The generated image quality results have PSNR and SSIM results on par with or better than REIM-NeRF and Gaussian Pancakes. However, our LPIPS score is notably worse than the other methods. We attribute this to some dark delineations present in rendered images, which can be seen in Fig. 1. These black lines reflect the difficulty of resolving complex interactions between material properties and lighting in color calculation. However, our method is stable to train and does not rely on convergence of a different method as the input to our pipeline, as [4] does.

Experiment 2. Depth Estimation with Synthetic Data. The values for depth estimation results are reported in Table 2. First, we observe that the zero-shot depth estimation capabilities of DAv2 are imperfect even when median scaled to ground truth, which is supported by previous work [7]. Fine-tuning to a specific domain will be paramount to incorporate these foundation models to surgical 3D vision tasks. Furthermore, fine-tuning the model on synthetic data leads to similar results to fine-tuning with real images. This is demonstrated by the fact that all metrics are relatively similar between Original and our method. Because our synthetic data did not include any of the original training views or images, we claim leveraging simulated images is a promising venue to develop 3D vision tasks in endoscopy. To the best of our knowledge, this work is the first of its kind to evaluate the utility of synthetic data in this manner.

Conclusion. To summarize, we propose EndoPBR, a endoscopic simulation platform that leverages differentiable rendering to estimate material properties and lighting conditions from posed images and geometry. We demonstrate that our model’s novel view synthesis capabilities compete with existing methods that use NeRF and 3DGS while flexibly generalizing to novel views to synthesize a photorealistic dataset completely outside of its original training set. Furthermore, we show that realistic synthetic data generated from physics-informed learning is a promising avenue for robust 3D vision solutions in minimally invasive surgeries. We hope that this work inspires further exploration of generating realistic simulations.