Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time

Delio Vicini1     Sébastien Speierer1     Wenzel Jakob1    
1École Polytechnique Fédérale de Lausanne (EPFL),
ACM Transactions on Graphics (Proceedings of SIGGRAPH), 2021
Teaser

Inverse reconstruction of a scene with complex lighting and heterogeneous structure. Given the initialization (a), we seek to reconstruct the target (b) involving normal-mapped surface variation and roughness changes on the fish sculpture, and the addition of a plant based on triangular geometry. Using three rendered views of the target, we apply our proposed path replay backpropagation (PRB) (c) and a linear-time version of radiative backpropagation (RB) [Nimier-David et al. 2020] (d) to reconstruct the modified sculpture and a heterogeneous medium approximating the plant. Our method computes unbiased gradients and is able to converge to a higher-quality solution at equal time. The second and third rows show insets and PRB's convergence over time.

Abstract

Differentiable physically-based rendering has become an indispensable tool for solving inverse problems involving light. Most applications in this area jointly optimize a large set of scene parameters to minimize an objective function, in which case reverse-mode differentiation is the method of choice for obtaining parameter gradients. However, existing techniques that perform the necessary differentiation step suffer from either statistical bias or a prohibitive cost in terms of memory and computation time. For example, standard techniques for automatic differentiation based on program transformation or Wengert tapes lead to impracticably large memory usage when applied to physically-based rendering algorithms. A recently proposed adjoint method by Nimier-David et al. 2020 reduces this to a constant memory footprint, but the computation time for unbiased gradient estimates then becomes quadratic in the number of scattering events along a light path. This is problematic when the scene contains highly scattering materials like participating media. In this paper, we propose a new unbiased backpropagation algorithm for rendering that only requires constant memory, and whose computation time is linear in the number of scattering events (i.e., just like path tracing). Our approach builds on the invertibility of the local Jacobian at scattering interactions to recover the various quantities needed for reverse-mode differentiation. Our method also extends to specular materials such as smooth dielectrics and conductors that cannot be handled by prior work.

Video

Downloads

Text Reference

Delio Vicini, Sébastien Speierer, Wenzel Jakob. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. ACM Transactions on Graphics (Proceedings of SIGGRAPH), 40(4), August 2021.

BibTex Reference

@article{Vicini2021PathReplayBackprop,
  author  = {Vicini, Delio and Speierer, Sébastien and Jakob, Wenzel},
  title   = {Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time},
  journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH)},
  volume  = {40},
  number  = {4},
  year    = {2021},
  month   = aug,
  pages   = {108:1--108:14},
  doi     = {10.1145/3450626.3459804},
}