Originally in ArXiv
Now part of the TPAMI paper:
Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning
*denotes equal contribution
In this paper we are extracting surface reflectance and natural environmental illumination from a reflectance map, i.e. from a single 2D image of a sphere of one material under one illumination. This is a notoriously difficult problem, yet key to various re-rendering applications. With the recent advances in estimating reflectance maps from 2D images their further decomposition has become increasingly relevant.
To this end, we propose a Convolutional Neural Network (CNN) architecture to reconstruct both material parameters (i.e. Phong) as well as illumination (i.e. high-resolution spherical illumination maps), that is solely trained on synthetic data. We demonstrate that decomposition of synthetic as well as real photographs of reflectance maps, both in High Dynamic Range (HDR), and, for the first time, on Low Dynamic Range (LDR) as well. Results are compared to previous approaches quantitatively as well as qualitatively in terms of re-renderings where illumination, material, view or shape are changed.
This webpage contains the following material:
In the video below you can see the manipulation of illumination and material for a real object. Using  to derive the RM and the normals for the input image (for the car bodies) we use our approach to estimate illumination and material. Then, by clicking the corresponding sphere, we are able to perform illumination or material change.