DeLight-Net: Decomposing Reflectance Maps into Specular Materials and Natural Illumination

Stamatios Georgoulis*  Konstantinos Rematas*  Tobias Ritschel  Mario Fritz  Luc Van Gool  Tinne Tuytelaars 

Originally in ArXiv

Now part of the TPAMI paper:
Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning

*denotes equal contribution

Abstract

In this paper we are extracting surface reflectance and natural environmental illumination from a reflectance map, i.e. from a single 2D image of a sphere of one material under one illumination. This is a notoriously difficult problem, yet key to various re-rendering applications. With the recent advances in estimating reflectance maps from 2D images their further decomposition has become increasingly relevant.

To this end, we propose a Convolutional Neural Network (CNN) architecture to reconstruct both material parameters (i.e. Phong) as well as illumination (i.e. high-resolution spherical illumination maps), that is solely trained on synthetic data. We demonstrate that decomposition of synthetic as well as real photographs of reflectance maps, both in High Dynamic Range (HDR), and, for the first time, on Low Dynamic Range (LDR) as well. Results are compared to previous approaches quantitatively as well as qualitatively in terms of re-renderings where illumination, material, view or shape are changed.

Welcome to our project

This webpage contains the following material:

  • Video of illumination and material manipulation in real photos
  • Examples of illumination/material decomposition in synthetic RMs
  • Examples of illumination/material decomposition in real RMs
  • Analysis of the metrics (MSE, DSSIM) used in the paper
  • The dataset and MatConvNet CNN models and loss functions

In the video below you can see the manipulation of illumination and material for a real object. Using [8] to derive the RM and the normals for the input image (for the car bodies) we use our approach to estimate illumination and material. Then, by clicking the corresponding sphere, we are able to perform illumination or material change.