Alvin Pyngottu

Semester Work
Supervisors: Dr. Andres Romero Vergara, Dr. Danda Pani Paudel, and Prof. Dr. Luc van Gool

Multi-class 2D to 3D translation with disentangled shape, appearance, and attributes

The generation and manipulation of realistic facial expression in the 2D and 3D domain is a challenging task. Methods have been proposed tackling variety-rich 3D face model generation or 2D facial expression representation and transfers. However, often they lack in the ability to combine both domain and have realistic representations in each of them. In this project, we propose a framework learning combined 2D and 3D facial expressions. The system is able to translate between both domains, and is able to learn disentangled features of the models. We rely on a framework composed of two parts: In the first part, we make use of adversarial training to learn to generate realistic looking images of faces given the 3D models of them. We then learn disentangled representations for shape and appearance of the data. In the second part, we learn the reverse mapping, generating high-resolution shape given input RGB images by making use of implicit field models. To train the models, we use the BP4D-Spontaneous facial expression dataset. The experimental results show that our model is able to capture different facial expressions in 2D and 3D domain and generate appropriate shapes and images.