This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Search for Publication

Year(s) from:  to 
Keywords (separated by spaces):

Covariance Pooling for Facial Expression Recognition

Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, Luc Van Gool
Computer Vision and Pattern Recognition Workshops (CVPR)
June 2018


Classifying facial expressions into different categories requires capturing regional distortions of facial landmarks. We believe that second-order statistics such as covariance is better able to capture such distortions in regional facial features. In this work, we explore the benefits of using a manifold network structure for covariance pooling to improve facial expression recognition. In particular, we first employ such kind of manifold networks in conjunction with traditional convolutional networks for spatial pooling within individual image feature maps in an end-to-end deep learning manner. By doing so, we are able to achieve a recognition accuracy of 58.14% on the validation set of Static Facial Expressions in the Wild (SFEW 2.0) and 87.0% on the validation set of Real-World Affective Faces (RAF) Database1. Both of these results are the best results we are aware of. Besides, we leverage covariance pooling to capture the temporal evolution of per-frame features for video-based facial expression recognition. Our reported results demonstrate the advantage of pooling image-set features temporally by stacking the designed manifold network of covariance pooling on top of convolutional network layers.

Download in pdf format
  author = {Dinesh Acharya and Zhiwu Huang and Danda Pani Paudel and Luc Van Gool},
  title = {Covariance Pooling for Facial Expression Recognition },
  booktitle = {Computer Vision and Pattern Recognition Workshops (CVPR) },
  year = {2018},
  month = {June},
  keywords = {}