This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Search for Publication

Year(s) from:  to 
Keywords (separated by spaces):

Lip animation based on observed 3D speech dynamics

G. A. Kalberer and L. Van Gool
Proceedings of SPIE
San Jose CA, USA, January 2001


We are all experts in the perception and interpretation of faces and their dynamics. This makes facial animation a particularly demanding area of graphics. Increasingly, computer vision is brought to bear and 3D models and their motions are learned from observations. The paper subscribes to this strand for the 3D modeling of human speech. The approach follows a kind of bootstrap procedure. First, 3D shape statistics are learned from faces with a few markers. A 3D reconstruction of a speaking face is produced for each video frame. A topological mask of the lower half of the face is fitted to the motion. The 3D shape statistics are extracted and pricipal components analysis (PCA) reduces the dimension of the maskspace. The final speech tracker can work without markers, as it is only allowed to roam this constrained space of masks. Upon the representation of the different visemes in this space, speech or text can be used as input for animation.

Download in postscript format
Download in pdf format
  author = {G. A. Kalberer and L. Van Gool},
  title = {Lip animation based on observed 3D speech dynamics},
  booktitle = {Proceedings of SPIE},
  year = {2001},
  month = {January},
  pages = {16-25},
  volume = {4309},
  editor = {S. El-Hakim and A. Gruen},
  keywords = {facial animation, speech, visemes, eigenspace}