This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Search for Publication

Year(s) from:  to 
Keywords (separated by spaces):

Modeling and Synthesis of Realistic Visual Speech in 3D

G. A. Kalberer, P. Mueller and L. Van Gool
3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body
N. Sarris and M. G. Strintzis, Ed.
IDEA Group Inc., 2004


The problem of realistic face animation is a difficult one. This is hampering in the movies, the use of 3D face models in communications, the use of avatars and likenesses in virtual reality, and the production of games with more subtle scenarios. This work attempts to improve on the current stateof- the-art in face animation, especially for the creation of highly realistic lip and speech-related motions. To that end, 3D models of faces are used and - based on the latest technology - speech-related 3D face motion will be learned from examples. Thus, the chapter subscribes to the surging field of image-based modelling and widens its scope to include animation. The exploitation of detailed 3D motion sequences is quite unique, thereby narrowing the gap between modelling and animation. From measured 3D face deformations around the mouth area, typical motions are extracted for different visemes. Visemes are the basic motion patterns observed for speech and are comparable to the phonemes of auditory speech. The visemes are studied with sufficient detail to also cover natural variations and differences between individuals. Furthermore, the transition between visemes is analysed in terms of co-articulation effects, i.e., the visual blending of visemes as required for fluent, natural speech. The work presented in this chapter also encompasses the animation of faces for which no visemes have been observed and extracted. The transplantation of visemes to novel faces for which no viseme data have been recorded and for which only a static 3D model is available allows for the animation of faces without an extensive learning procedure for each individual.

Download in pdf format
  title = {Modeling and Synthesis of Realistic Visual Speech in 3D},
  booktitle = {3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body},
  pages = {266-294},
  year = {2004},
  publisher = {IDEA Group Inc.},
  keywords = {computer animation, speech, viseme space, ica, realism, physiognomy, face space}