This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Search for Publication

Year(s) from:  to 
Keywords (separated by spaces):

Combining RGB and ToF Cameras for Real-time 3D Hand Gesture Interaction

Michael Van den Bergh, Luc Van Gool
Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV 2011)
Kona, Hawaii, January 2011


Time-of-Flight (ToF) and other IR-based cameras that register depth are becoming more and more affordable in consumer electronics. This paper aims to improve a real- time hand gesture interaction system by augmenting it with a ToF camera. First, the ToF camera and the RGB camera are calibrated, and a mapping is made from the depth data to the RGB image. Then, a novel hand detection algorithm is introduced based on depth and color. This not only improves detection rates, but also allows for the hand to overlap with the face, or with hands from other persons in the background. The hand detection algorithm is evaluated in these settings, and compared to previous algorithms. Furthermore, the depth information allows us to track the position of the hand in 3D, allowing for more interesting modes of interaction. Finally, the hand gesture recognition algorithm is applied to the depth data as well, and compared to the recognition based on the RGB images. The result is a real-time hand gesture interaction system that allows for complex 3D gestures and is not disturbed by objects or persons in the background.

Download in pdf format
  author = {Michael Van den Bergh and Luc Van Gool},
  title = {Combining RGB and ToF Cameras for Real-time 3D Hand Gesture Interaction},
  booktitle = {Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV 2011)},
  year = {2011},
  month = {January},
  keywords = {}