Markerless Motion Capture of Interacting Characters Using Multi-view Image Segmentation

Yebin Liu, Carsten Stoll, Juergen Gall, Hans-Peter Seidel, and Christian Theobalt

Abstract

We present a markerless motion capture approach that reconstructs the skeletal motion and detailed time-varying surface geometry of two closely interacting people from multi-view video. Due to ambiguities in feature-to-person assignments and frequent occlusions, it is not feasible to directly apply single-person capture approaches to the multiperson case. We therefore propose a combined image segmentation and tracking approach to overcome these difficulties. A new probabilistic shape and appearance model is employed to segment the input images and to assign each pixel uniquely to one person. Thereafter, a single-person markerless motion and surface capture approach can be applied to each individual, either one-by-one or in parallel, even under strong occlusions. We demonstrate the performance of our approach on several challenging multi-person motions, including dance and martial arts, and also provide a reference dataset for multi-person motion capture with ground truth.

Images/Videos

Video ~30MB (AVI)

Data

The data used in the paper is available for research purposes. To obtain the data, please contact Carsten Stoll. When using the data, please acknowledge the effort that went into data collection by referencing the corresponding paper.

Publications

Liu Y., Stoll C., Gall J., Seidel H.-P., and Theobalt C., Markerless Motion Capture of Interacting Characters Using Multi-view Image Segmentation (PDF), IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11), 1249-1256, 2011. ©IEEE

Gall J., Stoll C., de Aguiar E., Theobalt C., Rosenhahn B., and Seidel H.-P., Motion Capture Using Joint Skeleton Tracking and Surface Estimation (PDF), IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09), 2009. © IEEE

Slides: Markerless Motion Capture of Interacting Characters Using Multi-view Image Segmentation (PPTX), IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11), Colorado Springs, CO, USA, 2011