The MultiModal - MultiMedia Seminar Series

Information theoretic feature extraction for multimodal signal/image processing

Jean-Philippe Thiran, EPFL, Signal Processing Institute (http://ltswww.epfl.ch/~thiran)

Exploiting the complementary and redundant information contained in multimodal signals is a real challenge. In a multimodal classification talks, the extraction of relevant features from the different channels requires the measure of their marginal information as well as of the joint information shared by the different channels. How to extract such information and to exploit it for optimizing the classification performances is the topic of our research. In this talk, we will summarize the information theoretical framework that we have developed for several years. We will then show how to use it in practice in applications such as multimodal medical image processing, multimodal audio-visual speaker detection or audio-visual speech recognition.

Building Browsers with JFerret

Mike Flynn, Senior Research Scientist, IDIAP

JFerret is a multi-media meta-browser. It takes an XML description of a browser, and creates the user interface dynamically. Browsers can play multiple video and audio tracks, synchronized, and a display large range of usual interaction components. I will demonstrate various browsers produced for the AMI and IM2 projects, for exploring recordings of meetings.

Mike Flynn graduated as a computer scientist in 1981, initially working on commercial communications and fault-tolerant systems, but then moved into research on automatic VLSI design, formal methods and early object-orientation. In 1991, he joined Xerox PARC, producing a prototype Video Diary, and the well-known Forget-Me-Not memory prosthesis. A spin-off from this work became a Xerox product, "MobileDoc". In 2001 he worked for British Telecom Research Labs on satellite provision of broadband to the public, followed by a move to Symbian, as an architect of the "next generation" of mobile phone software. In 2003, he joined IDIAP to work on new ideas for multi-modal Browsers. He holds nine patents and has publications in a wide range of computer science.

A Discriminative Approach for the Retrieval of Images from Text Queries

Samy Bengio, IDIAP

This work proposes a new approach to the retrieval of images from text queries. Contrasting with previous work based on generative models and likelihood maximization, this method relies on a discriminative approach: the parameters are selected in order to minimize a loss related to the ranking performance of the model, i.e. its ability to rank the relevant pictures above the non-relevant ones when given a text query. In order to minimize this loss, we present two models: first a simple linear based system (that can be kernelized), using the same features as found in previous work; second, a neural network trained by stochastic gradient descent, where the extraction of a global image represention from local block descriptors is learned jointly with the retrieval problem. Both discriminant models are compared to state-of-the-art approaches over the Corel dataset and yield statistically significantly better performance.