Research, Former Projects

Projects finished in 2004

Visemes

Participants: Gregor Kalberer, Pascal Müller, Luc Van Gool

Partner: Eyetronics Inc.

Objective: Improvment on the current state-of-the-art in face animation, especially for the creation of highly realistic lip motions. To that end, 3D models of faces will be used and - using the latest technology - speech related 3D lip motions will be learned from examples.

The problem of realistic face animation is a difficult one. The serious restrictions in animators' capabilities to deal with human faces are hampering a further breakthrough of high-tech domains such as special effects in the movies, the use of 3D face models on communications, and the use of avatars and likenesses in virtual reality, the internet, games, and all kinds of interfaces.

This project wants to improve on the current state-of-the-art in face animation, especially for the creation of highly realistic lip motions. To that end, 3D models of faces will be used and - using the latest technology - speech related 3D lip motions will be learned from examples. Thus, the project subscribes to the surging field of image-based modeling and in fact widens its scope to include animation. Indeed, the capacity to extract detailed 3D motion sequences is quite unique and can be fully exploited for animation, which so far has been kept rather separated from modeling.From measured 3D face deformations around the mouth area, typical motions will be extracted for different 'visemes'. These are the basic lip motion patterns observed for speech and are comparable to the phonemes of auditory speech. The visemes will be studied with sufficient detail to also cover natural variations and differences between individuals.

The work also encompasses the animation of faces for which no visemes have been extracted. The 'transplan- tations' of visemes to novel faces for which no viseme data have been recorded and for which only a static 3D model is available will allow to animate faces without an extensive learning procedure for each individual. Last but not least the coarticulation effects will also be studied, i.e. the visual blending of visemes as is required for fluent, natural speech. The project will focus on spoken English and German.

3D MURALE - 3D Measurement and Virtual Reconstruction of Ancient Lost Worlds of Europe

Participants: Dominik Auf der Maur, Alexey Zalesny, Luc Van Gool

Partner:	Brunel University, London, UK
	EyeTronics Inc., Leuven, Belgium KULeuven , Katholieke Universiteit Leuven, Belgium Graz University of Technology , Austria Imagination Computer Services GesmbH Vienna University of Technology, Austria

Extended Information: Project Homepage

Objective: Development of 3D multimedia tools to measure, reconstruct and visualize the ancient city of Sagalassos in Turkey in virtual reality.

The archaeological site at Sagalassos is one of the largest archaeological projects in the Mediterranean dealing with a Greco-Roman site over a period of more than a thousand years (4th century BC-7th century AD).

This project aims at the development of 3D measurement, reconstruction and visualisation tools for use by archeological teams.The new multimedia technologies will produce rich new ways of recording, cataloguing, conserving, restoring and presenting archaeological artefacts, monuments and sites. These technologies will be used to model the Sagalassos site and show how they can be used for preserving and presenting the cultural heritage of Europe in two important ways:

By putting such new technologies in the hands of the archaeologists themselves rather than creating multimedia content after the excavations. As an important consequence, a more complete record of the finds can be created and presented to the public.
By presenting the site not as a static entity from a long-gone past, but as a vibrant place that underwent a lot of changes throughout its existence. This includes the visualisation of the situation in different eras and of the excavation as they proceeded through different time layers. Both these aspects of the project will help to produce records and visualisations that are more complete and scientifically precise.

ETH subgoal:
ETH aims at developing a texture synthesis procedures, which are able to produce the images of texture looking visually similar to the original textures, in particular, limestones, landscapes, and vegetation. The modeling process uses currently very simple pairwise pixel statistics gathered from the original image. To improve the modeling quality the pixel pair types will be properly and mutually dependent selected from the big class of the candidates. All the selected types form the neighborhood structure of the texture model. For enrichment of the class of textures that could be visually similar reproduced, further modeling improvement includes the texture presegmentation step. The complex texture or even the whole scene will be segmented onto the subtextures having simpler pixel interdependencies.
The so-called composite texture model includes then three types of submodels:

the segmentation map submodel
the submodel of every subtexture, and
the submodel of the interdependencies between the subtextures.

After the composite model design, the synthesis includes the creation of the synthetic map of segments, and the creation of every subtexture at the corresponding places on the map taking into account the interdependencies near the segments' boundary (see Figure with original and synthetic landscapes).

Projects finished in 2003

CogViSys, Semantic Interpretation of Geometry

Participants: Stephan Scholze, Luc Van Gool

Extended Information: Project Homepage

Objective: Employ high-level reasoning for geometric modelling.

This project aims at developing methods for geometric modelling of incomplete, imprecise and contaminated input data. Two main approaches are investigated which complement each other.

The first building block consists of a probabilistic interpretation of the input data. As application example one might think of grouping 3D line segments into polygons which is a task often encountered in 3D modelling of planar objects. A Bayesian formulation of the grouping problem which avoids overfitting the data but which is still able to capture relevant details is investigated.

The preliminary geometric models derived in the previous step may still be incomplete or topologically not correct, mainly due to the imprecise nature of the input data. To overcome the remaining deficiencies, a semantic interpretation of the polygon geometries is performed. A domain specific semantic labelling allows to infer missing parts of the model and to correct the overall model topology. This implies closing of gaps in the model. The semantic interpretation of the geometry renders this task selective and thus enables to preserve details of the models.

A main application area for the developed methods is the automated reconstruction of building roofs from high resolution multiview aerial imagery.

Visual Grouping

Participants: Andreas Turina, Luc Van Gool

Partner: KULeuven - Katholieke Universiteit Leuven

Objective: Detection and grouping of repeated patterns in images.
Grouping is a process in computer vision where the computer identifies meaningful entities in images.It is an important step between low-level feature extraction and scene interpretation.

This project aims at developing novel, robust grouping strategies. In particular, the project will consider grouping from a geometric perspective. The propounded approach is principled in that it presents a single mathematical framework in which most traditional grouping rules are encapsulated. It introduces a natural hierarchy of in creasingly specific geometric configurations. It is powerful as it takes perspective distortions fully into account, whereas previous grouping approaches have been restricted to the case of fronto-parallel viewing or cases where depth effects could be modelled as an affine rather than a perspective skew.

Finally, it is efficient as it eliminates much of the search by which grouping strategies are so often plagued. This is achieved through a combination of invariant-based indexing and the Hough transform.

Implant Migration Measurement using Standard Radiographs

Participants: Kathrin Burckhardt, Gabor Székely

Partner: University Hospital Balgrist

Extended Information: X-Ray Image Database

Objective: Design of a practicable and precise (0.1 mm) method for measuring the migration of artificial hip sockets.

About 400.000 artificial hip joints are implanted every year only in Europe. However, despite its success, hip replacement still involves complications. One major problem is the loosening of the cup, the acetabular part of the implant. Migration of the implant, which means the change of the cup's position in the bone, is interpreted as the only quantifiable sign for loosening. There already exist different 2D and one 3D method for measuring the migration over time. For the former methods, the standard x-ray images of the patients' follow-up studies after implantation are used, whereas for the latter markers are implanted in the pelvis and special stereo x-ray images are acquired. The problem is that the 2D methods are not precise enough and the 3D method is impractical for the clinical routine. Therefore, we work on a method for measuring the cup migration with a precision in the submillimeter range and useable under clinical conditions. Our approach is a 2D measurement using standard x-ray images, which is as insensitive as possible towards the variable orientation and position of the pelvis at the exposure and which uses state of the art image processing algorithms to locate the cup and bony landmarks.

New Metaphors for Interactive 3D Volume Segmentation

Participants: Matthias Harders, Gabor Székely

Partner:	Semmelweis University Budapest, University Hospital Zurich,
	University Regensburg, Varian Medical Systems

Objective : development of a new framework for three-dimensional interactive segmentation based on new man-machine interfacing paradigms as offered by virtual reality (VR).

In spite of considerable efforts during the past decades, image segmentation is still one of the major bottlenecks in medical image analysis. Neither purely manual nor fully automatic approaches are appropriate for the correct, efficient and reproducible identification of organs in 3D data volumes. Goal of this pilot project is the exploration of the power of new man-machine interfacing paradigms as offered by virtual reality (including graphics, audio and haptics), e.g., resulting in new closed-loop segmentation systems, allowing an optimal cooperation between computer-based image analysis algorithms and human operators.

LaSSo - LAparoscopic Surgery SimulatOr

Participants: Matthias Harders, Johannes Hug, Gabor Székely

Partner:	University Hospital Zürich
	IBT - Institute of Biomedical Engineering, IfE - Electronics Laboratory
	IfR - Institute of Robotics, Institute of Mechanics

Extended Information: Project Homepage

Objective: Development of a laparoscopic surgery simulator device using the techniques of virtual reality to provide nearly realistic training environment.

The basic idea of laparoscopic surgery is to minimize damage to healthy tissue while reaching the actual surgical location. This results in major gain in patient recovery after operation. The price for this advantage is paid by the surgeon, who loses direct contact with the operation site. The operations are usually performed under mono-scopic vision and highly restricted manipulative freedom, which requires very special skills from the surgeon. Up to now no appropriate training devices are available, which would allow to fully acquire these skills before actual intervention on patients. The goal of the project is the development of a laparoscopic surgery simulator device using the techniques of virtual reality which provides nearly realistic training environment.

Projects finished in 2002

AMOBE II - Automation of Digital Terrain Model Generation and Man-Made Object Extraction from Aerial Images

Participants: Stephan Scholze, Luc Van Gool

Partner: Institute of Geodesy and Photogrammetry, ETH

Extended Information: Project Homepage

Objective: The purpose of the project is the development of automatic methods for extracting quantitative 3D information on man-made objects from aerial images.

The project partners Institute of Geodesy and Photogrammetry and the Computer Vision Laboratory work in the automatic detection and reconstruction of man-made objects - especially houses - in high resolution, aerial color images. In a previous project (AMOBE I) it has been successfully shown, that an automatic reconstruction of isolated buildings in suburban scenes is possible, if the location of the building in the image is known.

In the AMOBE II project, the given task is extended to densely built--up urban areas. This causes qualitatively and quantitatively new difficulties stemming from the more complicated roof shapes and the typical situation of buildings located close or contiguous to each other.

For 3D building reconstruction straight line segments at roof edges need to be matched between corresponding views. Solving the correspondence problem is not straightforward. Due the weak geometric constraint ruling stereo vision geometry, corresponding pairs of line segments in different views can not be identified unequivocally. To overcome the geometric ambiguities at the stereo matching step, we take into account the color distribution in the regions flanking the line segments forming a putative pair. We have developed a line segment matching algorithm for 3D reconstruction of static scenes. This algorithm makes extensive use of color information. It also allows to exploit additional geometric and chromatic information from additional views of the scene.

The main part of the project concentrates on the further processing of the 3D line hypotheses. Since we achieve a high discriminative power using color information for line segment matching, the resultant 3D line hypotheses are very reliable and also small in number due to the lack of mismatches. This enables us to keep the combinatorics under control, and thus simplifies the ongoing development of an intelligent algorithm to generate reliable and stable hypotheses of roof parts and complete roofs.

CATS - Classification And Tracking in advanced Video Surveillance Systems

Participants: Esther Koller-Meier, Luc Van Gool

Partner: ASCOM Systec AG

Objective: Detection of ``unusual'' motion events for surveillance applications in video sequences.

In many video surveillance applications, a human operator is required to observe video sequences from a large number of sensors being displayed on monitors in a control room, in order to detect the occurrence of dangerous events. The support of automatic video processing systems should observable relieve the operator. Such a system should be able to detect the occurrence of ``events'' and perform a screening of the ``normal'' ones, just requiring the human evaluation for the ``most interesting'' or ``abnormal'' cases.

CATS is a 1-year KTI-project in collaboration with the industry to develop such a self-learning event detection system. This surveillance system is primarily intended to be used in public rooms. As human motions can be modeled as temporal trajectories, which give the spatio-temporal coordinates of a person, we try to learn characteristical behavior patterns. In the case that people act in accordance with the learned motion patterns, a ``normal'' event is detected, while in all other cases the operators' attention should be focused on it.

Projects finished in 2001

Computer-Assisted Radiographic Hip Joint Measurement

Participants: Christian St\366cklin, Gabor Székely

Partner: Orthopedic Devision of the University Hospital Balgrist, Z\374rich

Objective: Development of a computer assisted measurement tool for precise and fast analysis of digitized medical images.

The early identification and treatment of hip dysplasia has a long tradition in medical science and is of particular importance for the treating orthopedist. In practice, it can be shown that by early diagnose and special therapy in most cases a satisfactory healing can be realized. The decision of the diagnose if the studied object is displastic or normal is heavy influenced by the subjective sense of the operator. The fundamental problem is that the difference between ''normal'' and ''displastic'' is difficult to define. In order to find new statistical robust analysis criteria, two larger clinical studies are performed.

Lesion Evolution in Multiple Sclerosis

Participants: Daniel Welti, Gabor Székely

Partner: University Hospital Basel

Objective: The goal of the project is to characterize lesion evolution by quantifying MR-based spatio-temporal changes over time.

Traditionally, the characterization of MS lesion development is mostly based purely on the spatial pattern of lesions. Although lesion load measurements provide a more objective and sensitive measure of disease evolution than clinical measures, the poor correlation between changes of lesion load and changes of disability is of concern. Purely intensity based segmentation has strong limitations and does not provide satisfactory results in many cases. By examining temporal changes in consecutive MR scans, active MS lesions can be segmented and characterized straightly. But lesion development is a complex spatio-temporal process, consequently concentrating exclusively on the spatial or temporal aspects of it cannot be expected to provide optimal results. The goal of the project is to characterize lesion evolution by quantifying MR-based spatio-temporal changes over time. Spatio-temporal lesion models will be used to get a better understanding of MS pathogenesis and hopefully allow a classification of MS manifestation that can distinguish different lesion behavior and perhaps give a better explanation of clinical findings. These models will be used to provide a spatio-temporal segmentation method.

Projects finished in 2000

Modelling daily runoff from snow and glacier melt using remote sensing data

Participants: Jesko Schaper, Klaus Seidel, Jaroslav Martinec

Partner: RSL - Remote Sensing Laboratory, University of Zurich

Extended Information: Project Homepage

Objective: Using satellite images for simulating the effect of a possible climate warming on the areal extent of the seasonal snowcover, the glacier retreat and on the runoff regime in the Swiss alps.

The concept of the project requires, that GIS as well as remote sensing techniques are involved in combination with the runoff model SRM+G. The project uses high resolution optical satellite images for a detailled analysis of the regional distribution of snow cover as well as bare glacier ice of three Swiss runoff basins.

A main part in the project is the runoff simulation with the SRM+G model for different years. This model version was evolved to analyse quantitatively melt processes in highly glaciated basins. Another part of the project deals with the different contributions to the total runoff like glacier ice, rain, new snow and seasonal snow cover. Adapting various scenarios with changed climate conditions to the SRM+G model we evaluate the consequences to the areal extent of the seasonal snowcover, the glacier retreat and the snow- and icemelt for each basin. This topic will gain further significance since we can state a constant warming of the earths atmosphere during the 20th century.

CARTESIAN - Cost effective Application of Remote sensing to enviromenTal aspects of ski rEgions; a Ski region monItoring and mAnagement informatioN system

Participants: Dominik Brander, Marcel Zurflüh, Christian Huggel, Hans-Caspar Bodmer, Klaus Seidel

Partner:	Resource Analysis (Netherlands), Cemagref (France)
	Stand Montafon (Austria), Les Arcs (France)
	Ecoscan (Switzerland), University of Amsterdam (Netherlands)
	RSL - Remote Sensing Lab (Switzerland), Silvretta Nova (Austria)
	Het Frankrijk Huis (Netherlands), Sion 2006 Bid Committee (Swizerland)

Objective: Development of a management information system (MIS) to evaluate impacts of ski-resort activities on the environment and to support the sustainable management and tourism of a region.

This project aims at developing a methodology based on satellite data to provide a cost-effective assistance in the monitoring and sustainable maintenance of ski-regions. This includes issues as: Environmental aspects such as vegetation indices, landscape changes, snow cover, socioeconomic issues such as tourism potential and economic development.
More info

MINORA - MINiaturized Optical Range camera for safety, surveillance and automotive Applications

Participants: Esther Koller-Meier, Frank Ade

Partner:	CSEM - Centre Suisse d'Electronique et de Microtechnique SA
	Institute for Computer Science and Applied Mathematics, University Berne
	Design Center for Integrated Circuits, EPF Lausanne
	Microswiss, Leica AG, CEDES AG

Objective: Development of universal range cameras based on optical ranging techniques for safety and surveillance applications.

Modern everyday life is characterized by an ever increasing interaction between man and machines leading to a growing number of potentially harmful situations through accidents, malfunctions or human oversight. Related to this is our investigation into reliable presence detection systems based on range image sequences.

MINORA is a 4-year project in the OPTIQUE II programme of the ETH Council in collaboration with several academic and industrial partners. The purpose of the project is to develop range cameras based on optical ranging techniques (time-of-flight or AM laser radar), with which a large part of today's safety and surveillance applications can be solved. These new sensors, working in the near infrared, will be fast, cheap and can supply 3D information with high accuracy. However, a necessary trade-off means that the sensors provide coarse range images resulting from the need of inexpensive sensor and computing hardware.
More info

COGNIS - COmputer Guided Nannofossil Identification System

Participants: Siegfried Brechner, Frank Ade

Partner:

A 3-year ETH project in collaboration with the Institute of Geology. The purpose of the project is to develop time-efficient semi-automatic and automatic methods to find, measure, identify and count micrometer-sized coccoliths in Scanning Electron Microscope (SEM) images from ocean sediment specimens.

Projects finished in 1999

BIOMORPH

Participants: Paula Muñoz, Manuel Sturm, Martin Styner, Gabor Székely, Guido Gerig

Partner:	University of Canterbury at Kent, UK
	Oxford University, Department of Psychiatry, UK
	Catholic University of Leuven, Belgium
	INRIA, Sophia Antipolis, France
	The Maudsley Hospital, London

Goal: Evaluation and extension of computing techniques for morphometry.

The BIOMORPH project was a collaboration between leading European computer science and clinical groups to evaluate and extend state of the art computing techniques for morphometry, i.e.. for the quantification of size and shape of the biological structures. The project was focused on applications in schizophrenia and multiple sclerosis (MS), conditions where the need for improved brain morphometry was particularly clear. In schizophrenia, changes in the morphology of various brain structures provide important clues to the most fundamental brain abnormalities that underlie the condition. In MS, quantification of changes in lesions has become of great importance for pharmaceutical trials, and improved morphometry will reduce the cost of developing new drugs.

Several algorithms were developed for the representation of shape and for improved quantification of brain structure including a method for parametrizing the shape of objects such as the hippocampus. The programs were then applied to investigate the corpus callosum outline after it had been segmented in each high resolution scan of an MRI series from 71 individuals.

The development of techniques for MR scan analysis in MS patients was mainly focused on a set of twenty-five patients, who had been undergoing serial volumetric MR scanning. Our group developed a new approach for the automatic detection of temporal changes in this 4-dimensional Datasets motivated by techniques in functional MR imaging. These spatio-temporal data resulted using highly effective co-registration procedures provided by the cooperating BIOMORPH partners. Time-variant properties of all voxels were examined and combined to a probabilistic map for lesion activity during the observation period.

Portal Imaging

Participants: Martin Berger, Gabor Szekely, Guido Gerig

Partner: University Hospital Zürich

A clinical research project funded by the cancer research of the canton Zürich to improve quality assurance in radiotherapy treatment. Electronic Portal Imaging Devices (EPID) enable us to register megavoltage X-ray images of the treatment field during irradiation. These portal images are then analyzed using a high precision and area-based matching algorithm in order to measure patient setup deviations.

PET-MRI

Participants: Jonathan Oakley, Gabor Szekely, Guido Gerig

Partner: PSI - Paul Scherrer Institute, Switzerland

The aim of this project is to support the better quantification of Positron Emission Tomography (PET) images on the basis of associated structural information. This would involve an addressal of the Partial Volume Effect (PVE) , which basically demands better resolution data. Improved resolution is possible using statistical methods of reconstruction. The approach taken here uses Bayesian methods to employ a priori estimates of the activity distribution to regularise the solution whilst encouraging distinct variation across structural boundaries. The prior is derived on the basis of a ``forward model'' of the emission process, a correction of the PET data constrained according to the known structure. The result is a high resolution estimate of tracer distribution toward which the reconstruction solution may be drawn.

Image Indexing

Participants: Alex Dimai, Gabor Székely

Partner:

An ETH Project: The project "an integrated image analysis and retrieval system" is a joint project with the computer systems and the database group of the Computer Science Department the ETH. The vision lab studies which visual features can efficient indexing into an image database. The development the appropriate feature extraction algorithms is examined, as well.