Multi-view reconstruction of the human cornea as a reflective surface

Project overview

In order to diagnose different distortions, injuries of the human cornea,, e.g. for a post-surgery state monitoring, ophthalmologists (eye specialists) use a camera- and image processing-based diagnostic instrument, the corneal topographer. In the classical approach, a pattern consisting of concentric circles (so-called Placido-rings) are projected on the corneal surface which is then photographed by a camera. Based on the distortions of the rings, an image processing system deduces the potential 3D geometry of the surface under analysis. The result is typically displayed as a curvature map, a diopter map or an elevation map. The approach is non-invasive and builds on the reflective nature of the cornea. However, local information can be extracted in the radial direction only, i.e. no circumpolar shape information is available, and the reconstruction is sensitive to an assumption concerning the mean shape of the cornea.

Figure. Traditional cornea topography projecting rings onto the human eye (left) and computing, e.g. an elevation map (right) from the distortions of the rings.

To eleminate these drawbacks, the Cornea project aims to develop a prototype multi-view high-precision reflective corneal topographer device to determine the 3D shape of the human cornea, and to visualize a map or 3D model of the measured cornea for close inspection by the ophthalmologist (eye specialist). The project covers the design and implementation of all calibration and reconstruction algorithms, software and hardware components of the device. I was responsible for multi-view calibration of the device, and the development of the reconstruction algorithm, as well as testing the reconstruction concept via ray-tracing simulations.

Project participants are the Systems and Control Lab (Computer and Automation Research Institute, MTA-SZTAKI, Budapest), the Department of Ophthalmology (Semmelweis University, Budapest) and the Department of Numerical Methods (Eötvös Loránd University, Budapest).

Measurement instrument

The device consists of a head fixation to support the chin of the patient, a flat LCD display oriented towards the head of the patient to project a pattern into his/her eye, and three cameras placed around the display, each oriented to observe the eye of the patient. The device takes synchronized photographs of the cornea with the reflected pattern. The computational task is then to locate and identify each feature point of the reflected pattern in each image and then to reconstruct the unknown reflective surface from the observed distortions of the generated pattern.

Figure. The developed corneal topographer device and its usage. Pattern reflected in a human eye and in an artificial cornea. In addition, several different artificial corneas were manufactured that emulate different typical abnormal eye deformations.

Reconstruction algorithm

Assuming that the display-camera arrangement is known to high precision, the first task is to estimate the distortion of the generated pattern in the eye based on the identified and detected feature points. Then, features identified in many views, i.e. laying in the narrow overlapping visibility regions of the surface are used to reconstruct individual 3D points on the surface based on reflection and stereo principles. These reconstructed 3D seed points of the surface are finally used to extend the 3D reconstruction over the visible part of the surface per view, based on the observed 2D distortions of the pattern.

Figure. Model of the pattern generated on the display (left) and its distorted projection to the image taken by a camera (right).

Figure. Stereo reconstruction of seed points on the corneal surface (left) and extending the solution from seed points based on the estimated pattern distortions per view (right).

Figure. Reconstruction results. (1) Spline interpolation of the pattern grid using the detected and identified feature points in an image, (2) reconstruction paths over the pattern, starting from a seed point, (3) visible area and 3D reconstruction of the surface region with feature coverage in the image, (4) surface visibility and reconstructions of the areas observed by three cameras (overlayed on the mean sphere).

High-precision calibration

The reconstruction algorithm relies on precise knowledge of the arrangement, i.e. intrinsic paremeters and exact poses of the cameras-pattern ensemble. To calibrate the system, we first use a tiny checkerboard-pattern placed on the slope of a small wedge, which is rotated around a vertical axis at the approximate future location of the eye of the patient. The precise location does not need to be known in advance. The camera intrinsics can than be calibrated individually. We estimate the coarse poses of the cameras with respect to a flat checkerboard equally visible by each camera.

Figure. 1st calibration stage: (1) checkerboard pattern fixed to a wedge for intrinsic calibration, (2) flat checkerboard pattern equally visible by each camera for relative pose estimation, (3) images collected from the calibration procedure for one of the views, (4) computed checkerboard positions relative to the same view.

The checkerboard method is not precise enough for this application. To show this, we replace the checkerboard with an artificial cornea, generate a grid pattern on the display, and backtrace the rays accross the grid points detected in the images (ray-tracing). If the model of the arrangement is precise, the rays should hit the pattern at the correct feature location. However, a significant discrepancy is observed. Therefore, in a second stage, we further optimize the poses in the camera-display arrangement (together with the pose of the artificial cornea) by minimizing these backprojection errors on the pattern.

Figure. 2nd calibration stage: (1) discrepancy between known feature points of the pattern and the back-traced locations of the feature points in the image using ray-tracing, (2) the same figure after optimizing the arrangement by minimizing ray-tracing discrepancies, (3-4) 3D model of the optimal arrangement.

Figure.The three images of the artificial cornea used for calibration and the overlayed feature points. Renderings from the same three viewpoints based on the optimal 3D arrangement using our ray-tracer developed in Matlab. The top part of the heimsphere (corresponding to the cornea) is identical in the simulation and in reality, showing a precise calibration.