The process of infrastructure management includes: (i) monitoring and evaluation, (ii) planning (iii) design, (iv) construction, and (v) operations and maintenance, all linked through a common management information system. Several of these activities require sensing the infrastructure and its environment. Advanced sensing approaches are therefore more and more viewed as important tools for assessing performance, improving speed and quality of decision making processes, as well as increasing safety. Driven by new technologies, different applications have already been investigated and some of them have already been adopted by companies. They include: as-built status assessment using LADAR/LIDAR technology, compliance checking using a single-axis laser and primitives-fitting algorithms, as well as work environment modeling using a single-axis laser and convex-hull fitting algorithms. LADARs/LIDARS applications are characterized by significant data processing durations (a few days) and single-axis laser based applications are characterized by acquisition times of a few minutes per object. However, as previously explained, the construction industry needs to take real-time decisions, and for this reason it needs real-time information. Specifically, heavy construction faces safety and productivity issues directly related to the difficulties surrounding the operations of heavy equipment. Real-time automated obstacle avoidance and path planning in-board calculators would definitely improve these factors and more generally the overall performance. Real-time 3D modeling may become feasible with a new technology called ``flash LADAR''. Like a digital video camera it acquires streams of images with frequency up to 30 Hz, but, instead of brightness, each pixel of the 160x124 array stores a range value. Algorithms are now being developed to use these capacities with the aim of revolutionizing machine vision. Unfortunately, the flash LADAR images are very noisy, much more than brightness images in equivalent environments. This implies that, even with the best efforts for cleaning images, the accuracy for detecting objects and evaluating their sizes and speeds is low and possibly still insufficient for safe obstacle avoidance applications. This issue could be overcome by applying complex pattern recognition algorithms, but they are usually too slow to be applied on real-time applications. This paper proposes an approach to improving the quality and speed of obstacle detection algorithms by involving some a priori knowledge. Although such knowledge is not available in most pattern recognition problems, it is in the case of construction. 3D CAD provides site layout information that could be used by real-time machine vision programs. With this objective in mind, the issues of the integration of 3D CAD engines' and flash LADARs' data must be addressed. Indeed, design processes provide well defined information including perfectly parallel, perpendicular, flat, etc. forms (strong forms) like pipes, beams, columns and floors, whereas weak and non-parametric forms are produced from existing infrastructure conditions. There are then two major challenges: (1) defining a way to evaluate the two different types of data in a common model, and (2) developing methods for comparing one to the other in order to create reliable environment models in a real-time manner. The paper presents the investigation currently conducted in the Civil Engineering department at the University of Waterloo. The issues described above are discussed and solutions suggested. Finally, some early results are presented demonstrating the feasibility of this proposed approach and justifying the continuous effort in this direction.