Embedded systems engineering works

Vision system hardware designs using FPGAs

In our StereoLaneFinder project at the Systems and Control Lab, MTA-SZTAKI, we worked on a prototype system for detecting traffic lanes using a stereo vision system mounted to a test car, primiarily for lane departure warning and automatic leane keeping assistance. For more information on the application, please visit this project website.

The "eye" of the system is an Elphel333 network camera with an embedded FPGA and Axis processor running Linux. The camera, in its original configuration, could be accessed via Ethernet, through a browser interface from a PC to grab still images, access camera settings or start a streamer. An RTSP-capable media player could be then used to see a continous video stream from the camera remotely. The full camera configuration was relatively complex, including an MJPEG encoder in the FPGA, several Linux kernel-mode drivers, user-mode daemons and utility programs running in the camera.

As a result of around a year of development, we transformed two such network cameras into a synchronized stereo vision system capable of real-time stereo image processing. This transformation included a complete redesign of the FPGA configuration in Verilog, implementing new Linux drivers (I/O, DMA and interrupts), a user-level camera software API to access the FPGA and the CMOS vision chip, and C++ vision software for both mono and stereo image processing. The new system can be configured and can stream frames for the vision software using our proprietary protocols over TCP/IP.

Figure. The Elphel333 camera hardware platform. In several stages, it has been completely reconfigured, including a new Linux software distribution, new FPGA configuration (in Verilog), Linux drivers, embedded software (in C), TCP/IP video streaming and host-side acquisition and processing software (in C++). See also the hw block diagram of the camera.

Figure. Stereo vision system for demonstrating real-time stereo image processing algorithms. The system is based on the completely reconfigured and reprogrammed Elphel333 cameras.

Figure. Software (left) and FPGA hardware components (right) developed by our group for the real-time stereo image processing based on the Elphel333 camera. Click to enlarge. For more information on the host-side acquisition and vision software written in C++ by András Bódis-Szomorú, please visit this project website.

In another project at the Dept. of Measurement and Information Systems, TU Budapest, we built a low-cost stereo acquisition hardware from chip level. A USB link transmits control/status information and the acquired images from the camera to a PC. The theoretical throughput of the image streaming link is 7 MBytes/sec (56 Mbps) which is enough for 45 stereo frames per second with a resolution of 320x240 pixels. However, 15-20 fps (33-44% of theoretical throughput) suffices in most of our applications, such as obstacle detection by mobile robots. Our aim was a low cost acquisition hardware with simple design at chip level instead of very high achievable troughputs and local computational capacity.


Figure. Low-cost digital stereo acquisition electronics designed and built from discrete chips.

Features of the system are the followings.
  • on-board mini-lenses with 54-degrees field of view,
  • two sensor boards, each equipped with an automotive VGA-resolution CMOS vision chip operating with global shutter at 27 MHz dot clock. The CMOS chip has I2C configuration and 10-bit parallel pixel interface.
  • flexible stereo baseline setup depending on the application: parallel cable connections between the sensors and the main board with careful design for signal reflections,
  • FTDI chip providing 480 a Mbps USB 2.0 interface to the PC, an SPI interface (for camera control and status messages) and a 7 MBytes/sec parallel interface (for image data) to the internal logic,
  • low-cost Spartan-3 FPGA chip as glue logic between the communication chip and the sensors, for controlling the timings of image acquisitions (sensor syncronization), for streaming the data to the USB chip through large FIFO buffers, and for simple image preprocessing (e.g. collecting hisogram data or performing pixel-wise operations, image compression etc.),
  • AVR microcontroller as a communications engine with coordination tasks while keeping FPGA requirements low,
  • SPI, I2C and proprietary parallel interface between the microcontroller and the FPGA,
  • 5V to 3.3V, 2.5V and 1.2V power electronics,
  • extension connector with access to FPGA pins (e.g. for a future memory or wireless extension),
  • communications design: the user accesses the hardware for configuration or status retrieval from the PC via a C++ API built on top of the FPDI communications API. The microcontroller interprets user messages and serves as a gateway towards both the vision sensors and the internal FPGA modules. The sensors are triggered by the FPGA, which collects and caches the images, and streams them to the USB chip.
The work included system design, calculations and design of communication speeds and protocols, preliminary measurement of signal reflections, chip selection, PCB design, hardware assembly, FPGA hardware coding in a Hardware Description Language (Verilog HDL), embedded C coding for the microcontroller and C++ coding for the PC side.

FPGA-based image filtering

In 2005, I have been asked to create a simple FPGA-based image processing demo from limited hardware resources at the Systems and Control Lab, MTA-SZTAKI. Since the Spartan3 demo board given as a base had low-capability built-in standard ports (PS/2, RS-232, 3-bit color VGA) for such an application, I designed and built a VGA extension board hosting an Analog Devices triple 8-bit DAC chip to enhance at least the display capabilities of the FPGA-board.

Figure. Spartan3 board and its extension board to drive a VGA monitor in 24 bit colors using a 50-MHz VGA DAC chip. The main board is connected to a PC via RS-232 for downloading images, an FPGA cable for downloading FPGA configuration, whereas the extension board is connected to a monitor via a VGA-cable.

Using this setup, a real-time image filtering demo application was then implemented. A 640x480, 24-bit (900 KByte) raw image can be downloaded into the SRAM through the FPGA via the 115.2 kbps serial line in "download mode". Then after being switched into "process mode" with a switch manually, the FPGA reads the image from the SRAM and feeds row-by-row into a multi-purpose processing chain, whose output goes to the monitor through the DAC extension board. The core of the processing chain is a 3x3 FIR module that is pipelined to boost it for 150 MHz, triple of the 50 MHz dot clock available at the output of the processing module. The monitor is set into a 800x600@72 Hz mode by the corresponding interface module in the FPGA. There are several 3x3 sets of FIR weights hard-coded into the processing module: Sobel edge detector, Prewitt edge detector, several compass masks, Gaussian blur filter. These sets can be selected with the manual switches. Additionally, the image can be converted to gray scale and/or inverted on-the-fly. The FPGA configuration was designed and tested in Xilinx ISE Webpack in Verilog HDL. See this hardware block diagram for more info.

Motor control modules based on AVR microcontrollers

I have participated in the design of several motor control modules for the mitmot platform, a general-purpose modular embedded system (mote) with AVR/ARM CPU modules, sensor modules, radiocommunication modules etc.

The 1st motor control module can drive three special brushed DC micromotors: two servos with continous turn for moving a robot and an additional "head servo". It has three types of sensors: motor current measurement (low-side shunt) for torque control, optical sensors for rotation measurement and a potentiometer embedded into the head servo. The drive electronics are built from discrete MOSFET elements and gate drivers for flexibility. The core of the module is an 8 MHz AVR microcontroller (ATmega32A). It receives control commands via I2C interface from a supervisor CPU module, measures the current and the relative positions of the motors, and drives them by PWM signals according to the supervisory request.

The work included system design, motor modeling and simulations in Matlab Simulink, circuit simulations in PSpice, PCB designs in OrCAD, manual assembly of the hardware, design of the proprietary communication protocol over I2C interface, C programming of two AVR microcontrollers (using WinAVR) in Eclipse and AVR studio. The work was honored by a Siemens Excellence Award.

Figure. 1st version of AVR-based motor control and sensor modules designed, built and programmed by András Bódis-Szomorú in his one semester diploma work.

The 2nd motor control module has a simplified, clearer design with current sensing capability eliminated, decreased PCB size to about (4x6 cm), and redesigned software API for easy use by students. This version has been built into six three-wheel robots, and has been used for demonstrations and educational purposes both by fellows and students for years. This work was honored by the Josef Heim Award of the Scnell Foundation.

Figure. 2nd motor control and sensor modules designed and programmed by András Bódis-Szomorú. It was built into six three-while robots used for educational and demonstrational purposes by students and fellows of the department for years.

A 3rd motor control module was designed by Csaba Nemes and András Bódis-Szomorú for driving two high-quality precision micromotors with lower stall current (444 mA @ 12 V vs 800 mA @ 6 V), narrower stall dead-zone (0.1 V vs 0.4 V), higher speed (180 rpm vs 66 rpm on the outer axis), built-in quadrature encoders (resolution of 0.17 vs 6 degrees), compared to the previous motors. In this version, an L298 double full bridge drive circuit (with bipolar transistors) replaced the discrete MOSFET ICs of the original design. The saved PCB area could be used to re-implement the current measurement as a differencial phase shunt circuit. The related study also investigates the maximum torque required to keep this mobile inverted pendulum in balance on two wheels, one final purpose of the motor control module.

A 4th motor control module was designed by Tamás Szabó and András Bódis-Szomorú. This is a speed- and power-optimized version of its predecessor. The L298 bipolar power electronics (2.5 Ohms equiv. resistance) have been replaced by L6207 FET dual bridge (0.6 Ohms channel resistance) with built-in support for current control, which can be overriden by software for a voltage-controlled application. The 8-MHz ATmega32 is replaced by a 32-MHz ATXmega128 microcontroller that has a much improved internal design. Also, the slow I2C-communication is replaced by an SPI-communication which is around 10x faster (few us instead of few ms per command). Fast communication makes possible to implement closed-loop control at the level of a supervisor module rather than inside the motor control module. As an example, this simplified the overall modeling and control of this balancing robot.

Figure. 4th motor control module with L6207 dual-bridge FET drive electronics and ATxmega128 microcontroller. Designed by Tamás Szabó and András Bódis-Szomorú.

A panoramic ultrasonic obstacle detector

In robot navigation, ultrasonic transmitters and receivers provide a cheap, active solution for detecting obstacles in close range, compared to cameras, radars or lidars. Motivated by this, a 360-degrees ultrasonic module was developed by Edit Csakurda, supervised by András Bódis-Szomorú. The work was awarded the 2nd prize Thesis Award of the Hungarian Scientific Society for Measurement, Automation and Informatics (MATE) in 2010.

Figure. (1) AVR-based 360-degrees ultrasonic obstacle detector module designed by Edit Csakurda under the supervision of András Bódis-Szomorú, (2) a preliminary experiment with an emitter-sensor pair. Ultrasound generation with 10 pulses of 10 V and the signal reflected from an object at a distance of 5.6 cm from the source. The carrier is 40 kHz.

In sonars, the sound is typically generated with several voltage square wave cycles at the source and the ultrasonic wave reflected from objects is converted to voltage signals in one or more ultrasonic sensors. The distance of an object from the device can be calculated from the time elapsed between sound generation and detection of the reflected signal, using the known speed of ulrasound (346 m/s in the air at room temperature). The panoramic sonar module is compatible with the mitmot platform, a general-purpose modular embedded system (mote) with AVR/ARM CPU modules, sensor modules, radiocommunication and these motor control modules etc.

Our sonar works for obstacles up to 1 meter in a more effective way than off-the-shelf detectors. it is capable of driving up to 3 emitters at the same time and can measure all six, 60-degrees space regions within only 20 ms, while it effectively avoids secondary reflections from other sources. Signal conditioning is a relatively difficult task, as the receiver has to detect the envelope of even 20 mVpp signals at 40 kHz. The topology and the parts (e.g. opamps, diodes) incorporated have to be selected with special care. The receiver circuitry is a precision envelope detector consisted of a special two-way precision rectifier and a second-order Tsebyseff-filter in Sallen-Key topology. 3 channels can be measured parallely and the microcontroller can select the required channels via an analog multiplexer (MUX). The system works from +3.3 V power, although, it internaly generates a -3.3 V voltage for analog singal conditioning (as the detected voltage waveform is symmetric). The envelope of the reflected wave is sampled by the ATXmega microcontroller using its embedded A/D converter, operating at 16 MHz for precise distance measurement. The module provides I2C, SPI and UART communication interfaces. The study is very profound, including Matlab-based spectral analysis, circuit simulations, and a lots of preliminary measurements with preliminary verification of almost all surface-mounted circuitries used in the final design.

Bluetooth radio communication module

In this project, we built a Bluetooth radio communciation extension for the mitmot platform, a general-purpose modular embedded system (mote) with AVR/ARM CPU modules, sensor modules, RFI modules etc. The Bluetooth module is based on Bluegiga's WT12 Multi-Chip Module (MCM), which contains a 16-bit RISC microcontroller running the Bluetooth software stack, a baseband digital signal processing (DSP) unit, 2.4 GHz RF unit, chip antenna, 48 KBytes on-chip RAM and 8 MB Flash, all within a space of 25x14x2.4 mm. It has several communication interfaces: among others, an SPI interface for reading/writing chip configuration parameters, an UART and an USB interface both for chip configuration and radio data readout/sendings and an USB interface.

Figure.The Bluetooth radio communication module (left) and the extension module (right) that provides power, reset button and two port connectors for the SPI and UART interfaces of the Bluetooth chip.

Our Bluetooth board provides access to the communication interfaces of the Bluetooth chip. The USB interface of the bluetooth chip is connected to a mini USB connector, for direct access from a PC, while its SPI and UART interfaces are connected to a Xilinx CPLD that is capable of routing these channels either to the mitmot system bus or to the FTDI chip, depending, for example, on the state of the mode switches that can be set manually. The FTDI chip is capable of tunneling both the SPI and the UART channels through a USB to/from a PC via a second dedicated mini USB connector. The extra extension board allows us to use the Bluetooth module without the mitmót processor module by providing +3.3 V power and a reset signal to the bus. Moreover, it routes the SPI and UART signals to an external connector for connecting a computer to the Bluetooth chip.

Figure. Usage of the bluetooth card (1) with and (2) without the mitmót processor module.

The easiest way to use the Bluetooth chip is via its UART interface. The chip provides a simple command interpreter, called iWRAP, residing on the top of the Bluetooth protocol stack. In command mode, iWRAP interpretes ASCII commands received via UART, and sends back status info in ASCII. In data transmission mode, bytes sent/received through Bluetooth radio are communicated via UART. The chip hardware supports Bluetooth 2.0 Basic Data Rates up to 723 kbps, Enhanced Data Rates (EDR) up to 3x723 kbps=2.1 MBps and UART interface data rates up to 2700 kbps. However, the embedded iWRAP software interpreter restricts speeds to 650 kbps between a WT12 and a Bluetooth dongle, or to 570 kbps between two WT12 chips. However, using USB and Host Controller Interface (HCI) instead of UART and iWRAP, the overhead of the iWRAP interpreter can be eliminated and the data rate may increased. It is only a software configuration issue whether UART/iWRAP or USB/HCI is used. Even 570 kbps outperforms the radio transmission data rate (115.2 kbps) so far available in the mitmót platform.

Figure. iWRAP welcome screen as displayed in HyperTerminal running on a PC. This text is sent out by the Bluetooth chip after boot.