Image Aided Flight Control (IGSSE Project 4.03)

Project Overview

Recent advances in the field of machine vision together with the rapidly increasing computing power available on miniaturized platforms open the perspective of utilizing image based or derived measurements as auxiliary or primary source of measurement for flight control applications. Besides the need for deterministic real-time behavior at high frame-rates and low latencies , rigid requirements addressing accuracy, availability and integrity of the image derived measurements represent challenges to be overcome to successfully exploit the technologies for flight control purposes. The primary motivation to use image based information as a further signal source is to be capable of flying in areas where other sources of navigation data are unavailable, like for example GPS during indoor navigation and to orient in non-cooperative dynamic environments, where the location of obstacles and other objects cannot be determined by comparing the own position derived from conventional navigation sources with a database.

The particular research focus of the proposed project will be on the tight and close integration of the contributing components. New specific sensor data fusion algorithms are to be implemented. The tight bi-directional integration of image based and conventional navigation sources aims at increasing the operational robustness and reliability of the overall system. To fully utilize the capabilities of the approach, custom flight control algorithms are to be developed. Specific challenges on the image processing side is to maximize frame rates, decrease latencies and provide robustness against changing environmental conditions like light, shadow and contrast while simultaneously minimizing the amount of control power required. To prove the theoretical results obtained in the project, actual flight tests are planned with different small-scale flying vehicles. Benchmark applications could be position and attitude stabilization and autonomous formation flying.

Setup

Quadrocopter

As hardware platform, we are currently using the Hummingbird AutoPilot Quadrocopter from Ascending Technologies. The vehicle has the following specifications:

• Dimensions in cm: 40x40x10
• Weight (ready to fly): 500g
• Cruise speed: 36 km/h
• Max. speed: 50 km/h
• Max. Payload:200g
• Max. Flight Time: 20 min (10-12 min at max. payload)
• Sensors: GPS, MEMS Gyros, 3D-Magnetometer
• 2 ARM 7 micro processors (onboard stabilizing controller)
• Serial Port Interface (In: motor commands Out: all sensor data)

Tracking System

Currently we are using an external stereo tracking system, in order to determine the 3D position and orientation of the vehicle. The tracking measurements are sent to the controller application for further processing (sensor data fusion) and for generating the appropriate commands for the vehicle. The stereo camera system consists of two standard webcams, which have been calibrated for their intrinsic and extrinsic parameters.

The tracking algorithm uses an contour based approach, making use of a known model of the object to be tracked. Then online, the visible features points of the model are computed by taking into account the predicted state of the current time step. At these feature points, the algorithm searches for edges in the image, as well as computing color statistics for the model. These information get fused together using a Gauss-Newton optimization to obtain a single pose estimate.

Furthermore for security reasons and better visibility, we received additional rotor guards and casing for the Hummingbird as shown in the picture on the right. The whole system is divided into submodules. The visual tracking is handled using OpenTL (C++) whilst the control loop is developed using Matlab Simulink. Most parts of the control loop, already have been moved to the onboard processors of the vehicle, to minimize latency issues.

Current Development

• We are currently working on an inside out system. The goal is to perform autonomous indoor flights, as well as exploration and other higher level missions (see e.g. UAV competitions like EMAV, IMAV or ultimately IARC.
• On the controller side, the research focus goes into adaptive control and autonomous formation flights

Videos

IAFC at the Mathworks booth of the Embedded World 2010 in Nuremberg

This video shows our system running at the Embedded World fair 2010 in Nuremberg. The Mathworks, gave us the opportunity and space at their booth to show our system during the three days of the fair. During the fair, we constantly (every three seconds) recorded images and transformation data resulting in this time-lapse video.

Markerless vision assisted flight control

This video shows our markerless external tracking system. The system automatically (re-)initializes the tracking using the color information of the model (blue for the center and green for the heading). After init, we are making use of the intensity edges in the image and a known CAD-Model of the quadrocopter for sequential estimation. The tracking algorithm runs at 25Hz, while the onboard controller of the vehicle, fusing the visual measurments with the inertial measurements, runs at 1KHz. The video shows the vehicle flying different pre-stored trajectories, flying under joystick control and reactions to external disturbances.

Marker-based Visual Tracking for position holding

This video shows our first system. We were using two external cameras and a styrofoam marker attached to the vehicle to ease the tracking. The model of the marker is tracked by using a fusion of intensity edges and color statistics from both cameras. The overall framerate of the visual tracking algorithm was approx. 18 frames per second and the control loop runs in a simulink model on the same ground station PC. The video shows the flying quadrocopter, superimposed with the projected estimated position of the styrofoam marker in green.

People & Partners

Robotics and Embedded Systems

• Dipl.-Inf. Sebastian Klose

Flight System Dynamics

• Jian Wang, M.Sc.

Publications