Motion tracking of flying insects

Our studies of insect flight start with high-speed digital movies of flight and three-dimensional motion tracking of the captured sequences. For details about these procedures, see our paper in the “Publications” section of this site. We have three orthogonally-arranged high-speed cameras that aim on a clear flight chamber that contains the insects. Our approach is to automate the filming apparatus in order to get many, many movies, and afterwards we can pick our favorites for further analysis.

 

 

Here’s how to automate the apparatus: Aim two laser beams so that they cross in the region of interest and hit photodiodes, and then make a triggering circuit that outputs a signal when both beams are broken at the same time. Wiring this signal to the trigger-input of the cameras ensures that they are triggered only when an insect is in the filming region. This approach generates a lot of data without the need for constant attention.

 

Here’s a typical video that we get from our apparatus. The three panels represent the top view (left-most) and two horizontal views. This is a tiny fruit fly (D. melanogaster) which is about 3 mm in body length and beats its wings 200-250 times per second in flight. We’re filming at 8,000 frames per second and zooming in on the insect. Notice also that, because we back-light each view, the images are silhouettes of the insect on a lighter background.

 

 

All that video data leads us to a second problem: How do we extract useful information from the movies? One way is to manually extract the orientations and positions of animals and their wings. Many people, including us, have written manual tracking programs. You basically have to position and orient a model of a fly so that it overlays the filmed images in each camera view. Manual tracking is very tedious, so we also automated the tracking process.

 

 

We call our method HRMT, pronounced “hermit” and which stands for Hull Reconstruction Motion Tracking. HRMT automatically extracts the wing and body orientations and positions for each frame of the movies. Here’s the basic idea: We first ask, How much information is actually contained in these movies? Well, we have silhouette data from three views, so we are trying to determine shape from silhouette, a common computer vision problem. A multitude of 3D shapes can lead to identical silhouettes, so we do not know the 3D shape of the insect. However, we can form the maximal-volume shape that’s consistent with the three shadows: this is called the visual hull, and it’s shown above. You can think of this shape as the volume of dough you would get from intersecting three cookie-cutters that are in the shape of each silhouette.

 

 

This reconstructed beast, or more endearingly “Franken-fly”, is larger than the actual insect and includes protuberances and protrusions that correspond to regions blocked from all three views. To determine flight coordinates, we devised a set of procedures that extract positions and orientations in a way that is insensitive to these protrusions. First, we identify groups of nearby points, and the clustering algorithm picks out each wing and the body individually. Then, we associate the centroid of these clusters with the position of each the body, right wing, and left wing. To get the orientation of each part, we use a combination of principal components analysis (PCA) and geometrical information about the insect. In the end, HRMT automatically collects 18 coordinates for each frame: the position and orientation for the body, right wing, and left wing.

 

The approach is nice because it doesn’t require any inputs about the morphology of the insect and doesn’t require us to prepare the insects in any way (like speckling it with markers). It’s also fast and accurate: each stroke takes just a few minutes to run on your typical desktop computer, and we determine coordinates to within about 2 pixels in position and 4 degrees in orientation. We tested this by conducting a very thorough error analysis on synthetically-generated data. Basically, we imposed kinematics on a computer-generated model fly, then formed the shadows of this model, ran HRMT on the shadows, and finally compared the extracted coordinates to those we imposed.

 

We would be happy to share our code with others interested in motion tracking of moving animals. We have a mini-tutorial, a version of the code, and some sample images on this site. For more information, contact Leif Ristroph for more information: lgr24@cornell.edu.