OUR GOAL
Understanding the mechanisms of insect flight and implementing them in biomimetic robots
Flying insects can perform a wide range or aerial maneuvers better than any man-made flying device. Some insects can land upside down, change direction in a split of a second and engage prey in mid-air. To achieve such impressive capabilities, insects have evolved unique flight mechanisms, including a control system that exhibits reflexes among the fastest in the animal kingdom.
Our goal is to understand these flight mechanisms and develop biomimetic flying robots that use them. This challenge involves cutting-edge concepts in computer vision, machine learning, control theory, mechanical design, computational fluid dynamics, physiology and neuroscience.
TED-style talk: why is it hard to catch a fly?
Insect flight is a fascinating and interdisciplinary field of research, lying between physics, biology and engineering. In this TED-style talk, Tsevi discusses how our group studies the flight of fruit flies. The talk was given at the annual event of the Azrieli Fellows Program.
A hull reconstruction-reprojection algorithm for pose estimation of free-flying fruit flies
We developed an experimental system and a computer-vision algorithm for measuring the intricate motion of a flying fruit fly in 3D. The experimental setup consists of four ultra-fast cameras that go up to 22,000 frames per second. Our model-free algorithm reconstructs the wing boundaries and body in 3D. The segmentation of the body and wings is overlaid on each of the four video panels. Note that this method allows us to detect even parts of the wings that are partially occluded. Finally, the 3D reconstruction is used to measure the fly's motion, including body and wing orientation, as well as wing deformation.
Learning to soar like a vulture
Thermal soaring, used by large birds and human glider pilots, allows them to fly with high energetic efficiency for long distances. It’s a behavior with rich dynamics and nontrivial constraints, and a very appealing target for biomimetic and autonomous aerial vehicles. We present a deep reinforcement learning (deep-RL) framework for solving the thermal soaring problem in a simulation environment under strong horizontal winds. Compared with vulture flight data, the resulting autonomous agent spontaneously adopted a vulture-like soaring technique that successfully and robustly exploited thermal updrafts under horizontal winds. Our ‘agent’ managed to locate the moving thermal and exploit its updraft by simultaneously chasing it and circling its center. Further, we used this deep-RL framework to reveal principles of the thermal soaring problem and interpret the functionality of the agent’s neural network. For example, we found that this problem consists of two critical bottlenecks that must be solved sequentially: achieving stable flight and flying near the thermal center. Like learning to crawl before learning to walk and run, we find that thermal soaring is practically too difficult to solve directly, i.e., by posing the final behavioral goal. Rather, it must be solve sequentially, by guiding the learning system through the problem’s bottlenecks.
Lateral dynamics of fruit flies are unstable and governed by wing-wing interaction
Understanding the uncontrolled passive dynamics of flying insects is important for evaluating the constraints under which the insect flight control system operates and for developing biomimetic robots. In this work, we analyze the passive, uncontrolled stability of a fruit fly by combining accurately measured wing kinematics and full CFD analysis. Focusing on the fly's lateral dynamics, we find that they are unstable due to an oscillating-diverging mode with a doubling time of 17 wingbeats. This instability stems from a negative coupling between sideways velocity and roll torque, and we show that considering both wing-wing interaction and using accurate wing kinematics are crucial for understanding this coupling and instability. Furthermore, we managed to combine this open-loop dynamics with our current understanding of the fly's roll controller, which has a single-wingbeat latency. Solving the resulting closed-loop system shows that this controller is consistent with the lateral instability we have characterized. To the best of our knowledge, this is the first time such a link has been formed between a CFD-based instability analysis and the insect's flight controller. This link may shed light on the constraints under which the insect flight control system has evolved.
Band-type resonance: new non-discrete energetically-optimal resonant states
Conventional linear and nonlinear resonant states typically exist at specific discrete frequencies and specific symmetric waveforms. This discreteness can be an obstacle to resonant control modulation: deviating from these states, by modulating waveform asymmetry or drive frequency, generally leads to losses in system efficiency. For example, it is known that insects modulate their wingbeat frequency and can asymmetrically change their wing stroke kinematics, but do these modulations necessarily incur energetic loss?
In this work, we demonstrated a new strategy for achieving frequecy and asymmetric amplitude modulations at no loss of energetic efficiency. We characterized a new form of structural resonance: band-type resonance, describing a continuous band of energetically optimal resonant states existing around conventional discrete resonant states. These states are a counterexample to the common supposition that deviation from a linear (or nonlinear) resonant frequency necessarily involves a loss of efficiency.
Energy-resonance states
and the elastic-bound conditions
In oscillating systems, such as in biolocomotion, resonance can increase mechanical efficiency by reducing power consumption. This energetic efficiency is achieved by absorbing inertial loads – the forces required to accelerate and decelerate mass – via tuned structural elasticity. However, existing methods for computing energetically optimal nonlinear elasticities struggle when actuator energy regeneration is imperfect: when the system cannot reuse work performed on the actuator, as occurs in many realistic systems. In this work, we explored the fundamental relationship between resonance and energy efficiency in a range of general systems. A key concept is an energy-resonance state, in which the system’s actuator does not perform negative work on the system, i.e. power is transferred only from the actuator to the system and not vice versa. The solution space for the energy-resonant states generalizes the energetic properties of linear resonance, and is described completely via bounds on the system work loop: the elastic-bound conditions. Using this concept, we found some surprising phenomena. For instance, it is widely assumed that, if the system dynamics are linear, then linear resonant elasticity is the only energetically-optimal choice. We show, to the contrary, that there exists a continuum of infinitely many energy-resonant nonlinear elasticities that are also energetically optimal, and provide an elegant method for constructing these elasticities. The choice of nonlinear elasticities from within these bounds leads to new tools for systems design, with particular relevance to biomimetic propulsion systems, and provides new perspectives on the role of nonlinear elasticity in biological organisms.
Automatically tracking the motion of flies
We film free flying insects using three high-speed cameras. One of the main bottlenecks in analyzing these data is to accurately extract the motion of the insect's body and wings. To this end, we developed a model-based algorithm for tracking the motion of free-flying fruit flies. This method is based on fitting a 3D model of the fly, which includes the fly's body and wings, to the images taken by the fast cameras. The movie shows the fitted model superimposed on the raw movies.
Measuring pupil size and light response through closed eyelids
We developed a technology for measuring the size of pupils and their response to light through closed eyelids. These pupil parameters are key components in the neurological evaluation of comatose patients, for example after stroke or brain injury. Our method is based on side illuminating in near-IR through the temple and imaging through the closed eyelid. This technology has been successfully tested in a clinical trial and can be implemented as an automated device for continuous pupillary monitoring. Such a device may save staff resources and provide earlier alert to potential brain damage in comatose patients.
The bright spots in the figure are due to the near-IR light emanating from the eye through the closed eyelid. Imaging in invisible near-IR combined with illumination in visible white-light to induce the pupillary contraction reflex and image the pupil before and after contraction. Using a designated image processing algorithm, we then measure the pupil diameter dynamics as well as the scattering effect of the eyelid.
Mid-air perturbations reveal roll control in fruit flies
To study insect flight control we use the common fruit fly Drosophila melanogaster and a set of fast cameras that film the flies during free flight. We glue a tiny magnet to the back of each fly and use magnetic pulses to exert perturbation torques that rotate the flies in mid-air. By extracting the motion of the body and wings we aim to reverse-engineer the control-laws of the fly.
This movie shows a fruit fly recovers from a left-roll perturbation of 60 degrees. The magnetic perturbation pulse was one wing-beat long (5 ms, red line), and the fly started to respond within 5 ms from the pulse onset. The entire correction maneuver ended after 8 wing-beats.
Flies can handle extreme mechanical perturbations
Our method allows us to exert a wide range of perturbations and probe the flight envelope of the fly. This movie shows a perturbation where the fly was rolled over 8 times. During the perturbation the fly could not withstand the magnetic torque. but once the perturbation had ended, the fly went back in control within 4 wing beats. We haven't yet managed to make a fly dizzy.
Mimicry through motion:
ant-mimicry by a jumping spider
We discovered how the motion of an ant-mimicking spider makes it look like an ant to protect it from potential predators. Protective mimicry is a widespread phenomenon, in which a palatable species avoids predation by being mistaken for another, unpalatable species, called ‘model’. As such, protective mimicry is a striking example of adaptive evolution. Most studies on protective mimicry have focused on static traits, such as color and shape, rather than on dynamic traits like motion. Within terrestrial mimicry, mimicry of ants is among the most common, with spiders representing a large fraction of ant-mimicking species.
We used high-speed cameras and behavioral experiments to investigate the role of locomotor behavior in mimicry by the ant-mimicking jumping spider Myrmarachne formicaria. Contrary to previous suggestions, we found that mimics walk using all eight legs, raising their forelegs like ant antennae only for short stationary bouts. Strikingly, mimics exhibit winding trajectories that resemble the winding patterns of ants specifically engaged in pheromone-trail following, although mimics walked on chemically inert surfaces. Finally, we used behavioral experiments to show that a potential predator is, indeed, 'fooled' by the ant-like motion of the mimics.