GReTA - a novel Global and Recursive Tracking Algorithm in three dimensions

Tracking multiple moving targets allows quantitative measure of the dynamic behavior in systems as diverse as animal groups in biology, turbulence in fluid dynamics and crowd and traffic control. In three dimensions, tracking several targets becomes increasingly hard since optical occlusions are very likely, i.e. two featureless targets frequently overlap for several frames. Occlusions are particularly frequent in biological groups such as bird flocks, fish schools, and insect swarms, a fact that has severely limited collective animal behavior field studies in the past. This paper presents a 3D tracking method that is robust in the case of severe occlusions. To ensure robustness, we adopt a global optimization approach that works on all objects and frames at once. To achieve practicality and scalability, we employ a divide and conquer formulation, thanks to which the computational complexity of the problem is reduced by orders of magnitude. We tested our algorithm with synthetic data, with experimental data of bird flocks and insect swarms and with public benchmark datasets, and show that our system yields high quality trajectories for hundreds of moving targets with severe overlap. The results obtained on very heterogeneous data show the potential applicability of our method to the most diverse experimental situations.Comment: 13 pages, 6 figures, 3 tables. Version 3 was slightly shortened, and new comprative results on the public datasets (thermal infrared videos of flying bats) by Z. Wu and coworkers (2014) were included. in A. Attanasi et al., "GReTA - A Novel Global and Recursive Tracking Algorithm in Three Dimensions", IEEE Trans. Pattern Anal. Mach. Intell., vol.37 (2015

Simultaneous measurements of three-dimensional trajectories and wingbeat frequencies of birds in the field

This is the author accepted manuscript. The final version is available from the Royal Society via the DOI in this recordData accessibility: We provide data including images recorded by four cameras, camera parameters, videos showing the time variation of the bird 3D positions, and plain text files that include bird id number, positions, times, velocities, accelerations, and wingbeat frequencies at every time step. We also provide the Matlab codes that were used to: (i) detect birds on images; (ii) reconstruct birds' 3D locations using the new stereo-matching algorithm; (iii) track individual's 3D motions; and (iv) calculate wing motion and wingbeat frequency from tracking results. The code and data are available at: https://github.com/linghj/3DTracking.git and https://figshare.com/s/3c572f91b07b06ed30aa.Tracking the movements of birds in three dimensions is integral to a wide range of problems in animal ecology, behaviour and cognition. Multi-camera stereo-imaging has been used to track the three-dimensional (3D) motion of birds in dense flocks, but precise localization of birds remains a challenge due to imaging resolution in the depth direction and optical occlusion. This paper introduces a portable stereo-imaging system with improved accuracy and a simple stereo-matching algorithm that can resolve optical occlusion. This system allows us to decouple body and wing motion, and thus measure not only velocities and accelerations but also wingbeat frequencies along the 3D trajectories of birds. We demonstrate these new methods by analysing six flocking events consisting of 50 to 360 jackdaws (Corvus monedula) and rooks (Corvus frugilegus) as well as 32 jackdaws and 6 rooks flying in isolated pairs or alone. Our method allows us to (i) measure flight speed and wingbeat frequency in different flying modes; (ii) characterize the U-shaped flight performance curve of birds in the wild, showing that wingbeat frequency reaches its minimum at moderate flight speeds; (iii) examine group effects on individual flight performance, showing that birds have a higher wingbeat frequency when flying in a group than when flying alone and when flying in dense regions than when flying in sparse regions; and (iv) provide a potential avenue for automated discrimination of bird species. We argue that the experimental method developed in this paper opens new opportunities for understanding flight kinematics and collective behaviour in natural environments.Human Frontier Science Progra

Comparison of two 3D tracking paradigms for freely flying insects

In this paper, we discuss and compare state-of-the-art 3D tracking paradigms for flying insects such as Drosophila melanogaster. If two cameras are employed to estimate the trajectories of these identical appearing objects, calculating stereo and temporal correspondences leads to an NP-hard assignment problem. Currently, there are two different types of approaches discussed in the literature: probabilistic approaches and global correspondence selection approaches. Both have advantages and limitations in terms of accuracy and complexity. Here, we present algorithms for both paradigms. The probabilistic approach utilizes the Kalman filter for temporal tracking. The correspondence selection approach calculates the trajectories based on an overall cost function. Limitations of both approaches are addressed by integrating a third camera to verify consistency of the stereo pairings and to reduce the complexity of the global selection. Furthermore, a novel greedy optimization scheme is introduced for the correspondence selection approach. We compare both paradigms based on synthetic data with ground truth availability. Results show that the global selection is more accurate, while the previously proposed tracking-by-matching (probabilistic) approach is causal and feasible for longer tracking periods and very high target densities. We further demonstrate that our extended global selection scheme outperforms current correspondence selection approaches in tracking accuracy and tracking time

Multi-camera real-time three-dimensional tracking of multiple flying animals

Automated tracking of animal movement allows analyses that would not otherwise be possible by providing great quantities of data. The additional capability of tracking in real time—with minimal latency—opens up the experimental possibility of manipulating sensory feedback, thus allowing detailed explorations of the neural basis for control of behaviour. Here, we describe a system capable of tracking the three-dimensional position and body orientation of animals such as flies and birds. The system operates with less than 40 ms latency and can track multiple animals simultaneously. To achieve these results, a multi-target tracking algorithm was developed based on the extended Kalman filter and the nearest neighbour standard filter data association algorithm. In one implementation, an 11-camera system is capable of tracking three flies simultaneously at 60 frames per second using a gigabit network of nine standard Intel Pentium 4 and Core 2 Duo computers. This manuscript presents the rationale and details of the algorithms employed and shows three implementations of the system. An experiment was performed using the tracking system to measure the effect of visual contrast on the flight speed of Drosophila melanogaster. At low contrasts, speed is more variable and faster on average than at high contrasts. Thus, the system is already a useful tool to study the neurobiology and behaviour of freely flying animals. If combined with other techniques, such as ‘virtual reality’-type computer graphics or genetic manipulation, the tracking system would offer a powerful new way to investigate the biology of flying animals

Methods and measures for investigating microscale motility

Motility is an essential factor for an organism's survival and diversification. With the advent of novel single-cell technologies, analytical frameworks and theoretical methods, we can begin to probe the complex lives of microscopic motile organisms and answer the intertwining biological and physical questions of how these diverse lifeforms navigate their surroundings. Herein, we give an overview of different experimental, analytical, and mathematical methods used to study a suite of microscale motility mechanisms across different scales encompassing molecular-, individual- to population-level. We identify transferable techniques, pressing challenges, and future directions in the field. This review can serve as a starting point for researchers who are interested in exploring and quantifying the movements of organisms in the microscale world.Comment: 24 pages, 2 figure

Enzyme Powered Nanomotors Towards Biomedical Applications

[eng] The advancements in nanotechnology enabled the development of new diagnostic tools and drug delivery systems based on nanosystems, which offer unique features such as large surface area to volume ratio, cargo loading capabilities, increased circulation times, as well as versatility and multifunctionality. Despite this, the majority of nanomedicines do not translate into clinics, in part due to the biological barriers present in the body. Synthetic nano- and micromotors could be an alternative tool in nanomedicine, as the continuous propulsion force and potential to modulate the medium may aid tissue penetration and drug diffusion across biological barriers. Enzyme-powered motors are especially interesting for biomedical applications, owing to their biocompatibility and use of bioavailable substrates as fuel for propulsion. This thesis aims at exploring the potential applications of urease-powered nanomotors in nanomedicine. In the first work, we evaluated these motors as drug delivery systems. We found that active urease- powered nanomotors showed active motion in phosphate buffer solutions, and enhanced in vitro drug release profiles in comparison to passive nanoparticles. In addition, we observed that the motors were more efficient in delivering drug to cancer cells and caused higher toxicity levels, due to the combination of boosted drug release and local increase of pH produced by urea breakdown into ammonia and carbon dioxide. One of the major goals in nanomedicine is to achieve localized drug action, thus reducing side-effects. A commonly strategy to attain this is the use moieties to target specific diseases. In our second work, we assessed the ability of urease-powered nanomotors to improve the targeting and penetration of spheroids, using an antibody with therapeutic potential. We showed that the combination of active propulsion with targeting led to a significant increase in spheroid penetration, and that this effect caused a decrease in cell proliferation due to the antibody’s therapeutic action. Considering that high concentrations of nanomedicines are required to achieve therapeutic efficiency; in the third work we investigated the collective behavior of urease-powered nanomotors. Apart from optical microscopy, we evaluated the tracked the swarming behavior of the nanomotors using positron emission tomography, which is a technique widely used in clinics, due to its noninvasiveness and ability to provide quantitative information. We showed that the nanomotors were able to overcome hurdles while swimming in confined geometries. We observed that the nanomotors swarming behavior led to enhanced fluid convection and mixing both in vitro, and in vivo within mice’s bladders. Aiming at conferring protecting abilities to the enzyme-powered nanomotors, in the fourth work, we investigated the use of liposomes as chassis for nanomotors, encapsulating urease within their inner compartment. We demonstrated that the lipidic bilayer provides the enzymatic engines with protection from harsh acidic environments, and that the motility of liposome-based motors can be activated with bile salts. Altogether, these results demonstrate the potential of enzyme-powered nanomotors as nanomedicine tools, with versatile chassis, as well as capability to enhance drug delivery and tumor penetration. Moreover, their collective dynamics in vivo, tracked using medical imaging techniques, represent a step-forward in the journey towards clinical translation.[spa] Recientes avances en nanotecnología han permitido el desarrollo de nuevas herramientas para el diagnóstico de enfermedades y el transporte dirigido de fármacos, ofreciendo propiedades únicas como encapsulación de fármacos, el control sobre la biodistribución de estos, versatilidad y multifuncionalidad. A pesar de estos avances, la mayoría de nanomedicinas no consiguen llegar a aplicaciones médicas reales, lo cual es en parte debido a la presencia de barreras biológicas en el organismo que limitan su transporte hacia los tejidos de interés. En este sentido, el desarrollo de nuevos micro- y nanomotores sintéticos, capaces de autopropulsarse y causar cambios locales en el ambiente, podrían ofrecer una alternativa para la nanomedicina, promoviendo una mayor penetración en tejidos de interés y un mejor transporte de fármacos a través de las barreras biológicas. En concreto, los nanomotores enzimáticos poseen un alto potencial para aplicaciones biomédicas gracias a su biocompatibilidad y a la posibilidad de usar sustancias presentes en el organismo como combustible. Los trabajos presentados en esta tesis exploran el potenical de nanomotores, autopropulsados mediante la enzima ureasa, para aplicaciones biomédicas, y investigan su uso como vehículos para transporte de fármacos, su capacidad para mejorar penetración de tejidos diana, su versatilidad y movimiento colectivo. En conjunto, los resultados presentados en esta tesis doctoral demuestran el potencial del uso de nanomotores autopropulsados mediante enzimas como herramientas biomédicas, ofreciendo versatilidad en su diseño y una alta capacidad para promover el transporte de fármacos y la penetración en tumores. Por último, su movimiento colectivo observado in vivo mediante técnicas de imagen médicas representan un significativo avance en el viaje hacia su aplicación en medicina

Future state maximisation as an intrinsic motivation for decision making

The concept of an “intrinsic motivation" is used in the psychology literature to distinguish between behaviour which is motivated by the expectation of an immediate, quantifiable reward (“extrinsic motivation") and behaviour which arises because it is inherently useful, interesting or enjoyable. Examples of the latter can include curiosity driven behaviour such as exploration and the accumulation of knowledge, as well as developing skills that might not be immediately useful but that have the potential to be re-used in a variety of different future situations. In this thesis, we examine a candidate for an intrinsic motivation with wide-ranging applicability which we refer to as “future state maximisation". Loosely speaking this is the idea that, taking everything else to be equal, decisions should be made so as to maximally keep one's options open, or to give the maximal amount of control over what one can potentially do in the future. Our goal is to study how this principle can be applied in a quantitative manner, as well as identifying examples of systems where doing so could be useful in either explaining or generating behaviour. We consider a number of examples, however our primary application is to a model of collective motion in which we consider a group of agents equipped with simple visual sensors, moving around in two dimensions. In this model, agents aim to make decisions about how to move so as to maximise the amount of control they have over the potential visual states that they can access in the future. We find that with each agent following this simple, low-level motivational principle a swarm spontaneously emerges in which the agents exhibit rich collective behaviour, remaining cohesive and highly-aligned. Remarkably, the emergent swarm also shares a number of features which are observed in real flocks of starlings, including scale free correlations and marginal opacity. We go on to explore how the model can be developed to allow us to manipulate and control the swarm, as well as looking at heuristics which are able to mimic future state maximisation whilst requiring significantly less computation, and so which could plausibly operate under animal cognition

A Survey on Aerial Swarm Robotics

The use of aerial swarms to solve real-world problems has been increasing steadily, accompanied by falling prices and improving performance of communication, sensing, and processing hardware. The commoditization of hardware has reduced unit costs, thereby lowering the barriers to entry to the field of aerial swarm robotics. A key enabling technology for swarms is the family of algorithms that allow the individual members of the swarm to communicate and allocate tasks amongst themselves, plan their trajectories, and coordinate their flight in such a way that the overall objectives of the swarm are achieved efficiently. These algorithms, often organized in a hierarchical fashion, endow the swarm with autonomy at every level, and the role of a human operator can be reduced, in principle, to interactions at a higher level without direct intervention. This technology depends on the clever and innovative application of theoretical tools from control and estimation. This paper reviews the state of the art of these theoretical tools, specifically focusing on how they have been developed for, and applied to, aerial swarms. Aerial swarms differ from swarms of ground-based vehicles in two respects: they operate in a three-dimensional space and the dynamics of individual vehicles adds an extra layer of complexity. We review dynamic modeling and conditions for stability and controllability that are essential in order to achieve cooperative flight and distributed sensing. The main sections of this paper focus on major results covering trajectory generation, task allocation, adversarial control, distributed sensing, monitoring, and mapping. Wherever possible, we indicate how the physics and subsystem technologies of aerial robots are brought to bear on these individual areas