Flow Shape Design for Microfluidic Devices Using Deep Reinforcement Learning

Microfluidic devices are utilized to control and direct flow behavior in a wide variety of applications, particularly in medical diagnostics. A particularly popular form of microfluidics -- called inertial microfluidic flow sculpting -- involves placing a sequence of pillars to controllably deform an initial flow field into a desired one. Inertial flow sculpting can be formally defined as an inverse problem, where one identifies a sequence of pillars (chosen, with replacement, from a finite set of pillars, each of which produce a specific transformation) whose composite transformation results in a user-defined desired transformation. Endemic to most such problems in engineering, inverse problems are usually quite computationally intractable, with most traditional approaches based on search and optimization strategies. In this paper, we pose this inverse problem as a Reinforcement Learning (RL) problem. We train a DoubleDQN agent to learn from this environment. The results suggest that learning is possible using a DoubleDQN model with the success frequency reaching 90% in 200,000 episodes and the rewards converging. While most of the results are obtained by fixing a particular target flow shape to simplify the learning problem, we later demonstrate how to transfer the learning of an agent based on one target shape to another, i.e. from one design to another and thus be useful for a generic design of a flow shape.Comment: Neurips 2018 Deep RL worksho

Perching with fixed wings

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2008.Includes bibliographical references (leaves 43-46).Human pilots have the extraordinary ability to remotely maneuver small Unmanned Aerial Vehicles (UAVs) far outside the flight envelope of conventional autopilots. Given the tremendous thrust-to-weight ratio available on these small machines [1, 2], linear control approaches have recently produced impressive demonstrations that come close to matching this agility for a certain class of aerobatic maneuvers where the rotor or propeller forces dominate the dynamics of the aircraft [3, 4, 5]. However, as our flying machines scale down to smaller sizes (e.g. Micro Aerial Vehicles) operating at low Reynold's numbers, viscous forces dominate propeller thrust [6, 7, 8], causing classical control (and design) techniques to fail. These new technologies will require a different approach to control, where the control system will need to reason about the long term and time dependent effects of the unsteady fluid dynamics on the response of the vehicle. Perching is representative of a large class of control problems for aerobatics that requires and agile and robust control system with the capability of planning well into the future. Our experimental paradigm along with the simplicity of the problem structure has allowed us to study the problem at the most fundamental level. This thesis presents methods and results for identifying an aerodynamic model of a small glider at very high angles-of-attack using tools from supervised machine learning and system identification. Our model then serves as a benchmark platform for studying control of perching using an optimal control framework, namely reinforcement learning. Our results indicate that a compact parameterization of the control is sufficient to successfully execute the task in simulation.by Rick E. Cory.S.M

Active Control of Flow over Rotating Cylinder by Multiple Jets using Deep Reinforcement Learning

The real power of artificial intelligence appears in reinforcement learning, which is computationally and physically more sophisticated due to its dynamic nature. Rotation and injection have been a proven way of active flow control to reduce the drag force exerted on blunt bodies. Rotation will be added to the cylinder alongside the deep reinforcement learning (DRL) algorithm, which uses multiple controlled jets to reach maximum possible drag suppression. Characteristics of the DRL code, including controlling parameters, their limitations, and optimization of the DRL network for use with rotation will be presented. This work will focus on optimizing the number and positions of the jets, sensors location, and maximum allowed flow rate to jets in the form of maximum allowed flow rate of each actuation and the total number of them per episode. It is found that combining the rotation with the DRL tools is promising, since it suppresses the vortex shedding, stabilizes the Karman vortex street, and reduces the drag coefficient by up to 49.75%. Also, it will be showed that having more sensors at more locations is not always a good choice and the sensor number and location should be determined based on the need of the user and corresponding configuration. Also, allowing the agent to have access to higher flow rates, mostly reduces the performance, except when the cylinder rotates. In all cases, the agent can keep the lift coefficient at a value near zero, or stabilize it at a smaller number.Comment: arXiv admin note: text overlap with arXiv:1808.07664 by other author

Controlling Rayleigh-B\'enard convection via Reinforcement Learning

Thermal convection is ubiquitous in nature as well as in many industrial applications. The identification of effective control strategies to, e.g., suppress or enhance the convective heat exchange under fixed external thermal gradients is an outstanding fundamental and technological issue. In this work, we explore a novel approach, based on a state-of-the-art Reinforcement Learning (RL) algorithm, which is capable of significantly reducing the heat transport in a two-dimensional Rayleigh-B\'enard system by applying small temperature fluctuations to the lower boundary of the system. By using numerical simulations, we show that our RL-based control is able to stabilize the conductive regime and bring the onset of convection up to a Rayleigh number $Ra_c \approx 3 \cdot 10^4$, whereas in the uncontrolled case it holds $Ra_{c}=1708$. Additionally, for $Ra > 3 \cdot 10^4$, our approach outperforms other state-of-the-art control algorithms reducing the heat flux by a factor of about $2.5$. In the last part of the manuscript, we address theoretical limits connected to controlling an unstable and chaotic dynamics as the one considered here. We show that controllability is hindered by observability and/or capabilities of actuating actions, which can be quantified in terms of characteristic time delays. When these delays become comparable with the Lyapunov time of the system, control becomes impossible.Comment: 24 pages, 10 figure

An End-to-End Platform for Autonomous Dynamic Soaring in Wind Shear

Despite advancements in our understanding of flight in modern times, birds remain unmatched when it comes to maneuverability and energy efficiency in flight; in particular seabirds like the albatross are known to travel vast distances without stopping for food by performing an aerobatic maneuver called dynamic soaring. When the maneuver is executed in the presence of a wind field that varies in strength of direction, the albatross extracts kinetic energy from the field. In this dissertation, we present an end-to-end system designed to exploit wind as the albatross does. The system we designed consists of a gliding platform outfitted with sensors and computational hardware, an on-board software platform that enables autonomy, and a ground platform for monitoring mission performance and issuing commands.We contribute the design of an airframe, the Fox, capable of performing dynamic soaring at low altitudes (~400m above sea level). We validate the airframe against expected stressors (vibration, coefficient of lift, temperature, and communication signal strength), and show in simulation it can complete a dynamic soaring orbit in wind shear that varies in maximum wind speed from 8 to 12 m/s. We show that this airframe can reach speeds exceeding 40 m/s while soaring.We fit the airframe with a commercial off-the-shelf autopilot, as well as a custom on-board-computing (OBC) solution to provide the necessary facilities to enable autonomy. The OBC generates dynamic soaring trajectories that fit a wind-field map that is built as the aircraft is deployed and controls the Fox to follow them by sending commands to the autopilot using a sample-based controller scheme. This process is monitored by human operators on the ground via a portable ground station that is linked to the Fox via a radio antenna. Field tests are presented that validate real-world controller performance against simulated results.Finally, we present a learning controller that learns from and out-performs the sample-based controller in simulation. While not field tested, we believe a self-optimizing controller of this form is necessary to enable autonomy of a soaring aircraft subject to extended mission durations.While dynamic soaring field tests were not pursued in this work, we hope this dissertation will be a blueprint for future researchers to finally achieve autonomous soaring

Machine Learning for Fluid Mechanics

The field of fluid mechanics is rapidly advancing, driven by unprecedented volumes of data from field measurements, experiments and large-scale simulations at multiple spatiotemporal scales. Machine learning offers a wealth of techniques to extract information from data that could be translated into knowledge about the underlying fluid mechanics. Moreover, machine learning algorithms can augment domain knowledge and automate tasks related to flow control and optimization. This article presents an overview of past history, current developments, and emerging opportunities of machine learning for fluid mechanics. It outlines fundamental machine learning methodologies and discusses their uses for understanding, modeling, optimizing, and controlling fluid flows. The strengths and limitations of these methods are addressed from the perspective of scientific inquiry that considers data as an inherent part of modeling, experimentation, and simulation. Machine learning provides a powerful information processing framework that can enrich, and possibly even transform, current lines of fluid mechanics research and industrial applications.Comment: To appear in the Annual Reviews of Fluid Mechanics, 202