Low-reynolds-number locomotion via reinforcement learning

This dissertation summarizes computational results from applying reinforcement learning and deep neural network to the designs of artificial microswimmers in the inertialess regime, where the viscous dissipation in the surrounding fluid environment dominates and the swimmer’s inertia is completely negligible. In particular, works in this dissertation consist of four interrelated studies of the design of microswimmers for different tasks: (1) a one-dimensional microswimmer in free-space that moves towards the target via translation, (2) a one-dimensional microswimmer in a periodic domain that rotates to reach the target, (3) a two-dimensional microswimmer that switches gaits to navigate to the designated targets in a plane, and (4) a two-dimensional microswimmer trained to navigate in a non-stationary environment. The first and second studies focus on how reinforcement learning (specifically model-free, off-policy Q-learning) can be applied to generate one-dimensional translation (part 1) or net rotation (part 2) in low Reynolds number fluids. Through the interaction with the surrounding viscous fluid, the swimmer learns to break the time-reversal symmetry of Stokes flow in order to achieve the maximum displacement (reward) either in free-space or in a periodic domain. In the third part of the dissertation, a deep reinforcement learning approach (proximal policy optimization) is utilized to train a two-dimensional swimmer to develop complex strategies such as run-and-tumble to navigate through environments and move towards specific targets. Proximal policy optimization contains actor-critic model, the critic estimates the value function, the actor updates the policy distribution in the direction suggested by the critic. Results show the artificial trained swimmer can develop effective policy (gaits) such as translation and rotation, and the swimmer can move to specific targets by combining these gaits in an intelligent way. The simulation results also show that without being explicitly programmed, the trained swimmer is able to perform target navigation even under flow perturbation. Finally, in the last part of the dissertation, a generalized step-up reinforcement method with deep learning is developed for an environment that changes in time. In this work, the traditional reinforcement learning is combined with a high confidence context detection, allowing the swimmer to be trained to navigate amphibious non-stationary environments that consist of two distinct regions. Computational results show that the swimmer trained by this algorithm adapts to the environments faster, while developing more effective locomotory strategies in both environments, than traditional reinforcement learning approaches. Furthermore, the effective policies with traditional strategies are compared and analyzed. This work illustrates how deep reinforcement learning method can be conveniently adapted to a broader class of problems such as a microswimmer in a non-stationary environment. Results from this part highlight a powerful alternative to current traditional methods for applications in unpredictable, complex fluid environments and open a route towards future designs of “smart” microswimmers with trainable artificial intelligence

Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning

Training task-completion dialogue agents with reinforcement learning usually requires a large number of real user experiences. The Dyna-Q algorithm extends Q-learning by integrating a world model, and thus can effectively boost training efficiency using simulated experiences generated by the world model. The effectiveness of Dyna-Q, however, depends on the quality of the world model - or implicitly, the pre-specified ratio of real vs. simulated experiences used for Q-learning. To this end, we extend the recently proposed Deep Dyna-Q (DDQ) framework by integrating a switcher that automatically determines whether to use a real or simulated experience for Q-learning. Furthermore, we explore the use of active learning for improving sample efficiency, by encouraging the world model to generate simulated experiences in the state-action space where the agent has not (fully) explored. Our results show that by combining switcher and active learning, the new framework named as Switch-based Active Deep Dyna-Q (Switch-DDQ), leads to significant improvement over DDQ and Q-learning baselines in both simulation and human evaluations.Comment: 8 pages, 9 figures, AAAI 201

Modeling the emissions of nitrous oxide (N₂O) and methane (CH₄) from the terrestrial biosphere to the atmosphere

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Earth, Atmospheric, and Planetary Sciences, 1996.Includes bibliographical references (p. 211-219).by Yuexin Liu.Ph.D

Fast HEVC Intramode Decision Based on Hybrid Cost Ranking

To improve rate-distortion (R-D) performance, high efficiency video coding (HEVC) increases the intraprediction modes with heavy computational load, and thus the intracoding optimization is highly demanded for real-time applications. According to the conditional probabilities of most probable modes and the correlation of potential candidate subsets, this paper proposes a fast HEVC intramode decision scheme based on the hybrid cost ranking which includes both Hadamard cost and rate-distortion cost. The proposed scheme utilizes the coded results of the modified rough mode decision and the neighboring prediction units so as to obtain a potential candidate subset and then conditionally selects the optimal mode through early likelihood decision and hybrid cost ranking. By the experiment-driven methodology, the proposed scheme implements the early termination if the best mode from the candidate subset is equal to one or two neighboring intramodes. The experimental results demonstrate that the proposed scheme averagely provides about 23.7% encoding speedup with just 0.82% BD-rate loss in comparison with default fast intramode decision in HM16.0. Compared to other fast intramode decision schemes, the proposed scheme also significantly reduces intracoding time while maintaining similar R-D performance for the all-intraconfiguration in HM16.0 Main profile

Pupillary response to moving stimuli of different speeds

Purpose: To investigate the pupillary response to moving stimuli of different speeds and the influence of different luminance environments. Methods: Twenty-eight participants with normal or corrected-to-normal vision were included. The participants were required to track moving optotypes horizontally, and their pupils were videoed with an infrared camera. Stimuli of different speeds were presented in different luminance environments. Results: Experiment 1 demonstrated that the motion stimuli induced pupil dilation in a speed-dependent pattern. The pupil dilation increased as the speed increased, and the pupil dilation gradually increased, then reached saturation. Experiment 2 showed that a stimulus targeting the rod- or cone-mediated pathway could induce pupil dilation in a similar speed-dependent pattern. The absolute but not relative pupil dilation in the cone paradigm was significantly larger than that in the rod paradigm. As the speed increased, the pupil dilation in the cone paradigm reached saturation at speed slower than the rod paradigm. Conclusions: Motion stimuli induced pupil dilation in a speed-dependent pattern, and as the motion speed increased, the pupil dilation gradually increased and reached saturation. And the speed required to reach saturation in the cone paradigm was slower than in the rod paradigm

Implementation of a radial disk ionization profile in the relxill_nk model

Very steep reflection emissivity profiles in the inner part of accretion disks are commonly found in the analysis of X-ray observations of black hole binaries and AGN, but there is some debate about their exact origin. While steep reflection emissivity profiles can be naturally produced by compact coronae close to black holes, the measured radial emissivity parameter can be further increased by the radial disk ionization profile when the theoretical model assumes a disk with constant ionization. In this paper, we implement the possibility of a radial disk ionization profile in the reflection model RELXILL_NK, which is a package designed to calculate reflection spectra of "deformed" Kerr black holes. We analyze a NuSTAR observation of the black hole binary EXO 1846-031, which was previously found to have a very high inner emissivity index. We find that the model with a radial disk ionization profile improves the fit, but the impact on the estimate of the black hole spin parameter and on the constraint of the deformation parameter is modest. However, we show that the analysis of future observations of Athena and eXTP will necessarily require models with a radial disk ionization profile to have accurate constraints of the deformation parameters.Comment: 9 pages, 6 figures. v2: refereed versio

Integrated Analysis of Gene Expression and Tumor Nuclear Image Profiles Associated with Chemotherapy Response in Serous Ovarian Carcinoma

Small sample sizes used in previous studies result in a lack of overlap between the reported gene signatures for prediction of chemotherapy response. Although morphologic features, especially tumor nuclear morphology, are important for cancer grading, little research has been reported on quantitatively correlating cellular morphology with chemotherapy response, especially in a large data set. In this study, we have used a large population of patients to identify molecular and morphologic signatures associated with chemotherapy response in serous ovarian carcinoma.A gene expression model that predicts response to chemotherapy is developed and validated using a large-scale data set consisting of 493 samples from The Cancer Genome Atlas (TCGA) and 244 samples from an Australian report. An identified 227-gene signature achieves an overall predictive accuracy of greater than 85% with a sensitivity of approximately 95% and specificity of approximately 70%. The gene signature significantly distinguishes between patients with unfavorable versus favorable prognosis, when applied to either an independent data set (P = 0.04) or an external validation set (P<0.0001). In parallel, we present the production of a tumor nuclear image profile generated from 253 sample slides by characterizing patients with nuclear features (such as size, elongation, and roundness) in incremental bins, and we identify a morphologic signature that demonstrates a strong association with chemotherapy response in serous ovarian carcinoma.A gene signature discovered on a large data set provides robustness in accurately predicting chemotherapy response in serous ovarian carcinoma. The combination of the molecular and morphologic signatures yields a new understanding of potential mechanisms involved in drug resistance

Extreme Two-View Geometry From Object Poses with Diffusion Models

Human has an incredible ability to effortlessly perceive the viewpoint difference between two images containing the same object, even when the viewpoint change is astonishingly vast with no co-visible regions in the images. This remarkable skill, however, has proven to be a challenge for existing camera pose estimation methods, which often fail when faced with large viewpoint differences due to the lack of overlapping local features for matching. In this paper, we aim to effectively harness the power of object priors to accurately determine two-view geometry in the face of extreme viewpoint changes. In our method, we first mathematically transform the relative camera pose estimation problem to an object pose estimation problem. Then, to estimate the object pose, we utilize the object priors learned from a diffusion model Zero123 to synthesize novel-view images of the object. The novel-view images are matched to determine the object pose and thus the two-view camera pose. In experiments, our method has demonstrated extraordinary robustness and resilience to large viewpoint changes, consistently estimating two-view poses with exceptional generalization ability across both synthetic and real-world datasets. Code will be available at https://github.com/scy639/Extreme-Two-View-Geometry-From-Object-Poses-with-Diffusion-Models