Learning Search Strategies from Human Demonstrations

Decision making and planning with partial state information is a problem faced by all forms of intelligent entities. The formulation of a problem under partial state information leads to an exorbitant set of choices with associated probabilistic outcomes making its resolution difficult when using traditional planning methods. Human beings have acquired the ability of acting under uncertainty through education and self-learning. Transferring our know-how to artificial agents and robots will make it faster for them to learn and even improve upon us in tasks in which incomplete knowledge is available, which is the objective of this thesis. We model how humans reason with respect to their beliefs and transfer this knowledge in the form of a parameterised policy, following a Programming by Demonstration framework, to a robot apprentice for two spatial navigation tasks: the first task consists of localising a wooden block on a table and for the second task a power socket must be found and connected. In both tasks the human teacher and robot apprentice only rely on haptic and tactile information. We model the human and robot's beliefs by a probability density function which we update through recursive Bayesian state space estimation. To model the reasoning processes of human subjects performing the search tasks we learn a generative joint distribution over beliefs and actions (end-effector velocities) which were recorded during the executions of the task. For the first search task the direct mapping from belief to actions is learned whilst for the second task we incorporate a cost function used to adapt the policy parameters in a Reinforcement Learning framework and show a considerable improvement over solely learning the behaviour with respect to the distance taken to accomplish the task. Both search tasks above can be considered as active localisation as the uncertainty originates only from the position of the agent in the world. We consider searches in which both the position of the robot and features of the environment are uncertain. Given the unstructured nature of the belief a histogram parametrisation of the joint distribution of the robots position and features is necessary. However, naively doing so becomes quickly intractable as the space and time complexity is exponential. We demonstrate that by only parametrising the marginals and by memorising the parameters of the measurement likelihood functions we can recover the exact same solution as the naive parametrisations at a cost which is linear in space and time complexity

Learning search behaviour from humans

A frequent method for taking into account the partially observable nature of an environment in which robots interact lies in formulating the problem domain as a Partially Observable Markov Decision Process (POMDP). By having humans demonstrate how to act in this partially observable context we can leverage their prior knowledge, experience and intuition, which is difficult to encode directly in a controller, to solve a task formulated as a POMDP. In this work we learn search behaviours from human demonstrators and transfer this knowledge to a robot in a context where no visual information is available. The task consists of finding a block on a table. This is a non-trivial problem since no visual information is available and as a result, the belief of the demonstrator’s state (position in the environment) has to be inferred. We show that by representing the belief of the human’s position in the environment by a particle filter (PF) and learning a mapping from this belief to their end-effector velocities with a Gaussian Mixture Model (GMM), we model the human’s search process. We compare the different types of search behaviour demonstrated by the humans to that of our learned model, to validate that the search process has been successfully modelled. We then contrast the performance of this human-inspired search model to a greedy controller and show that (similarly to humans) the learned controller minimises uncertainty, hence demonstrating more robustness in the face of false belief

Trials

BACKGROUND: Postoperative upper gastrointestinal fistula (PUGIF) is a devastating complication, leading to high mortality (reaching up to 80%), increased length of hospital stay, reduced health-related quality of life and increased health costs. Nutritional support is a key component of therapy in such cases, which is related to the high prevalence of malnutrition. In the prophylactic setting, enteral nutrition (EN) is associated with a shorter hospital stay, a lower incidence of severe infectious complications, lower severity of complications and decreased cost compared to total parenteral nutrition (TPN) following major upper gastrointestinal (GI) surgery. There is little evidence available for the curative setting after fistula occurrence. We hypothesize that EN increases the 30-day fistula closure rate in PUGIF, allowing better health-related quality of life without increasing the morbidity or mortality. METHODS/DESIGN: The NUTRILEAK trial is a multicenter, randomized, parallel-group, open-label phase III trial to assess the efficacy of EN (the experimental group) compared with TPN (the control group) in patients with PUGIF. The primary objective of the study is to compare EN versus TPN in the treatment of PUGIF (after esophagogastric resection including bariatric surgery, duodenojejunal resection or pancreatic resection with digestive tract violation) in terms of the 30-day fistula closure rate. Secondary objectives are to evaluate the 6-month postrandomization fistula closure rate, time of first fistula closure (in days), the medical- and surgical treatment-related complication rate at 6 months after randomization, the fistula-related complication rate at 6 months after randomization, the type and severity of early (30 days after randomization) and late fistula-related complications (over 30 days after randomization), 30-day and 6-month postrandomization mortality rate, nutritional status at day 30, day 60, day 90 and day 180 postrandomization, the mean length of hospital stay, the patient's health-related quality of life (by self-assessment questionnaire), oral feeding time and direct costs of treatment. A total of 321 patients will be enrolled. DISCUSSION: The two nutritional supports are already used in daily practice, but most surgeons are reluctant to use the enteral route in case of PUGIF. This study will be the first randomized trial testing the role of EN versus TPN in PUGIF. TRIAL REGISTRATION: ClinicalTrials.gov: NCT03742752. Registered on 14 November 2018.This research program is funded by the French Ministry of Health through Programme Hospitalier de Recherche Clinique 2016

Learning search polices from humans in a partially observable context

Decision making and planning for which the state information is only partially available is a problem faced by all forms of intelligent entities they being either virtual, synthetic or biological. The standard approach to mathematically solve such a decisional problem is to formulate it as a partially observable decision process (POMDP) and apply the same optimisation techniques used in the Markov decision process (MDP). However, applying naively the same methodology to solve MDPs as with POMDPs makes the problem computationally intractable. To address this problem, we take a programming by demonstration approach to provide a solution to the POMDP in continuous state and action space. In this work, we model the decision making process followed by humans when searching blindly for an object on a table. We show that by representing the belief of the human’s position in the environment by a particle filter (PF) and learning a mapping from this belief to their end effector velocities with a Gaussian mixture model (GMM), we can model the human’s search process and reproduce it for any agent. We further categorize the type of behaviours demonstrated by humans as being either risk-prone or risk-averse and find that more than 70% of the human searches were considered to be risk-averse. We contrast the performance of this human-inspired search model with respect to greedy and coastal navigation search methods. Our evaluation metric is the distance taken to reach the goal and how each method minimises the uncertainty. We further analyse the control policy of the coastal navigation and GMM search models and argue that taking into account uncertainty is more efficient with respect to distance travelled to reach the goal

Petunia, Your Next Supermodel?

Plant biology in general, and plant evo-devo in particular would strongly benefit from a broader range of available model systems. In recent years, technological advances have facilitated the analysis and comparison of individual gene functions in multiple species, representing now a fairly wide taxonomic range of the plant kingdom. Because genes are embedded in gene networks, studying evolution of gene function ultimately should be put in the context of studying the evolution of entire gene networks, since changes in the function of a single gene will normally go together with further changes in its network environment. For this reason, plant comparative biology/evo-devo will require the availability of a defined set of 'super' models occupying key taxonomic positions, in which performing gene functional analysis and testing genetic interactions ideally is as straightforward as, e.g., in Arabidopsis. Here we review why petunia has the potential to become one of these future supermodels, as a representative of the Asterid Glade. We will first detail its intrinsic qualities as a model system. Next, we highlight how the revolution in sequencing technologies will now finally allows exploitation of the petunia system to its full potential, despite that petunia has already a long history as a model in plant molecular biology and genetics. We conclude with a series of arguments in favor of a more diversified multi-model approach in plant biology, and we point out where the petunia model system may further play a role, based on its biological features and molecular toolkit

Regulation of a maize HD-ZIP IV transcription factor by a non-conventional RDR2-dependent small RNA.

BAP GEAPSIInternational audienceSmall non-coding RNAs are versatile riboregulators that control gene expression at the transcriptional or post-transcriptional level, governing many facets of plant development. Here we present evidence for the existence of a 24 nt small RNA (named small1) that is complementary to the 3' UTR of OCL1 (Outer Cell Layer1), the founding member of the maize HD-ZIP IV gene family encoding plant-specific transcription factors that are mainly involved in epidermis differentiation and specialization. The biogenesis of small1 depends on DICER-like 3 (DCL3), RNA-dependent RNA polymerase 2 (RDR2) and RNA polymerase IV, components that are usually required for RNA-dependent DNA-methylation. Unexpectedly, GFP sensor experiments in transient and stable transformation systems revealed that small1 may regulate its target at the post-transcriptional level, mainly through translational repression. This translational repression is attenuated in an rdr2 mutant background in which small1 does not accumulate. Our experiments further showed the possible involvement of a secondary stem-loop structure present in the 3' UTR of OCL1 for efficient target repression, suggesting the existence of several regulatory mechanisms affecting OCL1 mRNA stability and translation

Petunia, Your Next Supermodel?

International audiencePlant biology in general, and plant evo-devo in particular would strongly benefit from a broader range of available model systems. In recent years, technological advances have facilitated the analysis and comparison of individual gene functions in multiple species, representing now a fairly wide taxonomic range of the plant kingdom. Because genes are embedded in gene networks, studying evolution of gene function ultimately should be put in the context of studying the evolution of entire gene networks, since changes in the function of a single gene will normally go together with further changes in its network environment. For this reason, plant comparative biology/evo-devo will require the availability of a defined set of 'super' models occupying key taxonomic positions, in which performing gene functional analysis and testing genetic interactions ideally is as straightforward as, e.g., in Arabidopsis. Here we review why petunia has the potential to become one of these future supermodels, as a representative of the Asterid Glade. We will first detail its intrinsic qualities as a model system. Next, we highlight how the revolution in sequencing technologies will now finally allows exploitation of the petunia system to its full potential, despite that petunia has already a long history as a model in plant molecular biology and genetics. We conclude with a series of arguments in favor of a more diversified multi-model approach in plant biology, and we point out where the petunia model system may further play a role, based on its biological features and molecular toolkit