Value Function Discovery in Markov Decision Processes with Evolutionary Algorithms

In this paper we introduce a novel method for discovery of value functions for Markov Decision Processes (MDPs). This method, which we call Value Function Discovery (VFD), is based on ideas from the Evolutionary Algorithm field. VFD’s key feature is that it discovers descriptions of value functions that are algebraic in nature. This feature is unique, because the descriptions include the model parameters of the MDP. The algebraic expression of the value function discovered by VFD can be used in several scenarios, e.g., conversion to a policy (with one-step policy improvement) or control of systems with time-varying parameters. The work in this paper is a first step towards exploring potential usage scenarios of discovered value functions. We give a detailed description of VFD and illustrate its application on an example MDP. For this MDP we let VFD discover an algebraic description of a value function that closely resembles the optimal value function. The discovered value function is then used to obtain a policy, which we compare numerically to the optimal policy of the MDP. The resulting policy shows near-optimal performance on a wide range of model parameters. Finally, we identify and discuss future application scenarios of discovered value functions

Throughput modeling of the IEEE MAC for sensor networks

In this paper we provide a model for analyzing the saturation throughput of the ieee 802.15.4 mac protocol, which is the de-facto standard for wireless sensor networks, ensuring fair access to the channel. To this end, we introduce the concept of a natural layer, which reflects the time that a sensor node typically has to wait prior to sending a packet. The model is simple and provides new insight how the throughput depends on the protocol parameters and the number of nodes in the network. Validation experiments with simulations demonstrate that the model is highly accurate for a wide range of parameter settings of the mac protocol, and applicable to both large and small networks. As a byproduct, we discuss fundamental differences in the protocol stack and corresponding throughput models of the popular 802.11 standard

Multiple technology approach based on stable isotope ratio analysis, Fourier transform infrared spectrometry and thermogravimetric analysis to ensure the fungal origin of the chitosan

Chitosan is a natural polysaccharide which has been authorized for oenological practices for the treatment of musts and wines. This authorization is limited to chitosan of fungal origin while that of crustacean origin is prohibited. To guarantee its origin, a method based on the measurement of the stable isotope ratios (SIR) of carbon δ13C, nitrogen δ15N, oxygen δ18O and hydrogen δ2H of chitosan has been recently proposed without indicating the threshold authenticity limits of these parameters which, for the first time, were estimated in this paper. In addition, on part of the samples analysed through SIR, Fourier transform infrared spectrometry (FTIR) and thermogravimetric analysis (TGA) were performed as simple and rapid discrimination methods due to limited technological resources. Samples having δ13C values above -14.2‱ and below -125.1‱ can be considered as authentic fungal chitosan without needing to analyse other parameters. If the δ13C value falls between -25.1‱ and -24.9‱, it is necessary to proceed further with the evaluation of the parameter δ15N, which must be above +2.7‱. Samples having δ18O values lower than +25.3‱ can be considered as authentic fungal chitosan. The combination of maximum degradation temperatures (obtained using TGA) and peak areas of Amide I and NH2/Amide II (obtained using FTIR) also allows the discrimination between the two origins of the polysaccharide. Hierarchical cluster analysis (HCA) and principal component analysis (PCA) based on TGA, FTIR and SIR data successfully distributed the tested samples into informative clusters. Therefore, we present the technologies described as part of a robust analytical strategy for the correct identification of chitosan samples from crustaceans or fung

Degradation of Film and Rigid Bioplastics During the Thermophilic Phase and the Maturation Phase of Simulated Composting

AbstractThe recent regulations, which impose limits on single use plastics and packaging, are encouraging the development of bioplastics market. Some bioplastics are labelled as compostable with the organic waste according to a specific certification (EN 13432), however the conditions of industrial composting plants are generally less favourable than the standard test conditions. Aiming at studying the effective degradation of marketable bioplastic products under composting, the current research stresses novel elements which can strongly influence bioplastics degradation: the simulation of industrial composting conditions and the thickness of bioplastic products, ranging between 50 and 500 µm. The research approaches these critical aspects simulating a composting test of 20 days of thermophilic phase followed by 40 days of maturation phase, on starch-based polymer Mater-Bi® (MB), polybutylene adipate terephthalate (PBAT), polylactic acid (PLA) of different thickness. Conventional low density polyethylene (LDPE) was introduced as negative control. An overall study with Fourier Transform InfraRed (FTIR), ThermoGravimetric Analysis (TGA), Gel Permeation Chromatography (GPC), Scanning Electron Microscope (SEM) and visual inspections was applied. Results highlighted that MB film presented the highest degradation rate, 45 ± 4.7% in terms of weight loss. Both MB and PBAT were subjected to physico-chemical features change, while LDPE presented slight degradation signs. The most critical observations have been done for PLA, which is strongly influenced both by thickness and thermophilic phase duration, shorter than the EN 13432 conditions

Heritability estimates for 361 blood metabolites across 40 genome-wide association studies

Metabolomics examines the small molecules involved in cellular metabolism. Approximately 50% of total phenotypic differences in metabolite levels is due to genetic variance, but heritability estimates differ across metabolite classes. We perform a review of all genome-wide association and (exome-) sequencing studies published between November 2008 and October 2018, and identify >800 class-specific metabolite loci associated with metabolite levels. In a twin-family cohort (N = 5117), these metabolite loci are leveraged to simultaneously estimate total heritability (h2 total), and the proportion of heritability captured by known metabolite loci (h2 Metabolite-hits) for 309 lipids and

Throughput modeling of the IEEE MAC for sensor networks

In this paper we provide a model for analyzing the saturation throughput of the ieee 802.15.4 mac protocol, which is the de-facto standard for wireless sensor networks, ensuring fair access to the channel. To this end, we introduce the concept of a natural layer, which reflects the time that a sensor node typically has to wait prior to sending a packet. The model is simple and provides new insight how the throughput depends on the protocol parameters and the number of nodes in the network. Validation experiments with simulations demonstrate that the model is highly accurate for a wide range of parameter settings of the mac protocol, and applicable to both large and small networks. As a byproduct, we discuss fundamental differences in the protocol stack and corresponding throughput models of the popular 802.11 standard

Value Function Discovery in Markov Decision Processes with Evolutionary Algorithms

In this paper we introduce a novel method for discovery of value functions for Markov Decision Processes (MDPs). This method, which we call Value Function Discovery (VFD), is based on ideas from the Evolutionary Algorithm field. VFD’s key feature is that it discovers descriptions of value functions that are algebraic in nature. This feature is unique, because the descriptions include the model parameters of the MDP. The algebraic expression of the value function discovered by VFD can be used in several scenarios, e.g., conversion to a policy (with one-step policy improvement) or control of systems with time-varying parameters. The work in this paper is a first step towards exploring potential usage scenarios of discovered value functions. We give a detailed description of VFD and illustrate its application on an example MDP. For this MDP we let VFD discover an algebraic description of a value function that closely resembles the optimal value function. The discovered value function is then used to obtain a policy, which we compare numerically to the optimal policy of the MDP. The resulting policy shows near-optimal performance on a wide range of model parameters. Finally, we identify and discuss future application scenarios of discovered value functions