Competitive function approximation for reinforcement learning

The application of reinforcement learning to problems with continuous domains requires representing the value function by means of function approximation. We identify two aspects of reinforcement learning that make the function approximation process hard: non-stationarity of the target function and biased sampling. Non-stationarity is the result of the bootstrapping nature of dynamic programming where the value function is estimated using its current approximation. Biased sampling occurs when some regions of the state space are visited too often, causing a reiterated updating with similar values which fade out the occasional updates of infrequently sampled regions. We propose a competitive approach for function approximation where many different local approximators are available at a given input and the one with expectedly best approximation is selected by means of a relevance function. The local nature of the approximators allows their fast adaptation to non-stationary changes and mitigates the biased sampling problem. The coexistence of multiple approximators updated and tried in parallel permits obtaining a good estimation much faster than would be possible with a single approximator. Experiments in different benchmark problems show that the competitive strategy provides a faster and more stable learning than non-competitive approaches.Preprin

Tiling solutions for optimal biological sensing

Biological systems, from cells to organisms, must respond to the ever changing environment in order to survive and function. This is not a simple task given the often random nature of the signals they receive, as well as the intrinsically stochastic, many body and often self-organized nature of the processes that control their sensing and response and limited resources. Despite a wide range of scales and functions that can be observed in the living world, some common principles that govern the behavior of biological systems emerge. Here I review two examples of very different biological problems: information transmission in gene regulatory networks and diversity of adaptive immune receptor repertoires that protect us from pathogens. I discuss the trade-offs that physical laws impose on these systems and show that the optimal designs of both immune repertoires and gene regulatory networks display similar discrete tiling structures. These solutions rely on locally non-overlapping placements of the responding elements (genes and receptors) that, overall, cover space nearly uniformly.Comment: 11 page

Evaluation of rate law approximations in bottom-up kinetic models of metabolism.

BackgroundThe mechanistic description of enzyme kinetics in a dynamic model of metabolism requires specifying the numerical values of a large number of kinetic parameters. The parameterization challenge is often addressed through the use of simplifying approximations to form reaction rate laws with reduced numbers of parameters. Whether such simplified models can reproduce dynamic characteristics of the full system is an important question.ResultsIn this work, we compared the local transient response properties of dynamic models constructed using rate laws with varying levels of approximation. These approximate rate laws were: 1) a Michaelis-Menten rate law with measured enzyme parameters, 2) a Michaelis-Menten rate law with approximated parameters, using the convenience kinetics convention, 3) a thermodynamic rate law resulting from a metabolite saturation assumption, and 4) a pure chemical reaction mass action rate law that removes the role of the enzyme from the reaction kinetics. We utilized in vivo data for the human red blood cell to compare the effect of rate law choices against the backdrop of physiological flux and concentration differences. We found that the Michaelis-Menten rate law with measured enzyme parameters yields an excellent approximation of the full system dynamics, while other assumptions cause greater discrepancies in system dynamic behavior. However, iteratively replacing mechanistic rate laws with approximations resulted in a model that retains a high correlation with the true model behavior. Investigating this consistency, we determined that the order of magnitude differences among fluxes and concentrations in the network were greatly influential on the network dynamics. We further identified reaction features such as thermodynamic reversibility, high substrate concentration, and lack of allosteric regulation, which make certain reactions more suitable for rate law approximations.ConclusionsOverall, our work generally supports the use of approximate rate laws when building large scale kinetic models, due to the key role that physiologically meaningful flux and concentration ranges play in determining network dynamics. However, we also showed that detailed mechanistic models show a clear benefit in prediction accuracy when data is available. The work here should help to provide guidance to future kinetic modeling efforts on the choice of rate law and parameterization approaches

Pro-poor intervention strategies in irrigated agriculture in Asia: poverty in irrigated agriculture: issues and options: India

Irrigated farming / Poverty / Institutions / Irrigation programs / Performance evaluation / Irrigation management / Water distribution / Water rates / Cost recovery / India

Dwarf galaxy formation with H2-regulated star formation

We describe cosmological galaxy formation simulations with the adaptive mesh refinement code Enzo that incorporate a star formation prescription regulated by the local abundance of molecular hydrogen. We show that this H2-regulated prescription leads to a suppression of star formation in low mass halos (M_h < ~10^10 M_sun) at z>4, alleviating some of the dwarf galaxy problems faced by theoretical galaxy formation models. H2 regulation modifies the efficiency of star formation of cold gas directly, rather than indirectly reducing the cold gas content with "supernova feedback". We determine the local H2 abundance in our most refined grid cells (76 proper parsec in size at z=4) by applying the model of Krumholz, McKee, & Tumlinson, which is based on idealized 1D radiative transfer calculations of H2 formation-dissociation balance in ~100 pc atomic--molecular complexes. Our H2-regulated simulations are able to reproduce the empirical (albeit lower z) Kennicutt-Schmidt relation, including the low Sigma_gas cutoff due to the transition from atomic to molecular phase and the metallicity dependence thereof, without the use of an explicit density threshold in our star formation prescription. We compare the evolution of the luminosity function, stellar mass density, and star formation rate density from our simulations to recent observational determinations of the same at z=4-8 and find reasonable agreement between the two.Comment: replaced with version published in Ap

Data management study, volume 5. Appendix A - Contractor data package technical description and system engineering /SE/ Final report

Technical description and systems engineering contractor data package for Voyager spacecraf

Pension fund investment and regulation - a macro study

Pension fund assets have been accumulated rapidly during the past decades, and it is evident that this trend will continue. An immediate problem arising from the rapid accumulation of such a large volume of assets across countries is how to invest them. Pension funds differ from other institutional investors, e.g. mutual funds, in that their investment horizons are relatively long, typically 30-40 years. In addition, they are pooled assets to support people’s retirement lives. The authorities have a policy concern about their investment performance, because otherwise, the shortfalls will have to be met by the nation state (Clark and Hu 2005a). In this paper, we seek to address this issue from the macro perspective. By using a unique dataset covering 39 countries (17 EMEs and 22 OECD) and based on the classic mean-variance optimisation approach, first we find a negative impact of international portfolio investment restrictions on pension fund returns and risk, and this issue is particularly serious for EMEs. Following a shift from the QAR to the PPR, the average risk is expected to fall by 27% for EMEs pension funds, while the figure is 10% for OECD pension funds. Second, there is evidence that if higher portfolio returns are wanted, higher proportion should be invested in equities and foreign assets. Third, our results show that pension funds should value the diversification benefit arising from property investment (Booth 2002)

Media And Government Relations In Papua New Guinea

All is not well with news media in Papua New Guinea. Media and government relations are stressed, a situation adverse to the country's development. Media organisations have to deal with operational difficulties, threats against editorial freedom, and harassment or physical danger experienced by journalists. Yet there are positive factors providing hope for the future, especially that key element, freedom to publish, which goes together with a habit of openess in public life as part of the national culture. That is the main finding of a study made during a working visit to Papua New Guinea