7,964 research outputs found
Recommended from our members
Safe Reinforcement Learning
This dissertation proposes and presents solutions to two new problems that fall within the broad scope of reinforcement learning (RL) research. The first problem, high confidence off-policy evaluation (HCOPE), requires an algorithm to use historical data from one or more behavior policies to compute a high confidence lower bound on the performance of an evaluation policy. This allows us to, for the first time, provide the user of any RL algorithm with confidence that a newly proposed policy (which has never actually been used) will perform well.
The second problem is to construct what we call a safe reinforcement learning algorithm---an algorithm that searches for new and improved policies, while ensuring that the probability that a bad policy is proposed is low. Importantly, the user of the RL algorithm may tune the meaning of bad (in terms of a desired performance baseline) and how low the probability of a bad policy being deployed should be, in order to capture the level of risk that is acceptable for the application at hand.
We show empirically that our solutions to these two critical problems require surprisingly little data, making them practical for real problems. While our methods allow us to, for the first time, produce convincing statistical guarantees about the performance of a policy without requiring its execution, the primary contribution of this dissertation is not the methods that we propose. The primary contribution of this dissertation is a compelling argument that these two problems, HCOPE and safe reinforcement learning, which at first may seem out of reach, are actually tractable. We hope that this will inspire researchers to propose their own methods, which improve upon our own, and that the development of increasingly data-efficient safe reinforcement learning algorithms will catalyze the widespread adoption of reinforcement learning algorithms for suitable real-world problems
Analysis of Nitrogen Loading Reductions for Wastewater Treatment Facilities and Non-Point Sources in the Great Bay Estuary Watershed
In 2009, the New Hampshire Department of Environmental Services (DES) published a proposal for numeric nutrient criteria for the Great Bay Estuary. The report found that total nitrogen concentrations in most of the estuary needed to be less than 0.3 mg N/L to prevent loss of eelgrass habitat and less than 0.45 mg N/L to prevent occurrences of low dissolved oxygen. Based on these criteria and an analysis of a compilation of data from at least seven different sources, DES concluded that 11 of the 18 subestuaries in the Great Bay Estuary were impaired for nitrogen. Under the Clean Water Act, if a water body is determined to be impaired, a study must be completed to determine the existing loads of the pollutant and the load reductions that would be needed to meet the water quality standard. Therefore, DES developed models to determine existing nitrogen loads and nitrogen loading thresholds for the subestuaries to comply with the numeric nutrient criteria. DES also evaluated the effects of different permitting scenarios for wastewater treatment facilities on nitrogen loads and the costs for wastewater treatment facility upgrades. This modeling exercise showed that: Nitrogen loads to the Great Bay, Little Bay, and the Upper Piscataqua River need to be reduced by 30 to 45 percent to attain the numeric nutrient criteria. Both wastewater treatment facilities and non-point sources will need to reduce nitrogen loads to attain the numeric nutrient criteria. The percent reduction targets for nitrogen loads only change minimally between wet and dry years. Wastewater treatment facility upgrades to remove nitrogen will be costly; however, the average cost per pound of nitrogen removed from the estuary due to wastewater facility upgrades is lower than for non-point source controls. The permitting options for some wastewater treatment facilities will be limited by requirements to not increase pollutant loads to impaired waterbodies. The numeric nutrient criteria and models used by DES are sufficiently accurate for calculating nitrogen loading thresholds for the Great Bay watershed. Additional monitoring and modeling is needed to better characterize conditions and nitrogen loading thresholds for the Lower Piscataqua River. This nitrogen loading analysis for Great Bay may provide a framework for setting nitrogen permit limits for wastewater treatment facilities and developing watershed implementation plans to reduce nitrogen loads
- …