Manifold Representations for Continuous-State Reinforcement Learning

Reinforcement learning (RL) has shown itself to be an effective paradigm for solving optimal control problems with a finite number of states. Generalizing RL techniques to problems with a continuous state space has proven a difficult task. We present an approach to modeling the RL value function using a manifold representation. By explicitly modeling the topology of the value function domain, traditional problems with discontinuities and resolution can be addressed without resorting to complex function approximators. We describe how manifold techniques can be applied to value-function approximation, and present methods for constructing manifold representations in both batch and online settings. We present empirical results demonstrating the effectiveness of our approach

Scheduling Design with Unknown Execution Time Distributions or Modes

Open soft real-time systems, such as mobile robots, experience unpredictable interactions with their environments and yet must respond both adaptively and with reasonable temporal predictability. Because of the uncertainty inherent in such interactions, many of the assumptions of the real-time scheduling techniques traditionally used to ensure predictable timing of system actions do not hold in those environments. In previous work we have developed novel techniques for scheduling policy design where up-front knowledge of execution time distributions can be used to produce both compact representations of resource utilization state spaces and efficient optimal scheduling policies over those state spaces. This paper makes two main contributions beyond our previous work, to the state of the art in scheduling open soft real-time systems: (1) it shows how to relax the assumption that the entire distribution of execution times is known up front, to allow online learning of an execution time distribution during system run-time; and (2) it shows how to relax the assumption that the execution time of a system action can be characterized by a single distribution, to accommodate different execution time distributions for an action being taken in one of multiple modes. Each of these contributions allows a wider range of system actions to be scheduled adaptively and with temporal predictability, which increases the applicability of our approach to even more general classes of open soft real-time systems

Optimal Time Utility Based Scheduling Policy Design for Cyber-Physical Systems

Classical scheduling abstractions such as deadlines and priorities do not readily capture the complex timing semantics found in many real-time cyber-physical systems. Time utility functions provide a necessarily richer description of timing semantics, but designing utility-aware scheduling policies using them is an open research problem. In particular, optimal utility accrual scheduling design is needed for real-time cyber-physical domains. In this paper we design optimal utility accrual scheduling policies for cyber-physical systems with periodic, non-preemptable tasks that run with stochastic duration. These policies are derived by solving a Markov Decision Process formulation of the scheduling problem. We use this formulation to demonstrate that our technique improves on existing heuristic utility accrual scheduling policies

Scalable Scheduling Policy Design for Open Soft Real-Time Systems

Open soft real-time systems, such as mobile robots, must respond adaptively to varying operating conditions, while balancing the need to perform multiple mission specific tasks against the requirement that those tasks complete in a timely manner. Setting and enforcing a utilization target for shared resources is a key mechanism for achieving this behavior. However, because of the uncertainty and non-preemptability of some tasks, key assumptions of classical scheduling approaches do not hold. In previous work we presented foundational methods for generating task scheduling policies to enforce proportional resource utilization for open soft real-time systems with these properties. However, these methods scale exponentially in the number of tasks, limiting their practical applicability. In this paper, we present a novel parameterized scheduling policy that scales our technique to a much wider range of systems. These policies can represent geometric features of the scheduling policies produced by our earlier methods, but only require a number of parameters that is quadratic in the number of tasks. We provide empirical evidence that the best of these policies are competitive with exact solution methods in small problems, and significantly outperform heuristic methods in larger ones

Estimating the costs and perceived benefits of oral pre-exposure prophylaxis (PrEP) delivery in ten counties of Kenya: a costing and a contingent valuation study

BackgroundKenya included oral PrEP in the national guidelines as part of combination HIV prevention, and subsequently began providing PrEP to individuals who are at elevated risk of HIV infection in 2017. However, as scale-up continued, there was a recognized gap in knowledge on the cost of delivering oral PrEP. This gap limited the ability of the Government of Kenya to budget for its PrEP scale-up and to evaluate PrEP relative to other HIV prevention strategies. The following study calculated the actual costs of oral PrEP scale-up as it was being delivered in ten counties in Kenya. This costing also allowed for a comparison of various models of service delivery in different geographic regions from the perspective of service providers in Kenya. In addition, the analysis was also conducted to understand factors that indicate why some individuals place a greater value on PrEP than others, using a contingent valuation technique.MethodsData collection was completed between November 2017 and September 2018. Costing data was collected from 44 Kenyan health facilities, consisting of 23 public facilities, 5 private facilities and 16 drop-in centers (DICEs) through a cross-sectional survey in ten counties. Financial and programmatic data were collected from financial and asset records and through interviewer administered questionnaires. The costs associated with PrEP provision were calculated using an ingredients-based costing approach which involved identification and costing of all the economic inputs (both direct and indirect) used in PrEP service delivery. In addition, a contingent valuation study was conducted at the same 44 facilities to understand factors that reveal why some individuals place a greater value on PrEP than others. Interviews were conducted with 2,258 individuals (1,940 current PrEP clients and 318 non-PrEP clients). A contingent valuation method using a “payment card approach” was used to determine the maximum willingness to pay (WTP) of respondents regarding obtaining access to oral PrEP services.ResultsThe weighted cost of providing PrEP was $253 per person year, ranging from$217 at health centers to $283 at dispensaries. Drop-in centers (DICEs), which served about two-thirds of the client volume at surveyed facilities, had a unit cost of$276. The unit cost was highest for facilities targeting MSM ($355), while it was lowest for those targeting FSW ($248). The unit cost for facilities targeting AGYW was $323 per person year. The largest percentage of costs were attributable to personnel (58.5$2 per month (mean was $4.07 per month). This covers only one-third of the monthly cost of the medication (approximately$6 per month) and less than 10% of the full cost of delivering PrEP ($21 per month). A sizable proportion of current clients (27%) were unwilling to pay anything for PrEP. Certain populations put a higher value on PrEP services, including: FSW and MSM, Muslims, individuals with higher education, persons between the ages of 20 and 35, and households with a higher income and expenditures.DiscussionThis is the most recent and comprehensive study on the cost of PrEP delivery in Kenya. These results will be used in determining resource requirements and for resource mobilization to facilitate sustainable PrEP scale-up in Kenya and beyond. This contingent valuation study does have important implications for Kenya's PrEP program. First, it indicates that some populations are more motivated to adopt oral PrEP, as indicated by their higher WTP for the service. MSM and FSW, for example, placed a higher value on PrEP than AGYW. Higher educated individuals, in turn, put a much higher value on PrEP than those with less education (which may also reflect the higher “ability to pay” among those with more education). This suggests that any attempt to increase demand or improve PrEP continuation should consider these differences in client populations. Cost recovery from existing PrEP clients would have potentially negative consequences for uptake and continuation

Naomi: a new modelling tool for estimating HIV epidemic indicators at the district level in sub-Saharan Africa.

INTRODUCTION: HIV planning requires granular estimates for the number of people living with HIV (PLHIV), antiretroviral treatment (ART) coverage and unmet need, and new HIV infections by district, or equivalent subnational administrative level. We developed a Bayesian small-area estimation model, called Naomi, to estimate these quantities stratified by subnational administrative units, sex, and five-year age groups. METHODS: Small-area regressions for HIV prevalence, ART coverage and HIV incidence were jointly calibrated using subnational household survey data on all three indicators, routine antenatal service delivery data on HIV prevalence and ART coverage among pregnant women, and service delivery data on the number of PLHIV receiving ART. Incidence was modelled by district-level HIV prevalence and ART coverage. Model outputs of counts and rates for each indicator were aggregated to multiple geographic and demographic stratifications of interest. The model was estimated in an empirical Bayes framework, furnishing probabilistic uncertainty ranges for all output indicators. Example results were presented using data from Malawi during 2016-2018. RESULTS: Adult HIV prevalence in September 2018 ranged from 3.2% to 17.1% across Malawi's districts and was higher in southern districts and in metropolitan areas. ART coverage was more homogenous, ranging from 75% to 82%. The largest number of PLHIV was among ages 35 to 39 for both women and men, while the most untreated PLHIV were among ages 25 to 29 for women and 30 to 34 for men. Relative uncertainty was larger for the untreated PLHIV than the number on ART or total PLHIV. Among clients receiving ART at facilities in Lilongwe city, an estimated 71% (95% CI, 61% to 79%) resided in Lilongwe city, 20% (14% to 27%) in Lilongwe district outside the metropolis, and 9% (6% to 12%) in neighbouring Dowa district. Thirty-eight percent (26% to 50%) of Lilongwe rural residents and 39% (27% to 50%) of Dowa residents received treatment at facilities in Lilongwe city. CONCLUSIONS: The Naomi model synthesizes multiple subnational data sources to furnish estimates of key indicators for HIV programme planning, resource allocation, and target setting. Further model development to meet evolving HIV policy priorities and programme need should be accompanied by continued strengthening and understanding of routine health system data

Modeling the epidemiological impact of the UNAIDS 2025 targets to end AIDS as a public health threat by 2030

BACKGROUND: UNAIDS has established new program targets for 2025 to achieve the goal of eliminating AIDS as a public health threat by 2030. This study reports on efforts to use mathematical models to estimate the impact of achieving those targets. METHODS AND FINDINGS: We simulated the impact of achieving the targets at country level using the Goals model, a mathematical simulation model of HIV epidemic dynamics that includes the impact of prevention and treatment interventions. For 77 high-burden countries, we fit the model to surveillance and survey data for 1970 to 2020 and then projected the impact of achieving the targets for the period 2019 to 2030. Results from these 77 countries were extrapolated to produce estimates for 96 others. Goals model results were checked by comparing against projections done with the Optima HIV model and the AIDS Epidemic Model (AEM) for selected countries. We included estimates of the impact of societal enablers (access to justice and law reform, stigma and discrimination elimination, and gender equality) and the impact of Coronavirus Disease 2019 (COVID-19). Results show that achieving the 2025 targets would reduce new annual infections by 83% (71% to 86% across regions) and AIDS-related deaths by 78% (67% to 81% across regions) by 2025 compared to 2010. Lack of progress on societal enablers could endanger these achievements and result in as many as 2.6 million (44%) cumulative additional new HIV infections and 440,000 (54%) more AIDS-related deaths between 2020 and 2030 compared to full achievement of all targets. COVID-19-related disruptions could increase new HIV infections and AIDS-related deaths by 10% in the next 2 years, but targets could still be achieved by 2025. Study limitations include the reliance on self-reports for most data on behaviors, the use of intervention effect sizes from published studies that may overstate intervention impacts outside of controlled study settings, and the use of proxy countries to estimate the impact in countries with fewer than 4,000 annual HIV infections. CONCLUSIONS: The new targets for 2025 build on the progress made since 2010 and represent ambitious short-term goals. Achieving these targets would bring us close to the goals of reducing new HIV infections and AIDS-related deaths by 90% between 2010 and 2030. By 2025, global new infections and AIDS deaths would drop to 4.4 and 3.9 per 100,000 population, and the number of people living with HIV (PLHIV) would be declining. There would be 32 million people on treatment, and they would need continuing support for their lifetime. Incidence for the total global population would be below 0.15% everywhere. The number of PLHIV would start declining by 2023

Constraint Modeling and Reformulation in the Context of Academic Task Assignment

We discuss the modeling and reformulation of a resource allocation problem, the assignment of graduate teaching assistants (GTAs) to courses. Our research contributes the following: • Formulation of the GTA assignment problem as a nonbinary CSP. • Design of a new convention for consistency checking to deal with over-constrained problem. • Definition of a new network-decomposable nonbinary confinement constraint. • Evaluation of the reformulation of confinement and equality constraints on 3 real-world data sets