Off-Policy Reinforcement Learning with Loss Function Weighted by Temporal Difference Error

Training agents via off-policy deep reinforcement learning (RL) requires a large memory, named replay memory, that stores past experiences used for learning. These experiences are sampled, uniformly or non-uniformly, to create the batches used for training. When calculating the loss function, off-policy algorithms assume that all samples are of the same importance. In this paper, we hypothesize that training can be enhanced by assigning different importance for each experience based on their temporal-difference (TD) error directly in the training objective. We propose a novel method that introduces a weighting factor for each experience when calculating the loss function at the learning stage. In addition to improving convergence speed when used with uniform sampling, the method can be combined with prioritization methods for non-uniform sampling. Combining the proposed method with prioritization methods improves sampling efficiency while increasing the performance of TD-based off-policy RL algorithms. The effectiveness of the proposed method is demonstrated by experiments in six environments of the OpenAI Gym suite. The experimental results demonstrate that the proposed method achieves a 33%~76% reduction of convergence speed in three environments and an 11% increase in returns and a 3%~10% increase in success rate for other three environments.Comment: to be submitted to an AI conferenc

Three Essays on the Efficiency of Medical Providers

Physicians use colonoscopy to detect recurrent colorectal cancer for patients who have had colorectal cancer surgery. Clinical studies show both underuse and overuse of this test among survivors of colorectal cancer in relation to guideline recommendations. Yet few studies have examined referrals and test findings. To identify determinants of colonoscopy referral for colorectal cancer survivors and ex-post test outcome, Chapter 1 examines Texas cancer registry data linked with Medicare claims from 2000 to 2009 for colorectal cancer survivors with a history of resection surgery. Risk-adjusted regression analyses are used to measure the association of patient, referring physician and clinical factors with referrals and test results. Intestinal symptoms, the timing of referrals, and referring physician specialty are associated with referral decisions and test results. Gastroenterologists are more likely to refer patients for colonoscopy than oncologists, surgeons or primary care physicians, but their rates of positive test results are the lowest. The discrepancy between referral decisions and test results suggests suboptimal test use. Chapter 2 studies referral patterns for colonoscopies and applies cost-benefit analysis to examine whether this test has been overused or underused among patients with a cancer history. A key aspect of the analysis is that the ex post value of colonoscopy is partially observable in insurance claims records based on whether the test identifies recurrent cancer. Estimating the physician-specific parameter representing physicians' practice styles, I find that referral patterns exhibit physician-level heterogeneity. The percentage of physicians who overuse colonoscopy varies with physician specialty: A significant number of gastroenterologists overuse colonoscopy, whereas a much lower portion of oncologists and primary care doctors overuse it. These findings illustrate the need for well-targeted health care policies to curb growing health care costs. Chapter 3 measures the relationship between hospital volume and patient mortality for six cancer operations (colectomy, esophagectomy, pancreatic resection, pneumonectomy, pulmonary lobectomy, and rectal resection). Analyzing hospital discharge data from Florida, New Jersey, and New York for the 12 years 2000 to 2011, we find that the statistical significance of hospital volume depends critically on the regression model used: for the data, logistic and random-effects models suggest that higher volume is associated with lower mortality, but fixed-effects models do not. Model-specification tests support either the fixed-effects or random-effects model, depending on the surgical procedure; the basic logistic model is always rejected. These findings illustrate the importance of testing alternative model specifications, especially when drawing policy conclusions about promoting high-volume facilities

Standard Deviation of Fiber-Coupling Efficiency for Free-Space Optical Communication Through Atmospheric Turbulence

In fiber-based optical receivers such as coherent receiver, the optical signal reaching the receiver's aperture is coupled to the optical fiber through a set of lenses. Due to atmospheric turbulence, however, the coupling efficiency, which is defined by the ratio of the fiber-coupled optical power to the optical power reaching the aperture, fluctuates over time, and thus should be treated as a random variable. Previous theoretical works predict the average coupling efficiency accurately, but fail to estimate the standard deviation, especially when the turbulence is strong. In this study, we propose new theoretical formulas for the standard deviation of coupling efficiency applicable to a wide range of turbulence conditions. For this purpose, we derive a new approximated formula for the second moment of coupling efficiency using the second-order Taylor expansion. Next, a new formula for the second moment of coupled optical power is developed by introducing the spatial filter to the fourth-order coherence function. The computer simulation is carried out to evaluate the accuracy of the formulas. The results show that our formulas agree with the simulation results over a wide range of turbulence strength. Also, the new formulas provide better accuracy than the previously reported ones

Health Reform Monitoring Survey-Texas, Issue Brief #10: Marketplace Plans: Premiums, Network Size and Market Competition

Since late 2013, individuals have been able to shop for and purchase private health insurance through the Marketplace via Healthcare.gov. Through the Marketplace, Texans can learn what plans are available to them, examine the features of the plans, determine whether they qualify for tax credits to lower their premiums, and purchase the plan that is best suited to their needs. During the first year of the Affordable Care Act’s Health Insurance Marketplace, consumers learned that many plans offered limited provider networks. This issue brief examines the availability of Marketplace plans in Texas, the number of in-network hospitals in the plans, and other plan characteristics influencing premiums. Our analysis focuses on Marketplace Silver plans, because they provide the benchmark by which premium-reducing tax credits are determined and because the majority of Texans who purchased Marketplace plans chose a Silver plan

Temperature-directed fluorescent switchable nanoparticles based on P3OT-PNIPAM nanogel composite

Poly(N-isopropylacrylamide) (PNIPAM) is a unique stimuli-responsive material that exhibits a lower critical solution temperature (LCST). Owing to this characteristic temperature-dependent behavior, PNIPAM has found extensive utilization as an active material in various applications, including sensors, drug delivery, and cellular imaging. Herein, we demonstrate temperature-directed fluorescent switchable nanoparticles based on poly(3octylthiophene-2,5-diyl) (P3OT) nanoaggregate-embedded PNIPAM nanogel composites (POPNs) featuring different crosslinker contents. The amount of P3OT loading in the nanogel composites can be gradually controlled by varying the crosslinking density of the PNIPAM matrix; this may be attributable to the efficient entrapment of P3OT nanoaggregates in case of a dense polymeric network with the increase in crosslinking density. POPNs exhibit dramatic temperature-dependent fluorescence enhancement (by a factor of 2.11). This is based on the environmental changes affecting fluorescent P3OT chains at temperatures below and above the LCST of the PNIPAM matrix. Based on this temperature-directed fluorescent switching capability, POPN could find potential applications in various fields, including biomedical imaging and sensors

Fabrication of Hydrophilic PDMS Surface using PVA Electrospun Nanofiber

2

Accelerated Disease Progression after Discontinuation of Sorafenib in a Patient with Metastatic Papillary Thyroid Cancer

Distant metastases from papillary thyroid carcinoma (PTC) are rare and are associated with a poor prognosis. Here, we describe a patient with metastatic PTC who was treated with a tyrosine kinase inhibitor (TKI, sorafenib) for several months that was acutely exacerbated by discontinuation. A 43-year-old male was diagnosed with PTC in February 2004 and underwent total thyroidectomy followed by two courses of high-dose radioactive iodine (RAI) therapy. Despite two additional courses of high-dose RAI therapy, lung and muscle metastases were developed. Treatment with sorafenib was begun in September 2010. After 11 months treatment of sorafenib, newly developed metastatic lesions were found in mediastinal lymph nodes, liver, and bones. Considered as treatment failure, the administration of sorafenib was discontinued. Two weeks after sorafenib treatment was stopped, the disease progressed abruptly and caused death of the patient by respiratory failure. In our patient, PTC progressed rapidly after the cessation of sorafenib treatment. Patients with several other types of cancer have also experienced such rapid disease progression, termed "flare-ups." Physicians should be aware that flare-ups may occur in advanced PTC patients following the cessation of TKI therapy

3D Antidrying Antifreezing Artificial Skin Device with Self-Healing and Touch Sensing Capability

Hydrogels are attractive, active materials for various e-skin devices based on their unique functionalities such as flexibility and biocompatibility. Still, e-skin devices are generally limited to simple structures, and the realization of optimal-shaped 3D e-skin devices for target applications is an intriguing issue of interest. Furthermore, hydrogels intrinsically suffer from drying and freezing issues in operational capability for practical applications. Herein, 3D artificial skin devices are demonstrated with highly improved device stability. The devices are fabricated in a target-oriented 3D structure by extrusion-based 3D printing, spontaneously heal mechanical damage, and enable stable device operation over time and under freezing conditions. Based on the material design to improve drying and freezing resistance, an organohydrogel, prepared by solvent displacement of hydrogel with ethylene glycol for 3 h, exhibits excellent drying resistance over 1000 h and improved freezing resistance by showing no phase transition down to -60 degrees C while maintaining its self-healing functionality. Based on the improved drying and freezing resistance, artificial skin devices in target-oriented optimal 3D structures are presented, which enable accurate positioning of touchpoints even on a complicated 3D structure stably over time and excellent operation at temperatures below 0 degrees C without losing their flexibility

Material Design for 3D Multifunctional Hydrogel Structure Preparation

Hydrogels are recognized as one of the most promising materials for e-skin devices because of their unique applicable functionalities such as flexibility, stretchability, biocompatibility, and conductivity. Beyond the excellent sensing functionalities, the e-skin devices further need to secure a target-oriented 3D structure to be applied onto various body parts having complex 3D shapes. However, most e-skin devices are still fabricated in simple 2D film-type devices, and it is an intriguing issue to fabricate complex 3D e-skin devices resembling target body parts via 3D printing. Here, a material design guideline is provided to prepare multifunctional hydrogels and their target-oriented 3D structures based on extrusion-based 3D printing. The material design parameters to realize target-oriented 3D structures via 3D printing are systematically derived from the correlation between material design of hydrogels and their gelation characteristics, rheological properties, and 3D printing processability for extrusion-based 3D printing. Based on the suggested material design window, ion conductive self-healable hydrogels are designed and successfully applied to extrusion-based 3D printing to realize various 3D shapes