Multi-step Reinforcement Learning: A Unifying Algorithm

Unifying seemingly disparate algorithmic ideas to produce better performing algorithms has been a longstanding goal in reinforcement learning. As a primary example, TD($\lambda$) elegantly unifies one-step TD prediction with Monte Carlo methods through the use of eligibility traces and the trace-decay parameter $\lambda$. Currently, there are a multitude of algorithms that can be used to perform TD control, including Sarsa, $Q$-learning, and Expected Sarsa. These methods are often studied in the one-step case, but they can be extended across multiple time steps to achieve better performance. Each of these algorithms is seemingly distinct, and no one dominates the others for all problems. In this paper, we study a new multi-step action-value algorithm called $Q(\sigma)$ which unifies and generalizes these existing algorithms, while subsuming them as special cases. A new parameter, $\sigma$, is introduced to allow the degree of sampling performed by the algorithm at each step during its backup to be continuously varied, with Sarsa existing at one extreme (full sampling), and Expected Sarsa existing at the other (pure expectation). $Q(\sigma)$ is generally applicable to both on- and off-policy learning, but in this work we focus on experiments in the on-policy case. Our results show that an intermediate value of $\sigma$, which results in a mixture of the existing algorithms, performs better than either extreme. The mixture can also be varied dynamically which can result in even greater performance.Comment: Appeared at the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18

Asteroid Distributions in the Ecliptic

We present analysis of the asteroid surface density distribution of main belt asteroids (mean perihelion $\Delta \simeq 2.404$ AU) in five ecliptic latitude fields, -17 \gtsimeq \beta(\degr) \ltsimeq +15, derived from deep \textit{Large Binocular Telescope} (LBT) $V-$band (85% completeness limit $V = 21.3$ mag) and \textit{Spitzer Space Telescope} IRAC 8.0 \micron (80% completeness limit $\sim 103 \mu$Jy) fields enabling us to probe the 0.5--1.0 km diameter asteroid population. We discovered 58 new asteroids in the optical survey as well as 41 new bodies in the \textit{Spitzer} fields. The derived power law slopes of the number of asteroids per square degree are similar within each $\sim 5$\degr{} ecliptic latitude bin with a mean value of $-0.111 \pm 0.077$. For the 23 known asteroids detected in all four IRAC channels mean albedos range from $0.24 \pm 0.07$ to $0.10 \pm 0.05$. No low albedo asteroids ($p_{V}$ \ltsimeq 0.1) were detected in the \textit{Spitzer} FLS fields, whereas in the SWIRE fields they are frequent. The SWIRE data clearly samples asteroids in the middle and outer belts providing the first estimates of these km-sized asteroids' albedos. Our observed asteroid number densities at optical wavelengths are generally consistent with those derived from the Standard Asteroid Model within the ecliptic plane. However, we find an over density at \beta \gtsimeq 5\degr{} in our optical fields, while the infrared number densities are under dense by factors of 2 to 3 at all ecliptic latitudes.Comment: 35 pages including 5 figures, accepted to The Astronomical Journa

Confiabilidade de questionário sobre uso de drogas por escolares, Brasil

OBJECTIVE: To analyze reliability of a self-applied questionnaire on substance use and misuse among adolescent students. METHODS: Two cross-sectional studies were carried out for the instrument test-retest. The sample comprised male and female students aged 1119 years from public and private schools (elementary, middle, and high school students) in the city of Salvador, Northeastern Brazil, in 2006. A total of 591 questionnaires were applied in the test and 467 in the retest. Descriptive statistics, the Kappa index, Cronbach's alpha and intraclass correlation were estimated. RESULTS: The prevalence of substance use/misuse was similar in both test and retest. Sociodemographic variables showed a "moderate" to "almost perfect" agreement for the Kappa index, and a "satisfactory" (>;0.75) consistency for Cronbach's alpha and intraclass correlation. The age which psychoactive substances (tobacco, alcohol, and cannabis) were first used and chronological age were similar in both studies. Test-retest reliability was found to be a good indicator of students' age of initiation and their patterns of substance use. CONCLUSIONS: The questionnaire reliability was found to be satisfactory in the population studied.OBJETIVO: Analizar la confiabilidad de un cuestionario auto aplicable sobre el uso y abuso de sustancias entre adolescentes escolares. MÉTODOS: Se realizaron dos estudios transversales para la prueba y re-prueba del cuestionario en muestra representativa de alumnos de ambos sexos, de 11 a 19 años, de escuelas públicas y privadas (del curso primaria y secundaria) de Salvador, Noreste de Brasil, en 2006. Se aplicaron 591 cuestionarios en la primera aplicación y 467 en la segunda. La estadística descriptiva, el índice Kappa, alfa de Cronbach y correlación intraclase fueron calculados. RESULTADOS: La prevalencia del uso/abuso de las sustancias fue semejante en ambas evaluaciones. Para las variables sociodemográficas el índice kappa indicó concordancia "moderada" a "casi perfecta"y el análisis del alfa de Cronbach y correlación intraclase indicó consistencia "satisfactoria". La edad de los estudiantes fue semejante en las dos evaluaciones. La edad de inicio del consumo y patrones de uso fueron considerados indicadores confiables. CONCLUSIONES: La confiabilidad del cuestionario fue satisfactoria para la población estudiada.OBJETIVO: Analisar a confiabilidade de um questionário auto-aplicável sobre o uso e abuso de substâncias entre adolescentes escolares. MÉTODOS: Foram realizados dois estudos transversais para teste e re-teste do questionário em amostra representativa de alunos de ambos os sexos, de 11 a 19 anos, de escolas públicas e privadas (do curso fundamental e médio) de Salvador, BA, em 2006. Foram aplicados 591 questionários na primeira aplicação e 467 na segunda. Foram calculados a estatística descritiva, o índice kappa, alfa de Cronbach e correlação intraclasse. RESULTADOS: A prevalência do uso/abuso das substâncias foi semelhante em ambas as avaliações. Para as variáveis sociodemográficas o índice kappa indicou concordância "moderada" a "quase perfeita" e a análise do alfa de Cronbach e correlação intraclasse indicaram consistência "satisfatória". A idade de experimentação das substâncias psicoativas (tabaco, álcool e maconha) e idade dos estudantes foram semelhantes nas duas avaliações. A idade de iniciação do consumo e padrões de uso foram considerados indicadores confiáveis. CONCLUSÕES: A confiabilidade do questionário foi satisfatória para a população estudada

SDSS-III: Massive Spectroscopic Surveys of the Distant Universe, the Milky Way Galaxy, and Extra-Solar Planetary Systems

Building on the legacy of the Sloan Digital Sky Survey (SDSS-I and II), SDSS-III is a program of four spectroscopic surveys on three scientific themes: dark energy and cosmological parameters, the history and structure of the Milky Way, and the population of giant planets around other stars. In keeping with SDSS tradition, SDSS-III will provide regular public releases of all its data, beginning with SDSS DR8 (which occurred in Jan 2011). This paper presents an overview of the four SDSS-III surveys. BOSS will measure redshifts of 1.5 million massive galaxies and Lya forest spectra of 150,000 quasars, using the BAO feature of large scale structure to obtain percent-level determinations of the distance scale and Hubble expansion rate at z<0.7 and at z~2.5. SEGUE-2, which is now completed, measured medium-resolution (R=1800) optical spectra of 118,000 stars in a variety of target categories, probing chemical evolution, stellar kinematics and substructure, and the mass profile of the dark matter halo from the solar neighborhood to distances of 100 kpc. APOGEE will obtain high-resolution (R~30,000), high signal-to-noise (S/N>100 per resolution element), H-band (1.51-1.70 micron) spectra of 10^5 evolved, late-type stars, measuring separate abundances for ~15 elements per star and creating the first high-precision spectroscopic survey of all Galactic stellar populations (bulge, bar, disks, halo) with a uniform set of stellar tracers and spectral diagnostics. MARVELS will monitor radial velocities of more than 8000 FGK stars with the sensitivity and cadence (10-40 m/s, ~24 visits per star) needed to detect giant planets with periods up to two years, providing an unprecedented data set for understanding the formation and dynamical evolution of giant planet systems. (Abridged)Comment: Revised to version published in The Astronomical Journa

The Fourteenth Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the Extended Baryon Oscillation Spectroscopic Survey and from the Second Phase of the Apache Point Observatory Galactic Evolution Experiment

The fourth generation of the Sloan Digital Sky Survey (SDSS-IV) has been in operation since 2014 July. This paper describes the second data release from this phase, and the 14th from SDSS overall (making this Data Release Fourteen or DR14). This release makes the data taken by SDSS-IV in its first two years of operation (2014–2016 July) public. Like all previous SDSS releases, DR14 is cumulative, including the most recent reductions and calibrations of all data taken by SDSS since the first phase began operations in 2000. New in DR14 is the first public release of data from the extended Baryon Oscillation Spectroscopic Survey; the first data from the second phase of the Apache Point Observatory (APO) Galactic Evolution Experiment (APOGEE-2), including stellar parameter estimates from an innovative data-driven machine-learning algorithm known as "The Cannon"; and almost twice as many data cubes from the Mapping Nearby Galaxies at APO (MaNGA) survey as were in the previous release (N = 2812 in total). This paper describes the location and format of the publicly available data from the SDSS-IV surveys. We provide references to the important technical papers describing how these data have been taken (both targeting and observation details) and processed for scientific use. The SDSS web site (www.sdss.org) has been updated for this release and provides links to data downloads, as well as tutorials and examples of data use. SDSS-IV is planning to continue to collect astronomical data until 2020 and will be followed by SDSS-V

Brachial Artery FMD Responses to Steady-State Moderate-Intensity and High-Intensity Interval Exercise in Mid-Spectrum Chronic Kidney Disease

Brachial artery flow-mediated dilation (FMD) is a nitric oxide-dependent measure of conduit artery endothelial function that is transiently potentiated by moderate-intensity steady state exercise (SSE) in healthy adults. Whether exercise imparts similar effects in adults with Stage 3 or 4 chronic kidney disease (CKD) has not been reported. Moreover, a comparison of SSE and high-intensity interval exercise (HIIE) may add to clinically-relevant findings for improving vascular function in mid-spectrum CKD. PURPOSE: To determine the influence of SSE and a comparable amount of HIIE on post-exercise brachial artery FMD in patients diagnosed with secondary Stage 3 or 4 CKD. METHODS: Twenty participants (n = 10 men; n = 10 women; age 62.0 + 9.9 yr; weight 80.9 + 16.2 kg; body fat 37.3 + 8.5% of weight; VO2max 19.4 + 4.7 ml/kg/min) completed 30 min of SSE at 65% VO2reserve or HIIE by treadmill walking (90% and 20% of VO2reserve in 3:2 min ratio) in a randomized crossover design. Both exercise conditions averaged ~ 65% VO2reserve. Ultrasound measurements of brachial artery FMD were obtained by the same technician under standardized conditions just before, 1 hr and 24 hrs after exercise. FMD responses were analyzed using 2 (condition) by 3 (sample point) repeated measures ANOVAs. RESULTS: Brachial artery FMD responses were augmented 1 hr after exercise in both exercise conditions (p \u3c 0.005 versus pre-exercise FMD). SSE (pre-exercise = 11.5 + 1.3; 1 hr = 17.2* + 1.8; 24 hr = 14.0* + 1.1%) HIIE (pre-exercise = 12.5 + 1.3; 1 hr = 15.6* + 1.5; 24 hr = 15.8* + 1.2%) CONCLUSION: We report for the first time that brachial artery FMD can be augmented by a single episode of exercise in mid-spectrum CKD. SSE and HIIE, averaging ~65% of VO2reserve, is equally effective at transiently improving conduit artery vascular function in this clinical population

Metabolic and Vascular Responses to Short, Disrupted Sleep and High-Intensity Interval Exercise in Healthy Men

Exercise is known to impart transient blood lipid and vascular responses that appear consistent with reduced cardiovascular disease risk. It is unclear how short, disrupted sleep (SDS) modifies post-exercise blood lipid and conduit artery responses to a single episode of exercise. PURPOSE: To determine the influence of a single night of SDS on fasting and postprandial blood lipid and conduit artery responses after high-intensity interval exercise (HIIE). METHODS: Fifteen male participants (age 31.1 + 5.3 yr; weight 83.5 + 11.4 kg; BMI 25.8 + 2.7 kg/m2; VO2max 49.1 + 8.5 ml/kg/min) completed a non-exercise control trial after 9 to 9.5 hrs of reference sleep (REF), HIIE by treadmill running (90% and 40% of VO2reserve in 3:2 min ratio) to expend 500 kcals after reference sleep (REF+EX) and HIIE after 3 to 3.5 hrs of short and disrupted sleep (SDS+EX) in a randomized crossover design. Blood samples were obtained by the same technician under standardized conditions just before, immediately after (IPE), 1 hr after exercise (1 HR) and just before a high-fat meal - 1240 kcals (56 g fat; 145 g carbohydrate; 38 g protein) and again 2, 4 and 6hrs after meal ingestion and at equal intervals during REF. Ultrasound measurements of brachial artery flow-mediated dilation (FMD) were obtained in a similar manner just before, 1 hr and 4 hrs after exercise. Total, high-density and low-density lipoprotein cholesterol (HDLC and LDLC) and paraoxonase-1 concentration were measured up to 1 hr post-exercise. Post-prandial triglyceride was measured and area under the curves - total (AUCt) and incremental (AUCi) were calculated. Lipid, lipid-related antioxidant, and FMD responses were analyzed using 3 (condition) by 3 (sample point) repeated measures ANOVAs. AUCt and AUCi were measured using one-way, 3 (condition) repeated measures ANOVAs. RESULTS: HDLC (+6.3%, p = 0.0023) and paraoxonase-1 (+10.8%, pCONCLUSION: Exercise transiently increased brachial artery FMD, fasting HDL cholesterol, and related antioxidant concentrations and reduced triglyceride levels, but did not modify total or incremental triglyceride AUC in response to a post-exercise high-fat meal. Exercise effects on blood lipids and vascular function were not influenced by a single night of short, disrupted sleep

The sources and methods used in the creation of the Levy Institute Measure of Economic Well-Being for the United States, 1959-2013

This paper documents the sources of data used in the construction of the estimates of the Levy Institute Measure of Economic Wellbeing (LIMEW) for the years 1959, 1972, 1982, 1989, 1992, 1995, 2000, 2001, 2004, 2007, 2010, and 2013. It also documents the methods used to combine the various sources of data into the synthetic dataset used to produce each year's LIMEW estimates

Stagnating economic well-being and unrelenting inequality: Post-2000 trends in the United States

Ajit Zacharias, Thomas Masterson, and Fernando Rios-Avila update the Levy Institute Measure of Economic Well-Being (LIMEW) for US households for the period 2000-13. The LIMEW - which comprises base income, income from wealth, net government expenditures, and the value of household production - is aimed at achieving a more comprehensive understanding of trends in living standards. This policy brief analyzes developments during this period at all levels of the LIMEW distribution, with a particular focus on the significant role played by net government expenditures. The overall trend for 2000 - 13 was one of historic stagnation in the growth of economic well-being for US households, but an examination of the different components of the measure reveals significant shifts taking place behind this headline trend