The History and Risks of Reinforcement Learning and Human Feedback

Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) easier to use and more effective. A core piece of the RLHF process is the training and utilization of a model of human preferences that acts as a reward function for optimization. This approach, which operates at the intersection of many stakeholders and academic disciplines, remains poorly understood. RLHF reward models are often cited as being central to achieving performance, yet very few descriptors of capabilities, evaluations, training methods, or open-source models exist. Given this lack of information, further study and transparency is needed for learned RLHF reward models. In this paper, we illustrate the complex history of optimizing preferences, and articulate lines of inquiry to understand the sociotechnical context of reward models. In particular, we highlight the ontological differences between costs, rewards, and preferences at stake in RLHF's foundations, related methodological tensions, and possible research directions to improve general understanding of how reward models function.Comment: 14 pages, 3 figure

Towards Studying Hierarchical Assembly in Real Time: A Milky Way Progenitor Galaxy at z = 2.36 under the Microscope

We use Hubble Space Telescope (HST) imaging and near-infrared spectroscopy from Keck/MOSFIRE to study the sub-structure around the progenitor of a Milky Way-mass galaxy in the Hubble Frontier Fields (HFF). Specifically, we study an $r_e = 40^{+70}_{-30}$pc, $M_{\star} \sim 10^{8.2} M_{\odot}$ rest-frame ultra-violet luminous "clump" at a projected distance of $\sim$100~pc from a $M_{\star} \sim 10^{9.8}$M$_{\odot}$ galaxy at $z = 2.36$ with a magnification $\mu = 5.21$. We measure the star formation history of the clump and galaxy by jointly modeling the broadband spectral energy distribution from HST photometry and H$\alpha$ from MOSFIRE spectroscopy. Given our inferred properties (e.g., mass, metallicity, dust) of the clump and galaxy, we explore scenarios in which the clump formed \emph{in-situ} (e.g., a star forming complex) or \emph{ex-situ} (e.g., a dwarf galaxy being accreted). If it formed \emph{in-situ}, we conclude that the clump is likely a single entity as opposed to a aggregation of smaller star clusters, making it one of the most dense star clusters cataloged. If it formed \emph{ex-situ}, then we are witnessing an accretion event with a 1:40 stellar mass ratio. However, our data alone are not informative enough to distinguish between \emph{in-situ} and \emph{ex-situ} scenarios to a high level of significance. We posit that the addition of high-fidelity metallicity information, such as [OIII]4363\AA, which can be detected at modest S/N with only a few hours of JWST/NIRSpec time, may be a powerful discriminant. We suggest that studying larger samples of moderately lensed sub-structures across cosmic time can provide unique insight into the hierarchical formation of galaxies like the Milky Way.Comment: Accepted to MNRA

The MOSDEF survey: a stellar mass-SFR-metallicity relation exists at z∼2.3z\sim2.3

We investigate the nature of the relation among stellar mass, star-formation rate, and gas-phase metallicity (the M$_*$-SFR-Z relation) at high redshifts using a sample of 260 star-forming galaxies at $z\sim2.3$ from the MOSDEF survey. We present an analysis of the high-redshift M$_*$-SFR-Z relation based on several emission-line ratios for the first time. We show that a M$_*$-SFR-Z relation clearly exists at $z\sim2.3$. The strength of this relation is similar to predictions from cosmological hydrodynamical simulations. By performing a direct comparison of stacks of $z\sim0$ and $z\sim2.3$ galaxies, we find that $z\sim2.3$ galaxies have $\sim0.1$ dex lower metallicity at fixed M$_*$ and SFR. In the context of chemical evolution models, this evolution of the M$_*$-SFR-Z relation suggests an increase with redshift of the mass-loading factor at fixed M$_*$, as well as a decrease in the metallicity of infalling gas that is likely due to a lower importance of gas recycling relative to accretion from the intergalactic medium at high redshifts. Performing this analysis simultaneously with multiple metallicity-sensitive line ratios allows us to rule out the evolution in physical conditions (e.g., N/O ratio, ionization parameter, and hardness of the ionizing spectrum) at fixed metallicity as the source of the observed trends with redshift and with SFR at fixed M$_*$ at $z\sim2.3$. While this study highlights the promise of performing high-order tests of chemical evolution models at high redshifts, detailed quantitative comparisons ultimately await a full understanding of the evolution of metallicity calibrations with redshift.Comment: 19 pages, 8 figures, accepted to Ap

The MOSDEF Survey: Kinematic and Structural Evolution of Star-Forming Galaxies at 1.4≤z≤3.81.4\leq z\leq 3.8

We present ionized gas kinematics for 681 galaxies at $z\sim 1.4-3.8$ from the MOSFIRE Deep Evolution Field survey, measured using models which account for random galaxy-slit misalignments together with structural parameters derived from CANDELS Hubble Space Telescope (HST) imaging. Kinematics and sizes are used to derive dynamical masses. Baryonic masses are estimated from stellar masses and inferred gas masses from dust-corrected star formation rates (SFRs) and the Kennicutt-Schmidt relation. We measure resolved rotation for 105 galaxies. For the remaining 576 galaxies we use models based on HST imaging structural parameters together with integrated velocity dispersions and baryonic masses to statistically constrain the median ratio of intrinsic ordered to disordered motion, $V/\sigma_{V,0}$. We find that $V/\sigma_{V,0}$ increases with increasing stellar mass and decreasing specific SFR (sSFR). These trends may reflect marginal disk stability, where systems with higher gas fractions have thicker disks. For galaxies with detected rotation we assess trends between their kinematics and mass, sSFR, and baryon surface density ($\Sigma_{\mathrm{bar},e}$). Intrinsic dispersion correlates most with $\Sigma_{\mathrm{bar},e}$ and velocity correlates most with mass. By comparing dynamical and baryonic masses, we find that galaxies at $z\sim 1.4-3.8$ are baryon dominated within their effective radii ($R_E$), with Mdyn/Mbaryon increasing over time. The inferred baryon fractions within $R_E$, $f_{\mathrm{bar}}$, decrease over time, even at fixed mass, size, or surface density. At fixed redshift, $f_{\mathrm{bar}}$ does not appear to vary with stellar mass but increases with decreasing $R_E$ and increasing $\Sigma_{\mathrm{bar},e}$. For galaxies at $z\geq2$, the median inferred baryon fractions generally exceed 100%. We discuss possible explanations and future avenues to resolve this tension.Comment: Accepted to ApJ. Added Figure 9, corrected sample size (main results unchanged). 28 pages, 13 figure

The MOSDEF Survey: An Improved Voronoi Binning Technique on Spatially Resolved Stellar Populations at z~2

We use a sample of 350 star-forming galaxies at $1.25<z<2.66$ from the MOSFIRE Deep Evolution Field survey to demonstrate an improved Voronoi binning technique that we use to study the properties of resolved stellar populations in $z\sim2$ galaxies. Stellar population and dust maps are constructed from the high-resolution CANDELS/3D-HST multi-band imaging. Rather than constructing the layout of resolved elements (i.e., Voronoi bins) from the S/N distribution of the $H_{160}$-band alone, we introduce a modified Voronoi binning method that additionally incorporates the S/N distribution of several resolved filters. The SED-derived resolved E(B-V)$_{\text{stars}}$, stellar population ages, SFRs, and stellar masses that are inferred from the Voronoi bins constructed from multiple filters are generally consistent with the properties inferred from the integrated photometry within the uncertainties, with the exception of the inferred E(B-V)$_{\text{stars}}$ from our $z\sim1.5$ sample due to their UV slopes being unconstrained by the resolved photometry. The results from our multi-filter Voronoi binning technique are compared to those derived from a "traditional" single-filter Voronoi binning approach. We find that single-filter binning produces inferred E(B-V)$_{\text{stars}}$ that are systematically redder by 0.02 mag on average, but could differ by up to 0.20 mag, and could be attributed to poorly constrained resolved photometry covering the UV slope. Overall, we advocate that our methodology produces more reliable SED-derived parameters due to the best-fit resolved SEDs being better constrained at all resolved wavelengths--particularly those covering the UV slope.Comment: 23 pages, 15 figures, accepted for publication in MNRA

The MOSDEF Survey: The First Direct Measurements of the Nebular Dust Attenuation Curve at High Redshift

We use a sample of 532 star-forming galaxies at redshifts z ≃ 1.4–2.6 with deep rest-frame optical spectra from the MOSFIRE Deep Evolution Field (MOSDEF) survey to place the first constraints on the nebular attenuation curve at high redshift. Based on the first five low-order Balmer emission lines detected in the composite spectra of these galaxies (Hα through Hε), we derive a nebular attenuation curve that is similar in shape to that of the Galactic extinction curve, suggesting that the dust covering fraction and absorption/scattering properties along the lines of sight to massive stars at high redshift are similar to those of the average Milky Way sight line. The curve derived here implies nebular reddening values that are, on average, systematically larger than those derived for the stellar continuum. In the context of stellar population synthesis models that include the effects of stellar multiplicity, the difference in reddening of the nebular lines and stellar continuum may imply molecular cloud crossing timescales that are a factor of ≳ 3x longer than those inferred for local molecular clouds, star formation rates that are constant or increasing with time such that newly formed and dustier OB associations always dominate the ionizing flux, and/or that the dust responsible for reddening the nebular emission may be associated with nonmolecular (i.e., ionized and neutral) phases of the interstellar medium. Our analysis points to a variety of investigations of the nebular attenuation curve that will be enabled with the next generation of ground- and space-based facilities

The MOSDEF Survey: the Variation of the Dust Attenuation Curve with Metallicity

We derive the UV-optical stellar dust attenuation curve of galaxies at z=1.4-2.6 as a function of gas-phase metallicity. We use a sample of 218 star-forming galaxies, excluding those with very young or heavily obscured star formation, from the MOSFIRE Deep Evolution Field (MOSDEF) survey with H$\alpha$, H$\beta$, and [NII]$\lambda 6585$ spectroscopic measurements. We constrain the shape of the attenuation curve by comparing the average flux densities of galaxies sorted into bins of dust obscuration using Balmer decrements, i.e., H$\alpha$-to-H$\beta$ luminosities. The average attenuation curve for the high-metallicity sample (12+log(O/H)>8.5, corresponding to $M_*\gtrsim10^{10.4}\,M_{\odot}$) has a shallow slope, identical to that of the Calzetti local starburst curve, and a significant UV 2175A extinction bump that is $\sim 0.5\times$ the strength of the Milky Way bump. On the other hand, the average attenuation curve of the low-metallicity sample (12+log(O/H) $\sim 8.2-8.5$) has a steeper slope similar to that of the SMC curve, only consistent with the Calzetti slope at the $3\sigma$ level. The UV bump is not detected in the low-metallicity curve, indicating the relative lack of the small dust grains causing the bump at low metallicities. Furthermore, we find that on average the nebular reddening (E(B-V)) is a factor of 2 times larger than that of the stellar continuum for galaxies with low metallicities, while the nebular and stellar reddening are similar for galaxies with higher metallicities. The latter is likely due to a high surface density of dusty clouds embedding the star forming regions but also reddening the continuum in the high-metallicity galaxies.Comment: 20 pages and 9 figures and 1 appendix, accepted for publication in Ap

The MOSDEF Survey: Significant Evolution in the Rest-Frame Optical Emission Line Equivalent Widths of Star-Forming Galaxies at z=1.4-3.8

We use extensive spectroscopy from the MOSFIRE Deep Evolution Field (MOSDEF) survey to investigate the relationships between rest-frame optical emission line equivalent widths ($W$) and a number of galaxy and ISM characteristics for a sample of $1134$ star-forming galaxies at redshifts $1.4\lesssim z\lesssim 3.8$. We examine how the equivalent widths of [OII]$\lambda\lambda 3727, 3730$, H$\beta$, [OIII]$\lambda\lambda 4960, 5008$, [OIII]$+$H$\beta$, H$\alpha$, and H$\alpha$+[NII]$\lambda\lambda 6550, 6585$, depend on stellar mass, UV slope, age, star-formation rate (SFR) and specific SFR (sSFR), ionization parameter and excitation conditions (O32 and [OIII]/H$\beta$), gas-phase metallicity, and ionizing photon production efficiency ($\xi_{\rm ion}$). The trend of increasing $W$ with decreasing stellar mass is strongest for [OIII] (and [OIII]+H$\beta$). More generally, the equivalent widths of all the lines increase with redshift at a fixed stellar mass or fixed gas-phase metallicity, suggesting that high equivalent width galaxies are common at high redshift. This redshift evolution in equivalent widths can be explained by the increase in SFR and decrease in metallicity with redshift at a fixed stellar mass. Consequently, the dependence of $W$ on sSFR is largely invariant with redshift, particularly when examined for galaxies of a given metallicity. Our results show that high equivalent width galaxies, specifically those with high $W({\rm [OIII]})$, have low stellar masses, blue UV slopes, young ages, high sSFRs, ISM line ratios indicative of high ionization parameters, high $\xi_{\rm ion}$, and low metallicities. As these characteristics are often attributed to galaxies with high ionizing escape fractions, galaxies with high $W$ are likely candidates for the population that dominates cosmic reionization.Comment: 34 pages, 8 tables, 28 figures; submitted 2018 August 23, accepted 2018 October 29 to the Astrophysical Journa