47 research outputs found
The History and Risks of Reinforcement Learning and Human Feedback
Reinforcement learning from human feedback (RLHF) has emerged as a powerful
technique to make large language models (LLMs) easier to use and more
effective. A core piece of the RLHF process is the training and utilization of
a model of human preferences that acts as a reward function for optimization.
This approach, which operates at the intersection of many stakeholders and
academic disciplines, remains poorly understood. RLHF reward models are often
cited as being central to achieving performance, yet very few descriptors of
capabilities, evaluations, training methods, or open-source models exist. Given
this lack of information, further study and transparency is needed for learned
RLHF reward models. In this paper, we illustrate the complex history of
optimizing preferences, and articulate lines of inquiry to understand the
sociotechnical context of reward models. In particular, we highlight the
ontological differences between costs, rewards, and preferences at stake in
RLHF's foundations, related methodological tensions, and possible research
directions to improve general understanding of how reward models function.Comment: 14 pages, 3 figure
Towards Studying Hierarchical Assembly in Real Time: A Milky Way Progenitor Galaxy at z = 2.36 under the Microscope
We use Hubble Space Telescope (HST) imaging and near-infrared spectroscopy
from Keck/MOSFIRE to study the sub-structure around the progenitor of a Milky
Way-mass galaxy in the Hubble Frontier Fields (HFF). Specifically, we study an
pc, rest-frame
ultra-violet luminous "clump" at a projected distance of 100~pc from a
M galaxy at with a magnification
. We measure the star formation history of the clump and galaxy by
jointly modeling the broadband spectral energy distribution from HST photometry
and H from MOSFIRE spectroscopy. Given our inferred properties (e.g.,
mass, metallicity, dust) of the clump and galaxy, we explore scenarios in which
the clump formed \emph{in-situ} (e.g., a star forming complex) or
\emph{ex-situ} (e.g., a dwarf galaxy being accreted). If it formed
\emph{in-situ}, we conclude that the clump is likely a single entity as opposed
to a aggregation of smaller star clusters, making it one of the most dense star
clusters cataloged. If it formed \emph{ex-situ}, then we are witnessing an
accretion event with a 1:40 stellar mass ratio. However, our data alone are not
informative enough to distinguish between \emph{in-situ} and \emph{ex-situ}
scenarios to a high level of significance. We posit that the addition of
high-fidelity metallicity information, such as [OIII]4363\AA, which can be
detected at modest S/N with only a few hours of JWST/NIRSpec time, may be a
powerful discriminant. We suggest that studying larger samples of moderately
lensed sub-structures across cosmic time can provide unique insight into the
hierarchical formation of galaxies like the Milky Way.Comment: Accepted to MNRA
The MOSDEF survey: a stellar mass-SFR-metallicity relation exists at
We investigate the nature of the relation among stellar mass, star-formation
rate, and gas-phase metallicity (the M-SFR-Z relation) at high redshifts
using a sample of 260 star-forming galaxies at from the MOSDEF
survey. We present an analysis of the high-redshift M-SFR-Z relation based
on several emission-line ratios for the first time. We show that a M-SFR-Z
relation clearly exists at . The strength of this relation is similar
to predictions from cosmological hydrodynamical simulations. By performing a
direct comparison of stacks of and galaxies, we find that
galaxies have dex lower metallicity at fixed M and
SFR. In the context of chemical evolution models, this evolution of the
M-SFR-Z relation suggests an increase with redshift of the mass-loading
factor at fixed M, as well as a decrease in the metallicity of infalling
gas that is likely due to a lower importance of gas recycling relative to
accretion from the intergalactic medium at high redshifts. Performing this
analysis simultaneously with multiple metallicity-sensitive line ratios allows
us to rule out the evolution in physical conditions (e.g., N/O ratio,
ionization parameter, and hardness of the ionizing spectrum) at fixed
metallicity as the source of the observed trends with redshift and with SFR at
fixed M at . While this study highlights the promise of
performing high-order tests of chemical evolution models at high redshifts,
detailed quantitative comparisons ultimately await a full understanding of the
evolution of metallicity calibrations with redshift.Comment: 19 pages, 8 figures, accepted to Ap
The MOSDEF Survey: Kinematic and Structural Evolution of Star-Forming Galaxies at
We present ionized gas kinematics for 681 galaxies at from
the MOSFIRE Deep Evolution Field survey, measured using models which account
for random galaxy-slit misalignments together with structural parameters
derived from CANDELS Hubble Space Telescope (HST) imaging. Kinematics and sizes
are used to derive dynamical masses. Baryonic masses are estimated from stellar
masses and inferred gas masses from dust-corrected star formation rates (SFRs)
and the Kennicutt-Schmidt relation. We measure resolved rotation for 105
galaxies. For the remaining 576 galaxies we use models based on HST imaging
structural parameters together with integrated velocity dispersions and
baryonic masses to statistically constrain the median ratio of intrinsic
ordered to disordered motion, . We find that
increases with increasing stellar mass and decreasing specific SFR (sSFR).
These trends may reflect marginal disk stability, where systems with higher gas
fractions have thicker disks. For galaxies with detected rotation we assess
trends between their kinematics and mass, sSFR, and baryon surface density
(). Intrinsic dispersion correlates most with
and velocity correlates most with mass. By comparing
dynamical and baryonic masses, we find that galaxies at are
baryon dominated within their effective radii (), with Mdyn/Mbaryon
increasing over time. The inferred baryon fractions within ,
, decrease over time, even at fixed mass, size, or surface
density. At fixed redshift, does not appear to vary with
stellar mass but increases with decreasing and increasing
. For galaxies at , the median inferred baryon
fractions generally exceed 100%. We discuss possible explanations and future
avenues to resolve this tension.Comment: Accepted to ApJ. Added Figure 9, corrected sample size (main results
unchanged). 28 pages, 13 figure
The MOSDEF Survey: An Improved Voronoi Binning Technique on Spatially Resolved Stellar Populations at z~2
We use a sample of 350 star-forming galaxies at from the
MOSFIRE Deep Evolution Field survey to demonstrate an improved Voronoi binning
technique that we use to study the properties of resolved stellar populations
in galaxies. Stellar population and dust maps are constructed from the
high-resolution CANDELS/3D-HST multi-band imaging. Rather than constructing the
layout of resolved elements (i.e., Voronoi bins) from the S/N distribution of
the -band alone, we introduce a modified Voronoi binning method that
additionally incorporates the S/N distribution of several resolved filters. The
SED-derived resolved E(B-V), stellar population ages, SFRs,
and stellar masses that are inferred from the Voronoi bins constructed from
multiple filters are generally consistent with the properties inferred from the
integrated photometry within the uncertainties, with the exception of the
inferred E(B-V) from our sample due to their UV
slopes being unconstrained by the resolved photometry. The results from our
multi-filter Voronoi binning technique are compared to those derived from a
"traditional" single-filter Voronoi binning approach. We find that
single-filter binning produces inferred E(B-V) that are
systematically redder by 0.02 mag on average, but could differ by up to 0.20
mag, and could be attributed to poorly constrained resolved photometry covering
the UV slope. Overall, we advocate that our methodology produces more reliable
SED-derived parameters due to the best-fit resolved SEDs being better
constrained at all resolved wavelengths--particularly those covering the UV
slope.Comment: 23 pages, 15 figures, accepted for publication in MNRA
The MOSDEF Survey: The First Direct Measurements of the Nebular Dust Attenuation Curve at High Redshift
We use a sample of 532 star-forming galaxies at redshifts z ≃ 1.4–2.6 with deep rest-frame optical spectra from the MOSFIRE Deep Evolution Field (MOSDEF) survey to place the first constraints on the nebular attenuation curve at high redshift. Based on the first five low-order Balmer emission lines detected in the composite spectra of these galaxies (Hα through Hε), we derive a nebular attenuation curve that is similar in shape to that of the Galactic extinction curve, suggesting that the dust covering fraction and absorption/scattering properties along the lines of sight to massive stars at high redshift are similar to those of the average Milky Way sight line. The curve derived here implies nebular reddening values that are, on average, systematically larger than those derived for the stellar continuum. In the context of stellar population synthesis models that include the effects of stellar multiplicity, the difference in reddening of the nebular lines and stellar continuum may imply molecular cloud crossing timescales that are a factor of ≳ 3x longer than those inferred for local molecular clouds, star formation rates that are constant or increasing with time such that newly formed and dustier OB associations always dominate the ionizing flux, and/or that the dust responsible for reddening the nebular emission may be associated with nonmolecular (i.e., ionized and neutral) phases of the interstellar medium. Our analysis points to a variety of investigations of the nebular attenuation curve that will be enabled with the next generation of ground- and space-based facilities
The MOSDEF Survey: the Variation of the Dust Attenuation Curve with Metallicity
We derive the UV-optical stellar dust attenuation curve of galaxies at
z=1.4-2.6 as a function of gas-phase metallicity. We use a sample of 218
star-forming galaxies, excluding those with very young or heavily obscured star
formation, from the MOSFIRE Deep Evolution Field (MOSDEF) survey with
H, H, and [NII] spectroscopic measurements. We
constrain the shape of the attenuation curve by comparing the average flux
densities of galaxies sorted into bins of dust obscuration using Balmer
decrements, i.e., H-to-H luminosities. The average attenuation
curve for the high-metallicity sample (12+log(O/H)>8.5, corresponding to
) has a shallow slope, identical to that of the
Calzetti local starburst curve, and a significant UV 2175A extinction bump that
is the strength of the Milky Way bump. On the other hand, the
average attenuation curve of the low-metallicity sample (12+log(O/H) ) has a steeper slope similar to that of the SMC curve, only consistent
with the Calzetti slope at the level. The UV bump is not detected in
the low-metallicity curve, indicating the relative lack of the small dust
grains causing the bump at low metallicities. Furthermore, we find that on
average the nebular reddening (E(B-V)) is a factor of 2 times larger than that
of the stellar continuum for galaxies with low metallicities, while the nebular
and stellar reddening are similar for galaxies with higher metallicities. The
latter is likely due to a high surface density of dusty clouds embedding the
star forming regions but also reddening the continuum in the high-metallicity
galaxies.Comment: 20 pages and 9 figures and 1 appendix, accepted for publication in
Ap
The MOSDEF Survey: Significant Evolution in the Rest-Frame Optical Emission Line Equivalent Widths of Star-Forming Galaxies at z=1.4-3.8
We use extensive spectroscopy from the MOSFIRE Deep Evolution Field (MOSDEF)
survey to investigate the relationships between rest-frame optical emission
line equivalent widths () and a number of galaxy and ISM characteristics for
a sample of star-forming galaxies at redshifts . We examine how the equivalent widths of [OII],
H, [OIII], [OIII]H, H, and
H+[NII], depend on stellar mass, UV slope,
age, star-formation rate (SFR) and specific SFR (sSFR), ionization parameter
and excitation conditions (O32 and [OIII]/H), gas-phase metallicity, and
ionizing photon production efficiency (). The trend of
increasing with decreasing stellar mass is strongest for [OIII] (and
[OIII]+H). More generally, the equivalent widths of all the lines
increase with redshift at a fixed stellar mass or fixed gas-phase metallicity,
suggesting that high equivalent width galaxies are common at high redshift.
This redshift evolution in equivalent widths can be explained by the increase
in SFR and decrease in metallicity with redshift at a fixed stellar mass.
Consequently, the dependence of on sSFR is largely invariant with redshift,
particularly when examined for galaxies of a given metallicity. Our results
show that high equivalent width galaxies, specifically those with high , have low stellar masses, blue UV slopes, young ages, high sSFRs, ISM
line ratios indicative of high ionization parameters, high , and
low metallicities. As these characteristics are often attributed to galaxies
with high ionizing escape fractions, galaxies with high are likely
candidates for the population that dominates cosmic reionization.Comment: 34 pages, 8 tables, 28 figures; submitted 2018 August 23, accepted
2018 October 29 to the Astrophysical Journa