Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models

Large language models produce human-like text that drive a growing number of applications. However, recent literature and, increasingly, real world observations, have demonstrated that these models can generate language that is toxic, biased, untruthful or otherwise harmful. Though work to evaluate language model harms is under way, translating foresight about which harms may arise into rigorous benchmarks is not straightforward. To facilitate this translation, we outline six ways of characterizing harmful text which merit explicit consideration when designing new benchmarks. We then use these characteristics as a lens to identify trends and gaps in existing benchmarks. Finally, we apply them in a case study of the Perspective API, a toxicity classifier that is widely used in harm benchmarks. Our characteristics provide one piece of the bridge that translates between foresight and effective evaluation.Comment: Accepted to NeurIPS 2022 Datasets and Benchmarks Track; 10 pages plus appendi

Phase Equilibrium, Excess Enthalpies, and Densities of Binary Mixtures of Trimethylbutylammonium Bis(trifluoromethylsulfonyl)imide with Ethanol, 1‑Propanol, and Dimethylformamide

The molecular interactions between the ionic liquid trimethylbutylammomium bis­(trifluoromethylsulfonyl)­imide, [N₁₁₁₄]­[Tf₂N], and three common solvents, including short chain alcohols (ethanol and 1-propanol) and an aprotic solvent (<i>N,N</i>-dimethylformamide or DMF) were investigated through experimental properties. The excess enthalpies were measured by calorimetry at temperatures from 308.15 K to 323.15 K over the whole composition range and excess volumes were calculated from density measurements. The vapor–liquid equilibrium (VLE) measurements of the [N₁₁₁₄]­[Tf₂N] + DMF mixture were obtained using a headspace gas chromatography instrument. It was determined that hydrogen bonds between the alcohol molecules contribute to the positive excess enthalpies, while the favorable interactions between DMF and [N₁₁₁₄]­[Tf₂N] might be explained by electrostatic interactions between the ionic liquid and the zwitterionic resonance structure of DMF. The nonrandom two liquid (NRTL) equation was successfully used to correlate the excess enthalpy and VLE results