Evaluating Human-Language Model Interaction

Many real-world applications of language models (LMs), such as writing assistance and code autocomplete, involve human-LM interaction. However, most benchmarks are non-interactive in that a model produces output without human involvement. To evaluate human-LM interaction, we develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems and dimensions to consider when designing evaluation metrics. Compared to standard, non-interactive evaluation, HALIE captures (i) the interactive process, not only the final output; (ii) the first-person subjective experience, not just a third-party assessment; and (iii) notions of preference beyond quality (e.g., enjoyment and ownership). We then design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. With four state-of-the-art LMs (three variants of OpenAI's GPT-3 and AI21 Labs' Jurassic-1), we find that better non-interactive performance does not always translate to better human-LM interaction. In particular, we highlight three cases where the results from non-interactive and interactive metrics diverge and underscore the importance of human-LM interaction for LM evaluation.Comment: Authored by the Center for Research on Foundation Models (CRFM) at the Stanford Institute for Human-Centered Artificial Intelligence (HAI

Atomic Resonance and Scattering

Contains reports on six research projects.National Science Foundation (PHY83-06273)Joint Services Electronics Program (DAAL03-86-K-0002)National Science Foundation (PHY84-11483)U.S. Navy-Office of Naval Research (Grant N00014-79-C-0183)Joint Services Electronics Program (Contract DAAG29-83-K-0003)National Science Foundation (Grant PHY83-07172-A01)U.S. Navy - Office of Naval Research (Grant N00014-83-K-0695)National Science Foundation (Grant CHE84-21392

Common variants in P2RY11 are associated with narcolepsy.

l e t t e r s Growing evidence supports the hypothesis that narcolepsy with cataplexy is an autoimmune disease. We here report genomewide association analyses for narcolepsy with replication and fine mapping across three ethnic groups (3,406 individuals of European ancestry, 2,414 Asians and 302 African Americans). We identify a SNP in the 3′ untranslated region of P2RY11, the purinergic receptor subtype P2Y 11 gene, which is associated with narcolepsy (rs2305795, combined P = 6.1 × 10 −10 , odds ratio = 1.28, 95% CI 1.19-1.39, n = 5689). The diseaseassociated allele is correlated with reduced expression of P2RY11 in CD8 + T lymphocytes (339% reduced, P = 0.003) and natural killer (NK) cells (P = 0.031), but not in other peripheral blood mononuclear cell types. The low expression variant is also associated with reduced P2RY11-mediated resistance to ATP-induced cell death in T lymphocytes (P = 0.0007) and natural killer cells (P = 0.001). These results identify P2RY11 as an important regulator of immune-cell survival, with possible implications in narcolepsy and other autoimmune diseases

Common variants in P2RY11 are associated with narcolepsy.

Growing evidence supports the hypothesis that narcolepsy with cataplexy is an autoimmune disease. We here report genome-wide association analyses for narcolepsy with replication and fine mapping across three ethnic groups (3,406 individuals of European ancestry, 2,414 Asians and 302 African Americans). We identify a SNP in the 3' untranslated region of P2RY11, the purinergic receptor subtype P2Y₁₁ gene, which is associated with narcolepsy (rs2305795, combined P = 6.1 × 10⁻¹⁰, odds ratio = 1.28, 95% CI 1.19-1.39, n = 5689). The disease-associated allele is correlated with reduced expression of P2RY11 in CD8(+) T lymphocytes (339% reduced, P = 0.003) and natural killer (NK) cells (P = 0.031), but not in other peripheral blood mononuclear cell types. The low expression variant is also associated with reduced P2RY11-mediated resistance to ATP-induced cell death in T lymphocytes (P = 0.0007) and natural killer cells (P = 0.001). These results identify P2RY11 as an important regulator of immune-cell survival, with possible implications in narcolepsy and other autoimmune diseases.journal articleresearch support, n.i.h., extramuralresearch support, non-u.s. gov'tresearch support, u.s. gov't, p.h.s.2011 Jan2010 12 19importedErratum in : Nat Genet. 2011 Oct;43(10):1040

Highly expressed recombinant human follicle-stimulating hormone from Chinese hamster ovary cells grown in serum-free medium and its effect on induction of folliculogenesis and ovulation

Objective: To develop efficient Chinese hamster ovary (CHO) cells that express recombinant human FSH (rhFSH) in serum-free conditions and to investigate the effect of this newly synthesized rhFSH on folliculogenesis and ovulation. Design: Experimental study. Setting: Seoul National University, South Korea. Animals: Forty immature hypophysectomized rats and 40 androgen-sterilized mice. Intervention(s): A stable single CHO cell that expresses rhFSH at a high level was obtained by introducing the human chorionic gonadotropin (hCG) alpha-subunit and FSH beta-subunit genes. After purification processing, we investigated the effect of this newly synthesized rhFSH on folliculogenesis in hypophysectomized rats and ovulation in androgen-sterilized mice. Main Outcome Measure(s): The ovary weight, uterine weight, number of follicles, and ovarian morphology were evaluated in immature hypophysectomized rats. The number of ovulated oocytes and ovarian morphology were examined in androgen-sterilized mice. Result(s): After purification processing, we analyzed the new rhFSH using matrix-associated laser desorption ionization-time of flight and found that this new rhFSH increased both ovarian weight and uterine weight in hypophysectomized rats and induced ovulation in androgen-sterilized mice. Conclusion(s): This newly synthesized rhFSH might be safely used in anovulatory infertile woman as well as in ovulation induction protocols for subfertile women. (Fertil Steril (R) 2010; 93: 2652-60. (C) 2010 by American Society for Reproductive Medicine.)Y

Additional file 1: of An adaptive detection method for fetal chromosomal aneuploidy using cell-free DNA from 447 Korean women

Figure S1 showed optimally adaptive reference samples extracted from all reference samples. Figure S2 showed that GC correction played an important role in reducing the CV. Figures S3.1, S3.2, S4.1, S4.2, S5 and S6 represented similar results to our adaptive sample selection. Figure S7 represented the relationship of the reads fractions and the GC contents of samples. (DOCX 2063 kb

Lineage-dependent gene expression programs influence the immune landscape of colorectal cancer

Immunotherapy for metastatic colorectal cancer is effective only for mismatch repair-deficient tumors with high microsatellite instability that demonstrate immune infiltration, suggesting that tumor cells can determine their immune microenvironment. To understand this cross-talk, we analyzed the transcriptome of 91,103 unsorted single cells from 23 Korean and 6 Belgian patients. Cancer cells displayed transcriptional features reminiscent of normal differentiation programs, and genetic alterations that apparently fostered immunosuppressive microenvironments directed by regulatory T cells, myofibroblasts and myeloid cells. Intercellular network reconstruction supported the association between cancer cell signatures and specific stromal or immune cell populations. Our collective view of the cellular landscape and intercellular interactions in colorectal cancer provide mechanistic information for the design of efficient immuno-oncology treatment strategies.status: publishe