Annotating patient clinical records with syntactic chunks and named entities: the Harvey corpus

The free text notes typed by physicians during patient consultations contain valuable information for the study of disease and treatment. These notes are difficult to process by existing natural language analysis tools since they are highly telegraphic (omitting many words), and contain many spelling mistakes, inconsistencies in punctuation, and non-standard word order. To support information extraction and classification tasks over such text, we describe a de-identified corpus of free text notes, a shallow syntactic and named entity annotation scheme for this kind of text, and an approach to training domain specialists with no linguistic background to annotate the text. Finally, we present a statistical chunking system for such clinical text with a stable learning rate and good accuracy, indicating that the manual annotation is consistent and that the annotation scheme is tractable for machine learning

Bringing Structure into Summaries: Crowdsourcing a Benchmark Corpus of Concept Maps

Concept maps can be used to concisely represent important information and bring structure into large document collections. Therefore, we study a variant of multi-document summarization that produces summaries in the form of concept maps. However, suitable evaluation datasets for this task are currently missing. To close this gap, we present a newly created corpus of concept maps that summarize heterogeneous collections of web documents on educational topics. It was created using a novel crowdsourcing approach that allows us to efficiently determine important elements in large document collections. We release the corpus along with a baseline system and proposed evaluation protocol to enable further research on this variant of summarization.Comment: Published at EMNLP 201

Confronting objections to performance pay: A study of the impact of individual and gain-sharing incentives on the job satisfaction of British employees

The increasing interest in incentive pay schemes in recent years has raised concerns regarding their potential damaging effect on intrinsic job satisfaction, or the security of employment. This study explores the impact of both individual and gain-sharing incentives on the overall job satisfaction of workers in the UK, as well as their satisfaction with various facets of jobs, namely total pay, job security, and the actual work itself. Using data from six waves (1998-2003) of the British Household Panel Survey (BHPS), and after correcting for the sorting problem that arises, no significant difference in overall job utility is found between those receiving performance-related pay (PRP) and those on other methods of compensation. In addition, non-economic arguments that PRP crowds-out the intrinsic satisfaction of jobs are also not supported, as are popular concerns regarding the adverse impact of PRP schemes on job security. An important asymmetry in the manner in which individual and gain-sharing incentives affect the utility of employees is nonetheless unearthed, as the latter are consistently found to have a positive effect on employee well-being

D7.1. Criteria for evaluation of resources, technology and integration.

This deliverable defines how evaluation is carried out at each integration cycle in the PANACEA project. As PANACEA aims at producing large scale resources, evaluation becomes a critical and challenging issue. Critical because it is important to assess the quality of the results that should be delivered to users. Challenging because we prospect rather new areas, and through a technical platform: some new methodologies will have to be explored or old ones to be adapted

Identifying student- and class-level correlates of sixth-grade students’ listening comprehension

Despite the importance of listening, little investigation of potential correlates of listening comprehension in the language of schooling is done. Therefore, the purpose of this study was to investigate which student- and class-level characteristics are related to sixth-grade students' listening skills in Flanders. A sample of 974 students in 70 classes completed a listening test in order to gather information on their ability to understand and interpret oral information. Further, different questionnaires were administered to the students, their parents and teachers. The results of the hierarchical regression analysis with multilevel design showed that the differences in listening comprehension skills could be primarily attributed to differences in student-level characteristics. The results indicated that students with higher working memory ability, more vocabulary knowledge and lower extrinsic listening motivation performed significantly better on the listening test. In addition, the educational level of the parents and the language diversity in the class was significantly related to students' listening skills in the language of schooling. This study is an important starting point in unraveling the black box of listening skills in the elementary school context. Suggestions for further research and practice were made

Self-determination, Success, and College Readiness of First Generation Students in A Higher Education Institution

The purpose of the study was to describe and compare if self-determination factors differed in first and non-first generation college students and success levels. Additionally, comparisons of college readiness levels were measured, and finally a measure of factors that contribute to college success based in first and second semester grade point averages were investigated using multiple linear regression analysis. Using a Self-Determination Theory framework of human motivation, helps identify a basic psychological need for autonomy as a central feature for understanding effective regulation and well-being (Moller, Ryan, & Deci, 2006). The target population in this study was 1,586 returning students of the 2008 freshmen cohort for the University of Texas Pan American in the Fall of 2010. About 10% or 187 of the returning students completed the Academic Motivation Scale survey. However, after additional information were collected only 146 subjects had usable data for all variables. A number of group comparisons were made based on college generation, success as measured by first and second semester college grade point averages, and college readiness as measured by ACT composite scores. The comparisons yielded no differences in motivation with the exception of intrinsic motivation, to experience stimulation, where successful students showed higher motivation than less successful students. This phenomenon was only apparent the second semester of college attendance, not the first semester. The overall descriptive analysis indicates that there are very few differences in intrinsic and extrinsic motivation in these groups of students. Multiple linear regression analyses revealed that external regulation (external motivation) contributed with 5.8 percent of the total variance in grade point average for the first semester, toward accomplishment (intrinsic motivation) contributed with 4.7 percent, and the ACT composite score contributed with an additional 3.9 percent of the total variance in first semester grade point average. The variables that explained the amount of variance in second semester grade point average were to know (internal motivation) with 2.7 percent, and external regulation (extrinsic motivation) with 3.6 percent

Surgicberta: a pre-trained language model for procedural surgical language

Pre-trained language models are now ubiquitous in natural language processing, being successfully applied for many different tasks and in several real-world applications. However, even though there is a wealth of high-quality written materials on surgery, and the scientific community has shown a growing interest in the application of natural language processing techniques in surgery, a pre-trained language model specific to the surgical domain is still missing. The creation and public release of such a model would serve numerous useful clinical applications. For example, it could enhance existing surgical knowledge bases employed for task automation, or assist medical students in summarizing complex surgical descriptions. For this reason, in this paper, we introduce SurgicBERTa, a pre-trained language model specific for the English surgical language, i.e., the language used in the surgical domain. SurgicBERTa has been obtained from RoBERTa through continued pre-training with the Masked language modeling objective on 300 k sentences taken from English surgical books and papers, for a total of 7 million words. By publicly releasing SurgicBERTa, we make available a resource built from the content collected in many high-quality surgical books, online textual resources, and academic papers. We performed several assessments in order to evaluate SurgicBERTa, comparing it with the general domain RoBERTa. First, we intrinsically assessed the model in terms of perplexity, accuracy, and evaluation loss resulting from the continual training according to the masked language modeling task. Then, we extrinsically evaluated SurgicBERTa on several downstream tasks, namely (i) procedural sentence detection, (ii) procedural knowledge extraction, (iii) ontological information discovery, and (iv) surgical terminology acquisition. Finally, we conducted some qualitative analysis on SurgicBERTa, showing that it contains a lot of surgical knowledge that could be useful to enrich existing state-of-the-art surgical knowledge bases or to extract surgical knowledge. All the assessments show that SurgicBERTa better deals with surgical language than a general-purpose pre-trained language model such as RoBERTa, and therefore can be effectively exploited in many computer-assisted applications in the surgical domain

Machine translation evaluation resources and methods: a survey

We introduce the Machine Translation (MT) evaluation survey that contains both manual and automatic evaluation methods. The traditional human evaluation criteria mainly include the intelligibility, fidelity, fluency, adequacy, comprehension, and informativeness. The advanced human assessments include task-oriented measures, post-editing, segment ranking, and extended criteriea, etc. We classify the automatic evaluation methods into two categories, including lexical similarity scenario and linguistic features application. The lexical similarity methods contain edit distance, precision, recall, F-measure, and word order. The linguistic features can be divided into syntactic features and semantic features respectively. The syntactic features include part of speech tag, phrase types and sentence structures, and the semantic features include named entity, synonyms, textual entailment, paraphrase, semantic roles, and language models. The deep learning models for evaluation are very newly proposed. Subsequently, we also introduce the evaluation methods for MT evaluation including different correlation scores, and the recent quality estimation (QE) tasks for MT. This paper differs from the existing works\cite {GALEprogram2009, EuroMatrixProject2007} from several aspects, by introducing some recent development of MT evaluation measures, the different classifications from manual to automatic evaluation measures, the introduction of recent QE tasks of MT, and the concise construction of the content