Contextual information and assessor characteristics in complex question answering

The ciqa track investigates the role of interaction in answering complex questions: questions that relate two or more entities by some specified relationship. In our submission to the first ciqa track we were interested in the interplay between groups of variables: variables describing the question creators, the questions asked and the presentation of answers to the questions. We used two interaction forms - html questionnaires completed before answer assessment - to gain contextual information from the answer assessors to better understand what factors influence assessors when judging retrieved answers to complex questions. Our results indicate the importance of understanding the assessor's personal relationship to the question - their existing topical knowledge for example - and also the presentation of the answers - contextual information about the answer to aid in the assessment of the answer

Persuasive, authorative and topical answers for complex question answering

The ciqa track investigates the role of interaction in answering complex questions: questions that relate two or more entities by some specified relationship. As in the ciqa 2006, our interest in ciqa 2007 was on contextual factors that may affect how answers are assessed. In ciqa 2006 we investigated factors such as topical knowledge or confidence in assessing answers through direct questioning – asking the ciqa assessors to directly estimate values for such variables using ordinal or categorical scales. In ciqa 2006 we found many useful relationships between personal contextual variables and how assessors judged answers. This year we were keen to follow this line of investigation, following a more specific research question: how do contextual variables affect the judgement of different types of information surrogate

A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries

There is growing interest in systems that generate timeline summaries by filtering high-volume streams of documents to retain only those that are relevant to a particular event or topic. Continued advances in algorithms and techniques for this task depend on standardized and reproducible evaluation methodologies for comparing systems. However, timeline summary evaluation is still in its infancy, with competing methodologies currently being explored in international evaluation forums such as TREC. One area of active exploration is how to explicitly represent the units of information that should appear in a 'good' summary. Currently, there are two main approaches, one based on identifying nuggets in an external 'ground truth', and the other based on clustering system outputs. In this paper, by building test collections that have both nugget and cluster annotations, we are able to compare these two approaches. Specifically, we address questions related to evaluation effort, differences in the final evaluation products, and correlations between scores and rankings generated by both approaches. We summarize advantages and disadvantages of nuggets and clusters to offer recommendations for future system evaluation

'My expectations remain the same. The student has to be competent to practise' : practice assessor perspectives on the new social work degree qualification in England

Research has emphasized the importance of practice learning to social work qualifying education but has tended to feature social work educator and student perspectives more strongly than the views of those responsible for assessing students' practice in the field. This article draws on 195 responses to a postal questionnaire sent at two points in time to practice assessors working with students from nine social work qualifying programmes run in six higher education institutions collected as part of the evaluation of the new social work degree qualification in England. While practice assessors described changes in their role and in the opportunities available to students, they also emphasized continuities, particularly in the skills that they expected students to possess. The key difficulty they identified was the heavy workload resulting from combining their role as practice assessors with their other responsibilities at work. Increases in the number of social work students and changes to the organization of services are likely to create further pressures on practice assessors. Given that these issues are faced by a number of different professions, the article concludes that there is potential for future studies to look at the experiences of practice educators across different professional qualifying programmes

Homeward bound or bound for a home? Assessing the capacity of dementia patients to make decisions about hospital discharge: Comparing practice with legal standards

Background This article stems from a larger project which considers ways of improving assessments of capacity and judgements about best interests in connection with people with dementia admitted to acute hospitals with respect to decisions about place of residence. Aims Our aim is to comment on how assessments of residence capacity are actually performed on general hospital wards compared with legal standards for the assessment of capacity set out in the Mental Capacity Act, 2005 (MCA). Method Our findings are grounded in ethnographic ward-based observations and in-depth interviews conducted in three hospital wards, in two hospitals (acute and rehabilitation), within two NHS healthcare trusts in the North of England over a period of nine months between 2008 and 2009. Twenty-nine patient cases were recruited to the study. We also draw from broader conceptions of capacity found in domestic and international legal, medical, ethical and social science literature. Results Our findings suggest that whilst professionals profess to be familiar with broad legal standards governing the assessment of capacity under the MCA, these standards are not routinely applied in practice in general hospital settings when assessing capacity to decide place of residence on discharge from hospital. We discuss whether the criteria set out in the MCA and the guidance in its Code of Practice are sufficient when assessing residence capacity, given the particular ambiguities and complexities of this capacity. Conclusions We conclude by suggesting that more specific legal standards are required when assessing capacity in this particular context

REFinD: Relation Extraction Financial Dataset

A number of datasets for Relation Extraction (RE) have been created to aide downstream tasks such as information retrieval, semantic search, question answering and textual entailment. However, these datasets fail to capture financial-domain specific challenges since most of these datasets are compiled using general knowledge sources such as Wikipedia, web-based text and news articles, hindering real-life progress and adoption within the financial world. To address this limitation, we propose REFinD, the first large-scale annotated dataset of relations, with $\sim$29K instances and 22 relations amongst 8 types of entity pairs, generated entirely over financial documents. We also provide an empirical evaluation with various state-of-the-art models as benchmarks for the RE task and highlight the challenges posed by our dataset. We observed that various state-of-the-art deep learning models struggle with numeric inference, relational and directional ambiguity

Leveraging Formulae and Text for Improved Math Retrieval

Large collections containing millions of math formulas are available online. Retrieving math expressions from these collections is challenging. Users can use formula, formula+text, or math questions to express their math information needs. The structural complexity of formulas requires specialized processing. Despite the existence of math search systems and online community question-answering websites for math, little is known about mathematical information needs. This research first explores the characteristics of math searches using a general search engine. The findings show how math searches are different from general searches. Then, test collections for math-aware search are introduced. The ARQMath test collections have two main tasks: 1) finding answers for math questions and 2) contextual formula search. In each test collection (ARQMath-1 to -3) the same collection is used, Math Stack Exchange posts from 2010 to 2018, introducing different topics for each task. Compared to the previous test collections, ARQMath has a much larger number of diverse topics, and improved evaluation protocol. Another key role of this research is to leverage text and math information for improved math information retrieval. Three formula search models that only use the formula, with no context are introduced. The first model is an n-gram embedding model using both symbol layout tree and operator tree representations. The second model uses tree-edit distance to re-rank the results from the first model. Finally, a learning-to-rank model that leverages full-tree, sub-tree, and vector similarity scores is introduced. To use context, Math Abstract Meaning Representation (MathAMR) is introduced, which generalizes AMR trees to include math formula operations and arguments. This MathAMR is then used for contextualized formula search using a fine-tuned Sentence-BERT model. The experiments show tree-edit distance ranking achieves the current state-of-the-art results on contextual formula search task, and the MathAMR model can be beneficial for re-ranking. This research also addresses the answer retrieval task, introducing a two-step retrieval model in which similar questions are first found and then answers previously given to those similar questions are ranked. The proposed model, fine-tunes two Sentence-BERT models, one for finding similar questions and another one for ranking the answers. For Sentence-BERT model, raw text as well as MathAMR are used