Technology Assisted Reviews: Finding the Last Few Relevant Documents by Asking Yes/No Questions to Reviewers

The goal of a technology-assisted review is to achieve high recall with low human effort. Continuous active learning algorithms have demonstrated good performance in locating the majority of relevant documents in a collection, however their performance is reaching a plateau when 80\%-90\% of them has been found. Finding the last few relevant documents typically requires exhaustively reviewing the collection. In this paper, we propose a novel method to identify these last few, but significant, documents efficiently. Our method makes the hypothesis that entities carry vital information in documents, and that reviewers can answer questions about the presence or absence of an entity in the missing relevance documents. Based on this we devise a sequential Bayesian search method that selects the optimal sequence of questions to ask. The experimental results show that our proposed method can greatly improve performance requiring less reviewing effort.Comment: This paper is accepted by SIGIR 201

Learning from rediscovering system dynamics models

This article deals with learning from the exploration of system dynamics models. System dynamics modeling intends to improve judgment and decision, but is very time consuming. Model-based interactive learning environments allow saving time, but critics doubt the effectiveness for deep learning. The question is if there is a third way in-between. Relevant examples from system dynamics are analyzed to identify the key activities that trigger learning; they are organized as a structured exploration process, making learners ask relevant questions, obtain valid responses and correctly interpret them. Based upon this, a process for guided rediscovery is proposed together with guidelines for the functional properties of a “systemic exploratory”. Guided rediscovery enables non-specialists to gain relevant insights into dynamically complex situations and is a tool for decision policy design

Question Driven Introductory Programming Instruction: A Pilot Study

For most beginners, learning computer programming is a complex undertaking. Demotivation and learned helplessness have been widely reported. In addition to the subject’s complexity, low in-class involvement has been linked to poor student performance. This work introduces a novel instructional technique called Student-Driven Probe Instruction (SDPI) to address the low levels of in-class involvement in basic programming courses. The concept was straightforward: rather than the teacher lecturing/explaining material to the class and requesting questions, the students were shown a piece of code or other relevant material and given the opportunity to ask questions first. Explanations followed only after the questions had been asked, not before. Participation was tracked through two metrics: the number of questions asked in class and emails/Slack contacts with the instructor. Significant improvements were recorded for in-class participation. Average quiz scores also improved meaningfully. According to a course evaluation survey, students favored SDPI over the conventional lecture format since it piqued their interest in the material and gave them the confidence to ask questions in class

Probing for requirements knowledge to stimulate architectural thinking

Software requirements specifications (SRSs) often lack the detail needed to make informed architectural decisions. Architects therefore either make assumptions, which can lead to incorrect decisions, or conduct additional stakeholder interviews, resulting in potential project delays. We previously observed that software architects ask Probing Questions (PQs) to gather information crucial to architectural decision-making. Our goal is to equip Business Analysts with appropriate PQs so that they can ask these questions themselves. We report a new study with over 40 experienced architects to identify reusable PQs for five areas of functionality and organize them into structured flows. These PQflows can be used by Business Analysts to elicit and specify architecturally relevant information. Additionally, we leverage machine learning techniques to determine when a PQ-flow is appropriate for use in a project, and to annotate individual PQs with relevant information extracted from the existing SRS. We trained and evaluated our approach on over 8,000 individual requirements from 114 requirements specifications and also conducted a pilot study to validate its usefulness.</p

A Questioning Agent for Literary Discussion

Developing a compelling and cohesive thesis for analytical writing can be a daunting task, even for those who have produced many written works, and finding others to engage with in literary discussion can be equally challenging. In this paper, we describe our solution: Questioner, a discussion tool that engages users in conversation about an academic topic of their choosing for the purpose of collecting thoughts on a subject and constructing an argument. This system will ask informed questions that prompt further discussion about the topic and provide a discussion report after the conversation has ended. We found that our system is effective in providing users with unique questions and excerpts that are relevant, significant, and engaging. Such a discussion tool can be used by writers building theses, students looking for study tools, and instructors who want to create individualized in-class discussions. Once more data is gathered, efficient and accurate machine learning models can be used to further improve the quality of question and excerpt recommendations. Co-creative discussion tools like Questioner are useful in assisting users in developing critical analyses of written works, helping to maximize human creativity

GP trainees’ perceptions on learning EBM using conversations in the workplace : a video-stimulated interview study

Background To be able to practice evidence-based medicine (EBM) when making decisions for individual patients, it is important to learn how to combine the best available evidence with the patient's preferences and the physician's clinical expertise. In general practice training, these skills can be learned at the workplace using learning conversations: meetings between the supervising general practitioner (GP) and GP trainee to discuss medical practice, selected topics or professional performance. This study aimed to give insight into the perceptions of GP trainees on their EBM learning processes during learning conversations. Methods We held semi-structured video-stimulated elicitation interviews (n = 22) with GP trainees affiliated to GP training institutes in the Netherlands and Belgium. GP trainees were shown fragments of their learning conversations, enabling reflection during the interview. Taking an inductive approach, interview recordings were transcribed verbatim and analysed with NVivo software. Results GP trainees perceived learning conversations as useful for learning and discussing EBM. Multiple EBM learning activities were identified, such as discussing evidence together, relating evidence to cases in daily practice and discussing the supervisor's experience and the specific local context in the light of what the evidence recommends. However, for learning to occur, trainees need and expect specific behaviour, both from their supervisors and themselves. Supervisors should supply well-substantiated answers that are applicable in practice and give the trainee confirmation. In turn, the trainee needs to prepare well in order to ask focused, in-depth questions. A safe space allowing equal and open discussion between trainee and supervisor is perceived as an essential context for optimal EBM learning. Conclusions Our findings show that trainees find learning conversations useful for EBM learning in general practice. To bring EBM learning to its full potential, attention should be paid to optimising the behavioural and contextual factors found relevant to enhancing EBM learning

Mexican American Parents of Elementary Students and Literacy Engagement: A Case Study of a Bilingual (Spanish/English) Parent Book Club Using Children\u27s Literature

This qualitative case study explored what occurred when seven Latinx parents and a bilingual coordinator engaged in a book club using the same historical fiction text as the school\u27s fifth grade classroom. Research questions focused on Book Club participation, connections to the focal text(s), and new learning. Data included audiotapes of the sessions, parent written/artistic artifacts, participant interviews, retrospective field notes, and reflective journal entries. Study findings suggest that positive partnerships between schools and parents can occur within familial text engagement opportunities built on existing relationships with school personnel; use of relevant literature, literacy activities that allow for parent choice and voice, drawing on cultural Funds of Knowledge and life experiences, the autonomy to ask questions and make inferences and connections, and facilitate learning for one another. Finally, the study improved perceptions that parents could better assist their children in learning due to new learning in literacy instruction and reading

Developing a Semantic Question Answering System for E-learning Environments using Linguistic Resources

The Question answering (QA) system plays a basic role in the acquisition of information and the e-learning environment is considered to be the field that is most in need of the question-answering system to help learners ask questions in natural language and get answers in short periods of time. The main problem in this context is how to understand the questions without any doubts in meaning and how to provide the most relevant answers to the questions. In this study, a question-answering system for specific courses has been developed to support the learning environment. The research outcomes indicate that the proposed method helps to solve the problem of ambiguities in meaning through the integration of natural language processing tools and semantic resources that can help to overcome several problems related to the natural language structure. This method also helps improve the capability to understand students’ needs and, consequently, to retrieve the most suitable answers

Poetic Research in the Second Language Classroom

Questions in second language teaching ask if literature is relevant in acquiring a language and if so how do students perceive literary works. The motivation of this study to investigate a small diverse group of international students is to understand how they perceive literature and all genres including poetry. This study conducts a Likert numerical survey to find what literary background students experience in their native country and what attitudes students have learning English. Analysis shows that students overall have positive L1 and L2 learning attitudes regarding literature and culture. Interview questions provide a more reflective understanding of students’ experiences. Pedagogical implications include models for incorporating poetry in lesson plans and how poetry lends itself to digital applications. These models can help develop the four language skills but equally important create cultural awareness and self-expressio

An empirical study of the “prototype walkthrough”: a studio-based activity for HCI education

For over a century, studio-based instruction has served as an effective pedagogical model in architecture and fine arts education. Because of its design orientation, human-computer interaction (HCI) education is an excellent venue for studio-based instruction. In an HCI course, we have been exploring a studio-based learning activity called the prototype walkthrough, in which a student project team simulates its evolving user interface prototype while a student audience member acts as a test user. The audience is encouraged to ask questions and provide feedback. We have observed that prototype walkthroughs create excellent conditions for learning about user interface design. In order to better understand the educational value of the activity, we performed a content analysis of a video corpus of 16 prototype walkthroughs held in two HCI courses. We found that the prototype walkthrough discussions were dominated by relevant design issues. Moreover, mirroring the justification behavior of the expert instructor, students justified over 80 percent of their design statements and critiques, with nearly one-quarter of those justifications having a theoretical or empirical basis. Our findings suggest that PWs provide valuable opportunities for students to actively learn HCI design by participating in authentic practice, and provide insight into how such opportunities can be best promoted