Predicting Comprehension from Students’ Summaries

International audienceComprehension among young students represents a key component of their formation throughout the learning process. Moreover, scaffolding students as they learn to coherently link information, while organically construct- ing a solid knowledge base, is crucial to students’ development, but requires regular assessment and progress tracking. To this end, our aim is to provide an automated solution for analyzing and predicting students’ comprehension levels by extracting a combination of reading strategies and textual complexity factors from students’ summaries. Building upon previous research and enhancing it by incorporating new heuristics and factors, Support Vector Machine classification models were used to validate our assumptions that automatically identified reading strategies, together with textual complexity indices applied on students’ summaries, represent reliable estimators of comprehension

iSTART: Interactive Strategy Training for Active Reading and Thinking

Interactive Strategy Training for Active Reading and Thinking (iSTART) is a Web-based application that provides young adolescent to college-age students with high-level reading strategy training to improve comprehension of science texts. iSTART is modeled after an effective, human-delivered intervention called self-explanation reading training (SERT), which trains readers to use active reading strategies to self-explain difficult texts more effectively. To make the training more widely available, the Web-based trainer has been developed. Transforming the training from a human-delivered application to a computer-based one has resulted in a highly interactive trainer that adapts its methods to the performance of the students. The iSTART trainer introduces the strategies in a simulated classroom setting with interaction between three animated characters—an instructor character and two student characters— and the human trainee. Thereafter, the trainee identifies the strategies in the explanations of a student character who is guided by an instructor character. Finally, the trainee practices self-explanation under the guidance of an instructor character. We describe this system and discuss how appropriate feedback is generated

SMAN : Stacked Multi-Modal Attention Network for cross-modal image-text retrieval

This article focuses on tackling the task of the cross-modal image-text retrieval which has been an interdisciplinary topic in both computer vision and natural language processing communities. Existing global representation alignment-based methods fail to pinpoint the semantically meaningful portion of images and texts, while the local representation alignment schemes suffer from the huge computational burden for aggregating the similarity of visual fragments and textual words exhaustively. In this article, we propose a stacked multimodal attention network (SMAN) that makes use of the stacked multimodal attention mechanism to exploit the fine-grained interdependencies between image and text, thereby mapping the aggregation of attentive fragments into a common space for measuring cross-modal similarity. Specifically, we sequentially employ intramodal information and multimodal information as guidance to perform multiple-step attention reasoning so that the fine-grained correlation between image and text can be modeled. As a consequence, we are capable of discovering the semantically meaningful visual regions or words in a sentence which contributes to measuring the cross-modal similarity in a more precise manner. Moreover, we present a novel bidirectional ranking loss that enforces the distance among pairwise multimodal instances to be closer. Doing so allows us to make full use of pairwise supervised information to preserve the manifold structure of heterogeneous pairwise data. Extensive experiments on two benchmark datasets demonstrate that our SMAN consistently yields competitive performance compared to state-of-the-art methods

Component skills of inferential processing in older readers

Thesis (M.S.)--Boston UniversityThe ability to make inferences has been shown to be a crucial component of successful reading in older students. The current project investigates differences in comprehension of text-based (factual) and inferential information across grade levels and modalities, and seeks to determine which component language and reading skills that are important in making inferences. 1,836 students in grades 6-12 were tested on a computerized battery of language subtests in the auditory and written modalities. Eleven subtests examining performance on lower levels of were administered in addition to a measure of factual and inferential discourse comprehension. Results demonstrated that students performed better overall in the written modality. Students in older grades were consistently faster and more accurate. Vocabulary knowledge had the biggest effect for performance on inferential questions in the written modality in middle school, while sentence-level skills were most important in high school. In the auditory modality, sentence-level skills were most predictive across question types and grade levels. Implications for theories of inferential processing and for teaching inferences within literacy education frameworks will be discussed

딥 뉴럴 네트워크 기반의 문장 인코더를 이용한 문장 간 관계 모델링

학위논문(박사)--서울대학교 대학원 :공과대학 컴퓨터공학부,2020. 2. 이상구.문장 매칭이란 두 문장 간 의미적으로 일치하는 정도를 예측하는 문제이다. 어떤 모델이 두 문장 사이의 관계를 효과적으로 밝혀내기 위해서는 높은 수준의 자연어 텍스트 이해 능력이 필요하기 때문에, 문장 매칭은 다양한 자연어 처리 응용의 성능에 중요한 영향을 미친다. 본 학위 논문에서는 문장 인코더, 매칭 함수, 준지도 학습이라는 세 가지 측면에서 문장 매칭의 성능 개선을 모색한다. 문장 인코더란 문장으로부터 유용한 특질들을 추출하는 역할을 하는 구성 요소로, 본 논문에서는 문장 인코더의 성능 향상을 위하여 Gumbel Tree-LSTM과 Cell-aware Stacked LSTM이라는 두 개의 새로운 아키텍처를 제안한다. Gumbel Tree-LSTM은 재귀적 뉴럴 네트워크(recursive neural network) 구조에 기반한 아키텍처이다. 구조 정보가 포함된 데이터를 입력으로 사용하던 기존의 재귀적 뉴럴 네트워크 모델과 달리, Gumbel Tree-LSTM은 구조가 없는 데이터로부터 특정 문제에 대한 성능을 최대화하는 파싱 전략을 학습한다. Cell-aware Stacked LSTM은 LSTM 구조를 개선한 아키텍처로, 여러 LSTM 레이어를 중첩하여 사용할 때 망각 게이트(forget gate)를 추가적으로 도입하여 수직 방향의 정보 흐름을 더 효율적으로 제어할 수 있도록 한다. 한편, 새로운 매칭 함수로서 우리는 요소별 쌍선형 문장 매칭(element-wise bilinear sentence matching, ElBiS) 함수를 제안한다. ElBiS 알고리즘은 특정 문제를 해결하는 데에 적합한 방식으로 두 문장 표현을 하나의 벡터로 합치는 방법을 자동으로 찾는 것을 목적으로 한다. 문장 표현을 얻을 때에 서로 같은 문장 인코더를 사용한다는 사실로부터 우리는 벡터의 각 요소 간 쌍선형(bilinear) 상호 작용만을 고려하여도 두 문장 벡터 간 비교를 충분히 잘 수행할 수 있다는 가설을 수립하고 이를 실험적으로 검증한다. 상호 작용의 범위를 제한함으로써, 자동으로 유용한 병합 방법을 찾는다는 이점을 유지하면서 모든 상호 작용을 고려하는 쌍선형 풀링 방법에 비해 필요한 파라미터의 수를 크게 줄일 수 있다. 마지막으로, 학습 시 레이블이 있는 데이터와 레이블이 없는 데이터를 함께 사용하는 준지도 학습을 위해 우리는 교차 문장 잠재 변수 모델(cross-sentence latent variable model, CS-LVM)을 제안한다. CS-LVM의 생성 모델은 출처 문장(source sentence)의 잠재 표현 및 출처 문장과 목표 문장(target sentence) 간의 관계를 나타내는 변수로부터 목표 문장이 생성된다고 가정한다. CS-LVM에서는 두 문장이 하나의 모델 안에서 모두 고려되기 때문에, 학습에 사용되는 목적 함수가 더 자연스럽게 정의된다. 또한, 우리는 생성 모델의 파라미터가 더 의미적으로 적합한 문장을 생성하도록 유도하기 위하여 일련의 의미 제약들을 정의한다. 본 학위 논문에서 제안된 개선 방안들은 문장 매칭 과정을 포함하는 다양한 자연어 처리 응용의 효용성을 높일 것으로 기대된다.Sentence matching is a task of predicting the degree of match between two sentences. Since high level of understanding natural language text is needed for a model to identify the relationship between two sentences, it is an important component for various natural language processing applications. In this dissertation, we seek for the improvement of the sentence matching module from the following three ingredients: sentence encoder, matching function, and semi-supervised learning. To enhance a sentence encoder network which takes responsibility of extracting useful features from a sentence, we propose two new sentence encoder architectures: Gumbel Tree-LSTM and Cell-aware Stacked LSTM (CAS-LSTM). Gumbel Tree-LSTM is based on a recursive neural network (RvNN) architecture, however unlike typical RvNN architectures it does not need a structured input. Instead, it learns from data a parsing strategy that is optimized for a specific task. The latter, CAS-LSTM, extends the stacked long short-term memory (LSTM) architecture by introducing an additional forget gate for better handling of vertical information flow. And then, as a new matching function, we present the element-wise bilinear sentence matching (ElBiS) function. It aims to automatically find an aggregation scheme that fuses two sentence representations into a single one suitable for a specific task. From the fact that a sentence encoder is shared across inputs, we hypothesize and empirically prove that considering only the element-wise bilinear interaction is sufficient for comparing two sentence vectors. By restricting the interaction, we can largely reduce the number of required parameters compared with full bilinear pooling methods without losing the advantage of automatically discovering useful aggregation schemes. Finally, to facilitate semi-supervised training, i.e. to make use of both labeled and unlabeled data in training, we propose the cross-sentence latent variable model (CS-LVM). Its generative model assumes that a target sentence is generated from the latent representation of a source sentence and the variable indicating the relationship between the source and the target sentence. As it considers the two sentences in a pair together in a single model, the training objectives are defined more naturally than prior approaches based on the variational auto-encoder (VAE). We also define semantic constraints that force the generator to generate semantically more plausible sentences. We believe that the improvements proposed in this dissertation would advance the effectiveness of various natural language processing applications containing modeling sentence pairs.Chapter 1 Introduction 1 1.1 Sentence Matching 1 1.2 Deep Neural Networks for Sentence Matching 2 1.3 Scope of the Dissertation 4 Chapter 2 Background and Related Work 9 2.1 Sentence Encoders 9 2.2 Matching Functions 11 2.3 Semi-Supervised Training 13 Chapter 3 Sentence Encoder: Gumbel Tree-LSTM 15 3.1 Motivation 15 3.2 Preliminaries 16 3.2.1 Recursive Neural Networks 16 3.2.2 Training RvNNs without Tree Information 17 3.3 Model Description 19 3.3.1 Tree-LSTM 19 3.3.2 Gumbel-Softmax 20 3.3.3 Gumbel Tree-LSTM 22 3.4 Implementation Details 25 3.5 Experiments 27 3.5.1 Natural Language Inference 27 3.5.2 Sentiment Analysis 32 3.5.3 Qualitative Analysis 33 3.6 Summary 36 Chapter 4 Sentence Encoder: Cell-aware Stacked LSTM 38 4.1 Motivation 38 4.2 Related Work 40 4.3 Model Description 43 4.3.1 Stacked LSTMs 43 4.3.2 Cell-aware Stacked LSTMs 44 4.3.3 Sentence Encoders 46 4.4 Experiments 47 4.4.1 Natural Language Inference 47 4.4.2 Paraphrase Identification 50 4.4.3 Sentiment Classification 52 4.4.4 Machine Translation 53 4.4.5 Forget Gate Analysis 55 4.4.6 Model Variations 56 4.5 Summary 59 Chapter 5 Matching Function: Element-wise Bilinear Sentence Matching 60 5.1 Motivation 60 5.2 Proposed Method: ElBiS 61 5.3 Experiments 63 5.3.1 Natural language inference 64 5.3.2 Paraphrase Identification 66 5.4 Summary and Discussion 68 Chapter 6 Semi-Supervised Training: Cross-Sentence Latent Variable Model 70 6.1 Motivation 70 6.2 Preliminaries 71 6.2.1 Variational Auto-Encoders 71 6.2.2 von Mises–Fisher Distribution 73 6.3 Proposed Framework: CS-LVM 74 6.3.1 Cross-Sentence Latent Variable Model 75 6.3.2 Architecture 78 6.3.3 Optimization 79 6.4 Experiments 84 6.4.1 Natural Language Inference 84 6.4.2 Paraphrase Identification 85 6.4.3 Ablation Study 86 6.4.4 Generated Sentences 88 6.4.5 Implementation Details 89 6.5 Summary and Discussion 90 Chapter 7 Conclusion 92 Appendix A Appendix 96 A.1 Sentences Generated from CS-LVM 96Docto

Bridging Cross-Modal Alignment for OCR-Free Content Retrieval in Scanned Historical Documents

In this work, we address the limitations of current approaches to document retrieval by incorporating vision-based topic extraction. While previous methods have primarily focused on visual elements or relied on optical character recognition (OCR) for text extraction, we propose a paradigm shift by directly incorporating vision into the topic space. We demonstrate that recognizing all visual elements within a document is unnecessary for identifying its underlying topic. Visual cues such as icons, writing style, and font can serve as sufficient indicators. By leveraging ranking loss functions and convolutional neural networks (CNNs), we learn complex topological representations that mimic the behavior of text representations. Our approach aims to eliminate the need for OCR and its associated challenges, including efficiency, performance, data-hunger, and expensive annotation. Furthermore, we highlight the significance of incorporating vision in historical documentation, where visually antiquated documents contain valuable cues. Our research contributes to the understanding of topic extraction from a vision perspective and offers insights into annotation-cheap document retrieval system

Tekstimõistmise hindamine ja tekstimõistmist toetavate strateegiate õpetamine Eesti põhikoolis

Väitekirja elektrooniline versioon ei sisalda publikatsiooneTekstidest arusaamine võimaldab omandada uusi teadmisi ja igapäevaelus edukalt hakkama saada. Tekstimõistmine on mitmetasandiline ja sisaldab paljude omavahel seotud komponentide ja protsesside samaaegset kasutamist. Selleks, et jälgida õpilaste arengut tekstimõistmise eri oskustes, võib jaotada tekstimõistmise kolmele tasandile: sõnasõnalisele, järeldavale ja hindavale. Tekstimõistmine järeldaval ja hindaval tasandil eeldab lugejalt tekstimõistmise strateegiate kasutamist, mida on tarvis õpilastele enamasti lugemistundides õpetada. Siinses doktoritöös uuriti, kuivõrd eesti keele taseme- ja eksamitööde tekstimõistmise ülesanded mõõdavad õpilaste loetust arusaamist eri tasanditel, ning töötati välja sekkumisprogramm tekstimõistmise strateegiate õpetamiseks emakeeletundides. Uuringust ilmnes, et üleriigilised hindamisvahendid sisaldavad palju õpilaste faktiteadmisi kontrollivaid ülesandeid ning vähe pööratakse tähelepanu hindava tasandi tekstimõistmise mõõtmisele. Lisaks selgus, et sama klassi ülesannete tasandiline jaotus oli aastati erinev, näiteks keskenduti ühel aastal peamiselt sõnasõnalise tasandi, kuid järgneval aastal järeldava tasandi ülesannetele. Tekstimõistmise ülesannete tasandilist jaotust arvestati sekkumisprogrammi koostamisel. Selle programmi efektiivsuse kontrollimisel selgus, et tekstimõistmise strateegiate õpetamine suurendas oluliselt õpilaste sõnavara ja tekstimõistmist sõnasõnalisel, järeldaval ja hindaval tasandil. Seevastu kontrollrühmas, kus strateegiaid ei õpetatud, arenes olulisel määral vaid õpilaste sõnasõnalise tekstimõistmise oskus. Doktoritöö tulemused toovad välja vajaduse muuta üleriigiliste hindamisvahendite koostamise põhimõtteid kooskõlas tekstimõistmise teooriatega. Lisaks kinnitavad tulemused tekstimõistmise strateegiate olulisust tekstimõistmise arendamisel. Sel põhjusel tuleks lisada strateegiate õpetamine riiklikku õppekavva ning emakeeletundidesse ning pakkuda õpetajatele teadmisi sellest, kuidas tekstimõistmise strateegiaid õpilastele eesmärgipäraselt õpetadaText comprehension enables to convey knowledge in school setting and in every-day life. Text comprehension involves a full range of interactively working processes and components to understand texts at various levels. To have an overview of students’ proficiency in these processes, the text comprehension can be transferred to the three levels: literal, inferential and evaluative. The processes at inferential and evaluative levels rely on various comprehension strategies that should be explicitly taught in reading lessons. In this doctoral study the comprehension levels among the text comprehension tasks in the national standard-determining tests were examined, and an intervention program for teaching comprehension strategies was developed. It appeared that national assessments tended to include too many literal-level tasks and few tasks at evaluative comprehension level. Also, distribution of tasks at different comprehension levels was fluctuating among the tests for the same grade in different years. Multidimensional nature of text comprehension was considered in developing intervention program. The results showed that explicit teaching of text comprehension strategies enhanced students’ vocabulary and text comprehension at literal, inferential and evaluative levels. However, the students who attended the regular reading classes without focused attention on teaching of comprehension strategies increased only their literal text comprehension. This doctoral study highlights the need for improving the compilation of national assessments that consider the theories of text comprehension. Further, the results confirm the importance of teaching explicitly various comprehension strategies in order to improve students’ text comprehension. For that reason, the strategy teaching should be a valued component in curriculum and reading lessons, and teachers should be provided the knowledge of how to purposefully teach text comprehension strategies in reading lessons.https://www.ester.ee/record=b536057

Exploring Reading Skills and Strategies Among Struggling Postsecondary Readers

Many students enter college underprepared to meet the literacy demands they encounter. There are calls for cognitively oriented research aimed at understanding the strengths and challenges of these readers, especially those enrolled in developmental education courses designed to improve literacy skills. The purpose of this dissertation was to better understand the basis of the difficulties faced by struggling college readers. In chapter one, the Reading Systems Framework (RFS; Perfetti & Stafure, 2014) was utilized to examine prior research on struggling college readers and accordingly, research related to word identification, lexical processes, and higher-level comprehension strategies was explored. Additionally, literature exploring complex, interactive relations between reading systems was explored. The review illustrates the utility of the RSF to understand struggling college readers and identifies areas where more research is needed. Chapter two presents a study that examined the relations among proficiency in component reading skills, one’s propensity to engage reading strategies, and enrollment in DE courses. Participants (N = 258) completed a measure of component reading skills (word recognition/decoding, vocabulary, morphology, sentence processing) as well as a think-aloud measure, wherein they produced written responses while reading texts. Responses were scored based on evidence of reading strategies (paraphrasing, bridging, and elaboration) and their overall quality in supporting comprehension. Logistic regression was used to assess the extent to which one’s proficiency in component reading skills and use of reading strategies could be utilized to predict whether participants were enrolled in DE courses. Results indicated that proficiency in reading skills was related to enrollment in DE courses but that the use of reading strategies was not. Cumulative links mixed effects models were used to assess the extent to which proficiency in component reading skills and DE enrollment were differentially related to the use of reading strategies and the overall quality of participant’s responses. Results indicated that vocabulary was a positive predictor of bridging and elaboration scores. Moreover, vocabulary and word recognition/decoding positively predicted the overall quality of responses. DE enrollment was a negative predictor of elaboration scores, suggesting that DE readers were less likely to produce elaborations. Implications for theory and practice are discussed

A Survey on Semantic Processing Techniques

Semantic processing is a fundamental research domain in computational linguistics. In the era of powerful pre-trained language models and large language models, the advancement of research in this domain appears to be decelerating. However, the study of semantics is multi-dimensional in linguistics. The research depth and breadth of computational semantic processing can be largely improved with new technologies. In this survey, we analyzed five semantic processing tasks, e.g., word sense disambiguation, anaphora resolution, named entity recognition, concept extraction, and subjectivity detection. We study relevant theoretical research in these fields, advanced methods, and downstream applications. We connect the surveyed tasks with downstream applications because this may inspire future scholars to fuse these low-level semantic processing tasks with high-level natural language processing tasks. The review of theoretical research may also inspire new tasks and technologies in the semantic processing domain. Finally, we compare the different semantic processing techniques and summarize their technical trends, application trends, and future directions.Comment: Published at Information Fusion, Volume 101, 2024, 101988, ISSN 1566-2535. The equal contribution mark is missed in the published version due to the publication policies. Please contact Prof. Erik Cambria for detail