Comparing Transformers and RNNs on predicting human sentence processing data

Recurrent neural networks (RNNs) have long been an architecture of interest for computational models of human sentence processing. The more recently introduced Transformer architecture has been shown to outperform recurrent neural networks on many natural language processing tasks but little is known about their ability to model human language processing. It has long been thought that human sentence reading involves something akin to recurrence and so RNNs may still have an advantage over the Transformer as a cognitive model. In this paper we train both Transformer and RNN based language models and compare their performance as a model of human sentence processing. We use the trained language models to compute surprisal values for the stimuli used in several reading experiments and use mixed linear modelling to measure how well the surprisal explains measures of human reading effort. Our analysis shows that the Transformers outperform the RNNs as cognitive models in explaining self-paced reading times and N400 strength but not gaze durations from an eye-tracking experiment

Asynchronous Changes in Vegetation, Runoff and Erosion in the Nile River Watershed during the Holocene

The termination of the African Humid Period in northeastern Africa during the early Holocene was marked by the southward migration of the rain belt and the disappearance of the Green Sahara. This interval of drastic environmental changes was also marked by the initiation of food production by North African huntergatherer populations and thus provides critical information on human-environment relationships. However, existing records of regional climatic and environmental changes exhibit large differences in timing and modes of the wet/dry transition at the end of the African Humid Period. Here we present independent records of changes in river runoff, vegetation and erosion in the Nile River watershed during the Holocene obtained from a unique sedimentary sequence on the Nile River fan using organic and inorganic proxy data. This high-resolution reconstruction allows to examine the phase relationship between the changes of these three parameters and provides a detailed picture of the environmental conditions during the Paleolithic/Neolithic transition. The data show that river runoff decreased gradually during the wet/arid transition at the end of the AHP whereas rapid shifts of vegetation and erosion occurred earlier between 8.7 and ,6 ka BP. These asynchronous changes are compared to other regional records and provide new insights into the threshold responses of the environment to climatic changes. Our record demonstrates that the degradation of the environment in northeastern Africa was more abrupt and occurred earlier than previously thought and may have accelerated the process of domestication in order to secure sustainable food resources for the Neolithic African populations

Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge

Distributional semantic models capture word-level meaning that is useful in many natural language processing tasks and have even been shown to capture cognitive aspects of word meaning. The majority of these models are purely text based, even though the human sensory experience is much richer. In this paper we create visually grounded word embeddings by combining English text and images and compare them to popular text-based methods, to see if visual information allows our model to better capture cognitive aspects of word meaning. Our analysis shows that visually grounded embedding similarities are more predictive of the human reaction times in a large priming experiment than the purely text-based embeddings. The visually grounded embeddings also correlate well with human word similarity ratings. Importantly, in both experiments we show that the grounded embeddings account for a unique portion of explained variance, even when we include text-based embeddings trained on huge corpora. This shows that visual grounding allows our model to capture information that cannot be extracted using text as the only source of information

Semantic sentence similarity: size does not always matter

This study addresses the question whether visually grounded speech recognition (VGS) models learn to capture sentence semantics without access to any prior linguistic knowledge. We produce synthetic and natural spoken versions of a well known semantic textual similarity database and show that our VGS model produces embeddings that correlate well with human semantic similarity judgements. Our results show that a model trained on a small image-caption database outperforms two models trained on much larger databases, indicating that database size is not all that matters. We also investigate the importance of having multiple captions per image and find that this is indeed helpful even if the total number of images is lower, suggesting that paraphrasing is a valuable learning signal. While the general trend in the field is to create ever larger datasets to train models on, our findings indicate other characteristics of the database can just as important important.Comment: This paper has been accepted at Interspeech 2021 where it will be presented and appear in the conference proceedings in September 202

The Learnability of the Wh-Island Constraint in Dutch by a Long Short-Term Memory Network

The current study investigates whether a Long Short-Term Memory (LSTM) network can learn the wh-island constraint in Dutch in a way comparable to human native speakers. After establishing with an acceptability judgement task that native speakers demonstrate a clear sensitivity to wh-island violations, the LSTM network was tested on the same sentences. Contrary to the results of the native speakers, the network was not able to recognize wh-islands and to block gap expectancies within them. This suggests that input and the network’s inductive biases alone might not be enough to learn about syntactic island constraints, and that built-in language knowledge or abilities might be necessary

The missing-VP effect in readers of English as a second language

English sentences with double center-embedded clauses are read faster when they are made ungrammatical by removing one of the required verb phrases. This phenomenon is known as the missing-VP effect. German and Dutch speakers do not experience the missing-VP effect when reading their native language, but they do when reading English as a second language (L2). We investigate whether the missing-VP effect when reading L2 English occurs in native Dutch speakers because their knowledge of English is similar to that of native English speakers (the high exposure account), or because of the difficulty of L2 reading (the low proficiency account). In an eye-tracking study, we compare the size of the missing-VP effect between native Dutch and native English participants, and across native Dutch participants with varying L2 English proficiency and exposure. Results provide evidence for both accounts, suggesting that both native-like knowledge of English and L2 reading difficulty play a role

Nanostructure-modulated planar high spectral resolution spectro-polarimeter

We present a planar spectro-polarimeter based on Fabry-P{\'e}rot cavities with embedded polarization-sensitive high-index nanostructures. A $7~\mu$m-thick spectro-polarimetric system for 3 spectral bands and 2 linear polarization states is experimentally demonstrated. Furthermore, an optimal design is theoretically proposed, estimating that a system with a bandwidth of 127~nm and a spectral resolution of 1~nm is able to reconstruct the first three Stokes parameters \textcolor{black}{with a signal-to-noise ratio of -13.14~dB with respect to the the shot noise limited SNR}. The pixelated spectro-polarimetric system can be directly integrated on a sensor, thus enabling applicability in a variety of miniaturized optical devices, including but not limited to satellites for Earth observation

UvA-DARE (Digital Academic Repository) Generalization and Systematicity in Echo State Networks

Abstract Echo state networks (ESNs) are recurrent neural networks that can be trained efficiently because the weights of recurrent connections remain fixed at random values. Investigations of these networks' ability to generalize in sentence-processing tasks have resulted in mixed outcomes. Here, we argue that ESNs do generalize but that they are not systematic, which we define as the ability to generally outperform Markov models on test sentences that violate the training sentences' grammar. Moreover, we show that systematicity in ESNs can easily be obtained by switching from arbitrary to informative representations of words, suggesting that the information provided by such representations facilitates connectionist systematicity

The metabolic vascular syndrome - guide to an individualized treatment

In ancient Greek medicine the concept of a distinct syndrome (going together) was used to label 'a group of signs and symptoms' that occur together and 'characterize a particular abnormality and condition'. The (dys)metabolic syndrome is a common cluster of five pre-morbid metabolic-vascular risk factors or diseases associated with increased cardiovascular morbidity, fatty liver disease and risk of cancer. The risk for major complications such as cardiovascular diseases, NASH and some cancers develops along a continuum of risk factors into clinical diseases. Therefore we still include hyperglycemia, visceral obesity, dyslipidemia and hypertension as diagnostic traits in the definition according to the term 'deadly quartet'. From the beginning elevated blood pressure and hyperglycemia were core traits of the metabolic syndrome associated with endothelial dysfunction and increased risk of cardiovascular disease. Thus metabolic and vascular abnormalities are in extricable linked. Therefore it seems reasonable to extend the term to metabolic-vascular syndrome (MVS) to signal the clinical relevance and related risk of multimorbidity. This has important implications for integrated diagnostics and therapeutic approach. According to the definition of a syndrome the rapid global rise in the prevalence of all traits and comorbidities of the MVS is mainly caused by rapid changes in life-style and sociocultural transition resp. with over- and malnutrition, low physical activity and social stress as a common soil

Unconditional cash transfers for reducing poverty and vulnerabilities: effect on use of health services and health outcomes in low-and middle-income countries

Background Unconditional cash transfers (UCTs; provided without obligation) for reducing poverty and vulnerabilities (e.g. orphanhood, old age or HIV infection) are a type of social protection intervention that addresses a key social determinant of health (income) in low‐ and middle‐income countries (LMICs). The relative effectiveness of UCTs compared with conditional cash transfers (CCTs; provided so long as the recipient engages in prescribed behaviours such as using a health service or attending school) is unknown. Objectives To assess the effects of UCTs for improving health services use and health outcomes in vulnerable children and adults in LMICs. Secondary objectives are to assess the effects of UCTs on social determinants of health and healthcare expenditure and to compare to effects of UCTs versus CCTs. Search methods We searched 17 electronic academic databases, including the Cochrane Public Health Group Specialised Register, the Cochrane Database of Systematic Reviews (the Cochrane Library 2017, Issue 5), MEDLINE and Embase, in May 2017. We also searched six electronic grey literature databases and websites of key organisations, handsearched key journals and included records, and sought expert advice. Selection criteria We included both parallel group and cluster‐randomised controlled trials (RCTs), quasi‐RCTs, cohort and controlled before‐and‐after (CBAs) studies, and interrupted time series studies of UCT interventions in children (0 to 17 years) and adults (18 years or older) in LMICs. Comparison groups received either no UCT or a smaller UCT. Our primary outcomes were any health services use or health outcome. Data collection and analysis Two reviewers independently screened potentially relevant records for inclusion criteria, extracted data and assessed the risk of bias. We tried to obtain missing data from study authors if feasible. For cluster‐RCTs, we generally calculated risk ratios for dichotomous outcomes from crude frequency measures in approximately correct analyses. Meta‐analyses applied the inverse variance or Mantel‐Haenszel method with random effects. We assessed the quality of evidence using the GRADE approach. Main results We included 21 studies (16 cluster‐RCTs, 4 CBAs and 1 cohort study) involving 1,092,877 participants (36,068 children and 1,056,809 adults) and 31,865 households in Africa, the Americas and South‐East Asia in our meta‐analyses and narrative synthesis. The 17 types of UCTs we identified, including one basic universal income intervention, were pilot or established government programmes or research experiments. The cash value was equivalent to 1.3% to 53.9% of the annualised gross domestic product per capita. All studies compared a UCT with no UCT, and three studies also compared a UCT with a CCT. Most studies carried an overall high risk of bias (i.e. often selection and/or performance bias). Most studies were funded by national governments and/or international organisations. Throughout the review, we use the words \u27probably\u27 to indicate moderate‐quality evidence, \u27may/maybe\u27 for low‐quality evidence, and \u27uncertain\u27 for very low‐quality evidence. UCTs may not have impacted the likelihood of having used any health service in the previous 1 to 12 months, when participants were followed up between 12 and 24 months into the intervention (risk ratio (RR) 1.04, 95% confidence interval (CI) 1.00 to 1.09, P = 0.07, 5 cluster‐RCTs, N = 4972, I² = 2%, low‐quality evidence). At one to two years, UCTs probably led to a clinically meaningful, very large reduction in the likelihood of having had any illness in the previous two weeks to three months (odds ratio (OR) 0.73, 95% CI 0.57 to 0.93, 5 cluster‐RCTs, N = 8446, I² = 57%, moderate‐quality evidence). Evidence from five cluster‐RCTs on food security was too inconsistent to be combined in a meta‐analysis, but it suggested that at 13 to 24 months\u27 follow‐up, UCTs could increase the likelihood of having been food secure over the previous month (low‐quality evidence). UCTs may have increased participants\u27 level of dietary diversity over the previous week, when assessed with the Household Dietary Diversity Score and followed up 24 months into the intervention (mean difference (MD) 0.59 food categories, 95% CI 0.18 to 1.01, 4 cluster‐RCTs, N = 9347, I² = 79%, low‐quality evidence). Despite several studies providing relevant evidence, the effects of UCTs on the likelihood of being moderately stunted and on the level of depression remain uncertain. No evidence was available on the effect of a UCT on the likelihood of having died. UCTs probably led to a clinically meaningful, moderate increase in the likelihood of currently attending school, when assessed at 12 to 24 months into the intervention (RR 1.06, 95% CI 1.03 to 1.09, 6 cluster‐RCTs, N = 4800, I² = 0%, moderate‐quality evidence). The evidence was uncertain for whether UCTs impacted livestock ownership, extreme poverty, participation in child labour, adult employment or parenting quality. Evidence from six cluster‐RCTs on healthcare expenditure was too inconsistent to be combined in a meta‐analysis, but it suggested that UCTs may have increased the amount of money spent on health care at 7 to 24 months into the intervention (low‐quality evidence). The effects of UCTs on health equity (or unfair and remedial health inequalities) were very uncertain. We did not identify any harms from UCTs. Three cluster‐RCTs compared UCTs versus CCTs with regard to the likelihood of having used any health services, the likelihood of having had any illness or the level of dietary diversity, but evidence was limited to one study per outcome and was very uncertain for all three. Authors\u27 conclusions This body of evidence suggests that unconditional cash transfers (UCTs) may not impact a summary measure of health service use in children and adults in LMICs. However, UCTs probably or may improve some health outcomes (i.e. the likelihood of having had any illness, the likelihood of having been food secure, and the level of dietary diversity), one social determinant of health (i.e. the likelihood of attending school), and healthcare expenditure. The evidence on the relative effectiveness of UCTs and CCTs remains very uncertain