Evaluating Information Retrieval and Access Tasks

This open access book summarizes the first two decades of the NII Testbeds and Community for Information access Research (NTCIR). NTCIR is a series of evaluation forums run by a global team of researchers and hosted by the National Institute of Informatics (NII), Japan. The book is unique in that it discusses not just what was done at NTCIR, but also how it was done and the impact it has achieved. For example, in some chapters the reader sees the early seeds of what eventually grew to be the search engines that provide access to content on the World Wide Web, today’s smartphones that can tailor what they show to the needs of their owners, and the smart speakers that enrich our lives at home and on the move. We also get glimpses into how new search engines can be built for mathematical formulae, or for the digital record of a lived human life. Key to the success of the NTCIR endeavor was early recognition that information access research is an empirical discipline and that evaluation therefore lay at the core of the enterprise. Evaluation is thus at the heart of each chapter in this book. They show, for example, how the recognition that some documents are more important than others has shaped thinking about evaluation design. The thirty-three contributors to this volume speak for the many hundreds of researchers from dozens of countries around the world who together shaped NTCIR as organizers and participants. This book is suitable for researchers, practitioners, and students—anyone who wants to learn about past and present evaluation efforts in information retrieval, information access, and natural language processing, as well as those who want to participate in an evaluation task or even to design and organize one

Estimating the Effect of Student Aid on College Enrollment: Evidence from a Government Grant Policy Reform

In this paper, we investigate the responsiveness of the demand for college to changes in student aid arising from a Danish reform. We separately identify the effect of aid from that of other observed and unobserved variables such as parental income. We exploit the combination of a kinked aid scheme and a reform of the student aid scheme to identify the effect of direct costs on college enrollment. To allow for heterogeneous responses due to borrowing constraints, we use detailed information on parents' assets. We find that enrollment is less responsive than found in other studies and that the presence of borrowing constraints only deters college enrollment to a minor extent.college attendance, educational subsidies, reform, kink regression

Robotic Training for the Integration of Material Performances in Timber Manufacturing

The research focuses on testing a series of material-sensitive robotic training methods that flexibly extend the range of subtractive manufacturing processes available to designers based on the integration of manufacturing knowledge at an early design stage. In current design practices, the lack of feedback information between the different steps of linear design workflows forces designers to engage with only a limited range of standard materials and manufacturing techniques, leading to wasteful and inefficient solutions. With a specific focus on timber subtractive manufacturing, the work presented in this thesis addresses the main issue hindering the utilisation of non-standard tools and heterogeneous materials in design processes which is the significant deviation between what is prescribed in the digital design environment and the respective fabrication outcome. To begin, it has been demonstrated the extent to which the heterogeneous properties of timber affect the outcome of the robotic carving process beyond the acceptable tolerance thresholds for design purposes. Resting on this premise, the devised strategy to address such a material variance involved capturing, transferring, augmenting and integrating manufacturing knowledge through the collection of real- world fabrication data, both by human experts and robotic sessions, and training of machine learning models (i.e. Artificial Neural Networks) to achieve an accurate simulation of the robotic manufacturing task informed by specific sets of tools affordances and material behaviours. The results of the training process have demonstrated that it is possible to accurately simulate the carving process to a degree sufficient for design applications, anticipating the influence of material and tool properties on the carved geometry. The collaborations with the industry partners of the project, ROK Architects (Zürich) and BIG (Copenhagen), provided the opportunity to assess the different practical uses and related implications of the tools in a real-world scenario following an open-ended and explorative approach based on several iterations of the full design-to-production cycle. The findings have shown that the devised strategy supports decision-making procedures at an early stage of the design process and enables the exploration of novel, previously unavailable, solutions informed by material and tool affordances

CEQE: Contextualized Embeddings for Query Expansion

In this work we leverage recent advances in context-sensitive language models to improve the task of query expansion. Contextualized word representation models, such as ELMo and BERT, are rapidly replacing static embedding models. We propose a new model, Contextualized Embeddings for Query Expansion (CEQE), that utilizes query-focused contextualized embedding vectors. We study the behavior of contextual representations generated for query expansion in ad-hoc document retrieval. We conduct our experiments on probabilistic retrieval models as well as in combination with neural ranking models. We evaluate CEQE on two standard TREC collections: Robust and Deep Learning. We find that CEQE outperforms static embedding-based expansion methods on multiple collections (by up to 18% on Robust and 31% on Deep Learning on average precision) and also improves over proven probabilistic pseudo-relevance feedback (PRF) models. We further find that multiple passes of expansion and reranking result in continued gains in effectiveness with CEQE-based approaches outperforming other approaches. The final model incorporating neural and CEQE-based expansion score achieves gains of up to 5% in P@20 and 2% in AP on Robust over the state-of-the-art transformer-based re-ranking model, Birch

The Mechanics of Embodiment: A Dialogue on Embodiment and Computational Modeling

Embodied theories are increasingly challenging traditional views of cognition by arguing that conceptual representations that constitute our knowledge are grounded in sensory and motor experiences, and processed at this sensorimotor level, rather than being represented and processed abstractly in an amodal conceptual system. Given the established empirical foundation, and the relatively underspecified theories to date, many researchers are extremely interested in embodied cognition but are clamouring for more mechanistic implementations. What is needed at this stage is a push toward explicit computational models that implement sensory-motor grounding as intrinsic to cognitive processes. In this article, six authors from varying backgrounds and approaches address issues concerning the construction of embodied computational models, and illustrate what they view as the critical current and next steps toward mechanistic theories of embodiment. The first part has the form of a dialogue between two fictional characters: Ernest, the �experimenter�, and Mary, the �computational modeller�. The dialogue consists of an interactive sequence of questions, requests for clarification, challenges, and (tentative) answers, and touches the most important aspects of grounded theories that should inform computational modeling and, conversely, the impact that computational modeling could have on embodied theories. The second part of the article discusses the most important open challenges for embodied computational modelling

Pretrained Transformers for Text Ranking: BERT and Beyond

The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query. Although the most common formulation of text ranking is search, instances of the task can also be found in many natural language processing applications. This survey provides an overview of text ranking with neural network architectures known as transformers, of which BERT is the best-known example. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing (NLP), information retrieval (IR), and beyond. In this survey, we provide a synthesis of existing work as a single point of entry for practitioners who wish to gain a better understanding of how to apply transformers to text ranking problems and researchers who wish to pursue work in this area. We cover a wide range of modern techniques, grouped into two high-level categories: transformer models that perform reranking in multi-stage architectures and dense retrieval techniques that perform ranking directly. There are two themes that pervade our survey: techniques for handling long documents, beyond typical sentence-by-sentence processing in NLP, and techniques for addressing the tradeoff between effectiveness (i.e., result quality) and efficiency (e.g., query latency, model and index size). Although transformer architectures and pretraining techniques are recent innovations, many aspects of how they are applied to text ranking are relatively well understood and represent mature techniques. However, there remain many open research questions, and thus in addition to laying out the foundations of pretrained transformers for text ranking, this survey also attempts to prognosticate where the field is heading

SHREC’20 Track:Retrieval of digital surfaces with similar geometric reliefs

International audienceThis paper presents the methods that have participated in the SHREC'20 contest on retrieval of surface patches with similar geometric reliefs and 1 the analysis of their performance over the benchmark created for this challenge. The goal of the context is to verify the possibility of retrieving 3D models only based on the reliefs that are present on their surface and to compare methods that are suitable for this task. This problem is related to many real world applications, such as the classification of cultural heritage goods or the analysis of different materials. To address this challenge, it is necessary to characterize the local "geometric pattern" information, possibly forgetting model size and bending. Seven groups participated in this contest and twenty runs were submitted for evaluation. The performances of the methods reveal that good results are achieved with a number of techniques that use different approaches

The Effects of Perceived Predation Threat on Stress Response and Memory in Birds

This study examined how perception of predator cues, across three sensory modalities, affects physiology and behaviour of songbirds. I hypothesized that the perception of predator threat would elicit physiological and behavioural responses in both acute and chronic exposure conditions. My first study examined the responses of wild-caught black-capped chickadees (Poecile atricapillus) and house sparrows (Passer domesticus), as well as lab-bred zebra finches (Taeniopygia guttata), to acute predation cue exposure by coding video recorded behaviour and corticosterone analysis. My second study examined changes in black-capped chickadees’ foraging behaviour and memory retention after chronic exposure to acoustic predation cues. There were no strong effects of the predator cue exposure on behaviour, corticosterone, or memory retention. This study expands on and fills gaps from the previous literature by examining different modalities of perception and predator effects on spatial memory abilities