The relationship of word error rate to document ranking

This paper describes two experiments that examine the relationship of Word Error Rate (WER) of retrieved spoken documents returned by a spoken document retrieval system. Previous work has demonstrated that recognition errors do not significantly affect retrieval effectiveness but whether they will adversely affect relevance judgement remains unclear. A user-based experiment measuring ability to judge relevance from the recognised text presented in a retrieved result list was conducted. The results indicated that users were capable of judging relevance accurately despite transcription errors. This lead an examination of the relationship of WER in retrieved audio documents to their rank position when retrieved for a particular query. Here it was shown that WER was somewhat lower for top ranked documents than it was for documents retrieved further down the ranking, thereby indicating a possible explanation for the success of the user experiment

Broad expertise retrieval in sparse data environments

Expertise retrieval has been largely unexplored on data other than the W3C collection. At the same time, many intranets of universities and other knowledge-intensive organisations offer examples of relatively small but clean multilingual expertise data, covering broad ranges of expertise areas. We first present two main expertise retrieval tasks, along with a set of baseline approaches based on generative language modeling, aimed at finding expertise relations between topics and people. For our experimental evaluation, we introduce (and release) a new test set based on a crawl of a university site. Using this test set, we conduct two series of experiments. The first is aimed at determining the effectiveness of baseline expertise retrieval methods applied to the new test set. The second is aimed at assessing refined models that exploit characteristic features of the new test set, such as the organizational structure of the university, and the hierarchical structure of the topics in the test set. Expertise retrieval models are shown to be robust with respect to environments smaller than the W3C collection, and current techniques appear to be generalizable to other settings

Crowdsourcing in Computer Vision

Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.Comment: A 69-page meta review of the field, Foundations and Trends in Computer Graphics and Vision, 201

Introspective physicalism as an approach to the science of consciousness

Most ‘theories of consciousness’ are based on vague speculations about the properties of conscious experience. We aim to provide a more solid basis for a science of consciousness. We argue that a theory of consciousness should provide an account of the very processes that allow us to acquire and use information about our own mental states – the processes underlying introspection. This can be achieved through the construction of information processing models that can account for ‘Type-C’ processes. Type-C processes can be specified experimentally by identifying paradigms in which awareness of the stimulus is necessary for an intentional action. The Shallice (1988b) framework is put forward as providing an initial account of Type-C processes, which can relate perceptual consciousness to consciously performed actions. Further, we suggest that this framework may be refined through the investigation of the functions of prefrontal cortex. The formulation of our approach requires us to consider fundamental conceptual and methodological issues associated with consciousness. The most significant of these issues concerns the scientific use of introspective evidence. We outline and justify a conservative methodological approach to the use of introspective evidence, with attention to the difficulties historically associated with its use in psychology

Formal models, usability and related work in IR (editorial for special edition)

The Glasgow IR group has carried out both theoretical and empirical work, aimed at giving end users efficient and effective access to large collections of multimedia data

Understanding analogical reasoning : viewpoints from psychology and related disciplines

Analogy and metaphor have a long history of study in linguistics, education, philosophy and psychology. Consensus over what analogy is or how analogy functions in language and thought, however, has been elusive. This paper, the first in a two part series, examines these various research traditions, attempting to bring out major lines of agreement over the role of analogy in individual human experience. As well as being a general literature review which may be helpful for newcomers to the study of analogy, this paper attempts to extract from these literatures existing theories, models and concepts which may be interesting or useful for computational studies of analogical reasoning

RECVID as a Re-Usable Test-Collection for Video Retrieval

TRECVID has been running as a video retrieval benchmarking platform for a number of years now. Some progress seems to be made in the area of video retrieval, but also it has been shown that many of the differences in scores between tested approaches are nonsignificant. This paper studies the reliability of the TRECVID search collections for measuring video retrieval effectiveness and investigates how useful the collections are for re-use

Social and interactional practices for disseminating current awareness information in an organisational setting.

Current awareness services are designed to keep users informed about recent developments based around user need profiles. In organisational settings, they may operate through both electronic and social interactions aimed at delivering information that is relevant, pertinent and current. Understanding these interactions can reveal the tensions in current awareness dissemination and help inform ways of making services more effective and efficient. We report an in-depth, observational study of electronic current awareness use within a large London law firm. The study found that selection, re-aggregation and forwarding of information by multiple actors gives rise to a complex sociotechnical distribution network. Knowledge management staff act as a layer of “intelligent filters” sensitive to complex, local information needs; their distribution decisions address multiple situational relevance factors in a situation fraught with information overload and restrictive time-pressures. Their decisions aim to optimise conflicting constraints of recall, precision and information quantity. Critical to this is the use of dynamic profile updates which propagate back through the network through formal and informal social interactions. This supports changes to situational relevance judgements and so allows the network to ‘self-tune’. These findings lead to design requirements, including that systems should support rapid assessment of information items against an individual’s interests; that it should be possible to organise information for different subsequent uses; and that there should be back-propagation from information consumers to providers, to tune the understanding of their information needs