Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings

PURPOSE: One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgments of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the “wisdom of crowds”. The aim of this study is to examine whether aggregated judgments will be more stable and thus more reliable over time than individual user judgments. DESIGN/METHODS: In this study two simple measures are proposed to calculate the aggregated judgments of search results and compare their reliability and stability to individual user judgments. In addition, the aggregated “wisdom of crowds” judgments were used as a means to compare the differences between human assessments of search results and search engine’s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgments were considered in this study: 1) relevance on a 4-point scale, and 2) ranking on a 10-point scale without ties. FINDINGS: It was found that aggregated judgments are much more stable than individual user judgments, yet they are quite different from search engine rankings. Practical implications: The proposed “wisdom of crowds” based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalization and adapting search engine’s ranking over time to changes in users preferences. ORIGINALITY/VALUE: This is a first study that applies the notion of “wisdom of crowds” to examine the under-explored phenomenon in the literature of “change in time” in user evaluation of relevance

Mobile, ubiquitous information seeking, as a group: the iBingo collaborative video retrieval system

iBingo features two or more users performing collaborative information seeking tasks, using mobile devices, Apple iPod iTouch in our case. The novelty in our work is that the system, called iBingo, mediates the collaborative searches among the users and performs a realtime division of labour among co-searchers so users are presented with documents which are both unique and tailored to the individual. This enables each user to explore unique subsets of the retrieved information space. We demonstrate iBingo mobile collabo-rative search on a video collection from TRECVid 2007

How algorithmic popularity bias hinders or promotes quality

Algorithms that favor popular items are used to help us select among many choices, from engaging articles on a social media news feed to songs and books that others have purchased, and from top-raked search engine results to highly-cited scientific papers. The goal of these algorithms is to identify high-quality items such as reliable news, beautiful movies, prestigious information sources, and important discoveries --- in short, high-quality content should rank at the top. Prior work has shown that choosing what is popular may amplify random fluctuations and ultimately lead to sub-optimal rankings. Nonetheless, it is often assumed that recommending what is popular will help high-quality content "bubble up" in practice. Here we identify the conditions in which popularity may be a viable proxy for quality content by studying a simple model of cultural market endowed with an intrinsic notion of quality. A parameter representing the cognitive cost of exploration controls the critical trade-off between quality and popularity. We find a regime of intermediate exploration cost where an optimal balance exists, such that choosing what is popular actually promotes high-quality items to the top. Outside of these limits, however, popularity bias is more likely to hinder quality. These findings clarify the effects of algorithmic popularity bias on quality outcomes, and may inform the design of more principled mechanisms for techno-social cultural markets

The other side of the social web: A taxonomy for social information access

The power of the modern Web, which is frequently called the Social Web or Web 2.0, is frequently traced to the power of users as contributors of various kinds of contents through Wikis, blogs, and resource sharing sites. However, the community power impacts not only the production of Web content, but also the access to all kinds of Web content. A number of research groups worldwide explore what we call social information access techniques that help users get to the right information using "collective wisdom" distilled from actions of those who worked with this information earlier. This invited talk offers a brief introduction into this important research stream and reviews recent works on social information access performed at the University of Pittsburgh's PAWS Lab lead by the author. Copyright © 2012 by the Association for Computing Machinery, Inc. (ACM)

On being a campus chaplain at the beginning of a new millennium

Proverbs 8:1-7, 10-11, 22-36; Phil 4:8; Mark 10:35-45

In Search of Reusable Educational Resources in the Web

[EN] Nowadays there is a high demand from teachers to precisely find online learning resources that are free from copyright restrictions or publicly licensed to use, adapt and redistribute in their own courses. This paper investigates the state of the art to support teachers in this search process. Repository based strategies for dissemination of educational resources are discussed and critiqued and the added value of a semantic web approach is shown. The ontology schema.org and its suitability for semantic annotation of educational resources is introduced. Current ways and weaknesses to discover educational resources based on appropriate semantic data are presented. The possibility to use the wisdom of the crowd of learners and teachers defining semantic knowledge about used learning resources is addressed. For demonstration purposes within all sections the course subject ‘Semantic SEO’, dealt in the course ‘SEO – Search Engine Optimization’ held by the author in 2016, is used.Steinberger, C. (2017). In Search of Reusable Educational Resources in the Web. En Proceedings of the 3rd International Conference on Higher Education Advances. Editorial Universitat Politècnica de València. 321-328. https://doi.org/10.4995/HEAD17.2017.518632132

Serious Games in Cultural Heritage

Although the widespread use of gaming for leisure purposes has been well documented, the use of games to support cultural heritage purposes, such as historical teaching and learning, or for enhancing museum visits, has been less well considered. The state-of-the-art in serious game technology is identical to that of the state-of-the-art in entertainment games technology. As a result the field of serious heritage games concerns itself with recent advances in computer games, real-time computer graphics, virtual and augmented reality and artificial intelligence. On the other hand, the main strengths of serious gaming applications may be generalised as being in the areas of communication, visual expression of information, collaboration mechanisms, interactivity and entertainment. In this report, we will focus on the state-of-the-art with respect to the theories, methods and technologies used in serious heritage games. We provide an overview of existing literature of relevance to the domain, discuss the strengths and weaknesses of the described methods and point out unsolved problems and challenges. In addition, several case studies illustrating the application of methods and technologies used in cultural heritage are presented

"Revolution? What Revolution?" Successes and limits of computing technologies in philosophy and religion

Computing technologies like other technological innovations in the modern West are inevitably introduced with the rhetoric of "revolution". Especially during the 1980s (the PC revolution) and 1990s (the Internet and Web revolutions), enthusiasts insistently celebrated radical changes— changes ostensibly inevitable and certainly as radical as those brought about by the invention of the printing press, if not the discovery of fire.\ud These enthusiasms now seem very "1990s�—in part as the revolution stumbled with the dot.com failures and the devastating impacts of 9/11. Moreover, as I will sketch out below, the patterns of diffusion and impact in philosophy and religion show both tremendous success, as certain revolutionary promises are indeed kept—as well as (sometimes spectacular) failures. Perhaps we use revolutionary rhetoric less frequently because the revolution has indeed succeeded: computing technologies, and many of the powers and potentials they bring us as scholars and religionists have become so ubiquitous and normal that they no longer seem "revolutionary at all. At the same time, many of the early hopes and promises instantiated in such specific projects as Artificial Intelligence and anticipations of virtual religious communities only have been dashed against the apparently intractable limits of even these most remarkable technologies. While these failures are usually forgotten they leave in their wake a clearer sense of what these new technologies can, and cannot do