Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings

PURPOSE: One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgments of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the “wisdom of crowds”. The aim of this study is to examine whether aggregated judgments will be more stable and thus more reliable over time than individual user judgments. DESIGN/METHODS: In this study two simple measures are proposed to calculate the aggregated judgments of search results and compare their reliability and stability to individual user judgments. In addition, the aggregated “wisdom of crowds” judgments were used as a means to compare the differences between human assessments of search results and search engine’s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgments were considered in this study: 1) relevance on a 4-point scale, and 2) ranking on a 10-point scale without ties. FINDINGS: It was found that aggregated judgments are much more stable than individual user judgments, yet they are quite different from search engine rankings. Practical implications: The proposed “wisdom of crowds” based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalization and adapting search engine’s ranking over time to changes in users preferences. ORIGINALITY/VALUE: This is a first study that applies the notion of “wisdom of crowds” to examine the under-explored phenomenon in the literature of “change in time” in user evaluation of relevance

When in doubt ask the crowd : leveraging collective intelligence for improving event detection and machine learning

[no abstract

A Multi-Dimensional Approach for Framing Crowdsourcing Archetypes

All different kinds of organizations – business, public, and non-governmental alike – are becoming aware of a soaring complexity in problem solving, decision making and idea development. In a multitude of circumstances, multidisciplinary teams, high-caliber skilled resources and world-class computer suites do not suffice to cope with such a complexity: in fact, a further need concerns the sharing and ‘externalization’ of tacit knowledge already existing in the society. In this direction, participatory tendencies flourishing in the interconnected society in which we live today lead ‘collective intelligence’ to emerge as key ingredient of distributed problem solving systems going well beyond the traditional boundaries of organizations. Resulting outputs can remarkably enrich decision processes and creative processes carried out by indoor experts, allowing organizations to reap benefits in terms of opportunity, time and cost. Taking stock of the mare magnum of promising opportunities to be tapped, of the inherent diversity lying among them, and of the enormous success of some initiative launched hitherto, the thesis aspires to provide a sound basis for the clear comprehension and systematic exploitation of crowdsourcing. After a thorough literature review, the thesis explores new ways for formalizing crowdsourcing models with the aim of distilling a brand-new multi-dimensional framework to categorize various crowdsourcing archetypes. To say it in a nutshell, the proposed framework combines two dimensions (i.e., motivations to participate and organization of external solvers) in order to portray six archetypes. Among the numerous significant elements of novelty brought by this framework, the prominent one is the ‘holistic’ approach that combines both profit and non-profit, trying to put private and public sectors under a common roof in order to examine in a whole corpus the multi-faceted mechanisms for mobilizing and harnessing competence and expertise which are distributed among the crowd. Looking at how the crowd may be turned into value to be internalized by organizations, the thesis examines crowdsourcing practices in the public as well in the private sector. Regarding the former, the investigation leverages the experience into the PADGETS project through action research – drawing on theoretical studies as well as on intensive fieldwork activities – to systematize how crowdsourcing can be fruitfully incorporated into the policy lifecycle. Concerning the private realm, a cohort of real cases in the limelight is examined – having recourse to case study methodology – to formalize different ways through which crowdsourcing becomes a business model game-changer. Finally, the two perspectives (i.e., public and private) are coalesced into an integrated view acting as a backdrop for proposing next-generation governance model massively hinged on crowdsourcing. In fact, drawing on archetypes schematized, the thesis depicts a potential paradigm that government may embrace in the coming future to tap the potential of collective intelligence, thus maximizing the utilization of a resource that today seems certainly underexploited

Delivering IoT Services in Smart Cities and Environmental Monitoring through Collective Awareness, Mobile Crowdsensing and Open Data

The Internet of Things (IoT) is the paradigm that allows us to interact with the real world by means of networking-enabled devices and convert physical phenomena into valuable digital knowledge. Such a rapidly evolving field leveraged the explosion of a number of technologies, standards and platforms. Consequently, different IoT ecosystems behave as closed islands and do not interoperate with each other, thus the potential of the number of connected objects in the world is far from being totally unleashed. Typically, research efforts in tackling such challenge tend to propose a new IoT platforms or standards, however, such solutions find obstacles in keeping up the pace at which the field is evolving. Our work is different, in that it originates from the following observation: in use cases that depend on common phenomena such as Smart Cities or environmental monitoring a lot of useful data for applications is already in place somewhere or devices capable of collecting such data are already deployed. For such scenarios, we propose and study the use of Collective Awareness Paradigms (CAP), which offload data collection to a crowd of participants. We bring three main contributions: we study the feasibility of using Open Data coming from heterogeneous sources, focusing particularly on crowdsourced and user-contributed data that has the drawback of being incomplete and we then propose a State-of-the-Art algorith that automatically classifies raw crowdsourced sensor data; we design a data collection framework that uses Mobile Crowdsensing (MCS) and puts the participants and the stakeholders in a coordinated interaction together with a distributed data collection algorithm that prevents the users from collecting too much or too less data; (3) we design a Service Oriented Architecture that constitutes a unique interface to the raw data collected through CAPs through their aggregation into ad-hoc services, moreover, we provide a prototype implementation

Enhancing Automation and Interoperability in Enterprise Crowdsourcing Environments

The last couple of years have seen a fascinating evolution. While the early Web predominantly focused on human consumption of Web content, the widespread dissemination of social software and Web 2.0 technologies enabled new forms of collaborative content creation and problem solving. These new forms often utilize the principles of collective intelligence, a phenomenon that emerges from a group of people who either cooperate or compete with each other to create a result that is better or more intelligent than any individual result (Leimeister, 2010; Malone, Laubacher, & Dellarocas, 2010). Crowdsourcing has recently gained attention as one of the mechanisms that taps into the power of web-enabled collective intelligence (Howe, 2008). Brabham (2013) defines it as “an online, distributed problem-solving and production model that leverages the collective intelligence of online communities to serve specific organizational goals” (p. xix). Well-known examples of crowdsourcing platforms are Wikipedia, Amazon Mechanical Turk, or InnoCentive. Since the emergence of the term crowdsourcing in 2006, one popular misconception is that crowdsourcing relies largely on an amateur crowd rather than a pool of professional skilled workers (Brabham, 2013). As this might be true for low cognitive tasks, such as tagging a picture or rating a product, it is often not true for complex problem-solving and creative tasks, such as developing a new computer algorithm or creating an impressive product design. This raises the question of how to efficiently allocate an enterprise crowdsourcing task to appropriate members of the crowd. The sheer number of crowdsourcing tasks available at crowdsourcing intermediaries makes it especially challenging for workers to identify a task that matches their skills, experiences, and knowledge (Schall, 2012, p. 2). An explanation why the identification of appropriate expert knowledge plays a major role in crowdsourcing is partly given in Condorcet’s jury theorem (Sunstein, 2008, p. 25). The theorem states that if the average participant in a binary decision process is more likely to be correct than incorrect, then as the number of participants increases, the higher the probability is that the aggregate arrives at the right answer. When assuming that a suitable participant for a task is more likely to give a correct answer or solution than an improper one, efficient task recommendation becomes crucial to improve the aggregated results in crowdsourcing processes. Although some assumptions of the theorem, such as independent votes, binary decisions, and homogenous groups, are often unrealistic in practice, it illustrates the importance of an optimized task allocation and group formation that consider the task requirements and workers’ characteristics. Ontologies are widely applied to support semantic search and recommendation mechanisms (Middleton, De Roure, & Shadbolt, 2009). However, little research has investigated the potentials and the design of an ontology for the domain of enterprise crowdsourcing. The author of this thesis argues in favor of enhancing the automation and interoperability of an enterprise crowdsourcing environment with the introduction of a semantic vocabulary in form of an expressive but easy-to-use ontology. The deployment of a semantic vocabulary for enterprise crowdsourcing is likely to provide several technical and economic benefits for an enterprise. These benefits were the main drivers in efforts made during the research project of this thesis: 1. Task allocation: With the utilization of the semantics, requesters are able to form smaller task-specific crowds that perform tasks at lower costs and in less time than larger crowds. A standardized and controlled vocabulary allows requesters to communicate specific details about a crowdsourcing activity within a web page along with other existing displayed information. This has advantages for both contributors and requesters. On the one hand, contributors can easily and precisely search for tasks that correspond to their interests, experiences, skills, knowledge, and availability. On the other hand, crowdsourcing systems and intermediaries can proactively recommend crowdsourcing tasks to potential contributors (e.g., based on their social network profiles). 2. Quality control: Capturing and storing crowdsourcing data increases the overall transparency of the entire crowdsourcing activity and thus allows for a more sophisticated quality control. Requesters are able to check the consistency and receive appropriate support to verify and validate crowdsourcing data according to defined data types and value ranges. Before involving potential workers in a crowdsourcing task, requesters can also judge their trustworthiness based on previous accomplished tasks and hence improve the recruitment process. 3. Task definition: A standardized set of semantic entities supports the configuration of a crowdsourcing task. Requesters can evaluate historical crowdsourcing data to get suggestions for equal or similar crowdsourcing tasks, for example, which incentive or evaluation mechanism to use. They may also decrease their time to configure a crowdsourcing task by reusing well-established task specifications of a particular type. 4. Data integration and exchange: Applying a semantic vocabulary as a standard format for describing enterprise crowdsourcing activities allows not only crowdsourcing systems inside but also crowdsourcing intermediaries outside the company to extract crowdsourcing data from other business applications, such as project management, enterprise resource planning, or social software, and use it for further processing without retyping and copying the data. Additionally, enterprise or web search engines may exploit the structured data and provide enhanced search, browsing, and navigation capabilities, for example, clustering similar crowdsourcing tasks according to the required qualifications or the offered incentives.:Summary: Hetmank, L. (2014). Enhancing Automation and Interoperability in Enterprise Crowdsourcing Environments (Summary). Article 1: Hetmank, L. (2013). Components and Functions of Crowdsourcing Systems – A Systematic Literature Review. In 11th International Conference on Wirtschaftsinformatik (WI). Leipzig. Article 2: Hetmank, L. (2014). A Synopsis of Enterprise Crowdsourcing Literature. In 22nd European Conference on Information Systems (ECIS). Tel Aviv. Article 3: Hetmank, L. (2013). Towards a Semantic Standard for Enterprise Crowdsourcing – A Scenario-based Evaluation of a Conceptual Prototype. In 21st European Conference on Information Systems (ECIS). Utrecht. Article 4: Hetmank, L. (2014). Developing an Ontology for Enterprise Crowdsourcing. In Multikonferenz Wirtschaftsinformatik (MKWI). Paderborn. Article 5: Hetmank, L. (2014). An Ontology for Enhancing Automation and Interoperability in Enterprise Crowdsourcing Environments (Technical Report). Retrieved from http://nbn-resolving.de/urn:nbn:de:bsz:14-qucosa-155187

Hierarchical Entity Resolution using an Oracle

In many applications, entity references (i.e., records) and entities need to be organized to capture diverse relationships like type-subtype, is-A (mapping entities to types), and duplicate (mapping records to entities) relationships. However, automatic identification of such relationships is often inaccurate due to noise and heterogeneous representation of records across sources. Similarly, manual maintenance of these relationships is infeasible and does not scale to large datasets. In this work, we circumvent these challenges by considering weak supervision in the form of an oracle to formulate a novel hierarchical ER task. In this setting, records are clustered in a tree-like structure containing records at leaf-level and capturing record-entity (duplicate), entity-type (is-A) and subtype-supertype relationships. For effective use of supervision, we leverage triplet comparison oracle queries that take three records as input and output the most similar pair(s). We develop HierER, a querying strategy that uses record pair similarities to minimize the number of oracle queries while maximizing the identified hierarchical structure. We show theoretically and empirically that HierER is effective under different similarity noise models and demonstrate empirically that HierER can scale up to million-size datasets

PowerAqua: Open Question Answering on the Semantic Web

With the rapid growth of semantic information in the Web, the processes of searching and querying these very large amounts of heterogeneous content have become increasingly challenging. This research tackles the problem of supporting users in querying and exploring information across multiple and heterogeneous Semantic Web (SW) sources. A review of literature on ontology-based Question Answering reveals the limitations of existing technology. Our approach is based on providing a natural language Question Answering interface for the SW, PowerAqua. The realization of PowerAqua represents a considerable advance with respect to other systems, which restrict their scope to an ontology-specific or homogeneous fraction of the publicly available SW content. To our knowledge, PowerAqua is the only system that is able to take advantage of the semantic data available on the Web to interpret and answer user queries posed in natural language. In particular, PowerAqua is uniquely able to answer queries by combining and aggregating information, which can be distributed across heterogeneous semantic resources. Here, we provide a complete overview of our work on PowerAqua, including: the research challenges it addresses; its architecture; the techniques we have realised to map queries to semantic data, to integrate partial answers drawn from different semantic resources and to rank alternative answers; and the evaluation studies we have performed, to assess the performance of PowerAqua. We believe our experiences can be extrapolated to a variety of end-user applications that wish to open up to large scale and heterogeneous structured datasets, to be able to exploit effectively what possibly is the greatest wealth of data in the history of Artificial Intelligence

Collective intelligence: creating a prosperous world at peace

XXXII, 612 p. ; 24 cmLibro ElectrónicoEn este documento se plantea un tema de interes general mas como lo es especificamente el tema de la evolucion de la sociedad en materia de industria y crecimiento de las actividades humanas en el aspecto de desarrollo de la creatividad enfocada a los mercadosedited by Mark Tovey ; foreword by Yochai Benkler (re-mixed by Hassan Masum) ; prefaces by Thomas Malone, Tom Atlee & Pierre Levy ; afterword by Paul Martin & Thomas Homer-Dixon.The era of collective intelligence has begun in earnest. While others have written about the wisdom of crowds, an army of Davids, and smart mobs, this collection of essays for the first time brings together fifty-five pioneers in the emerging discipline of collective intelligence. They provide a base of tools for connecting people, producing high-functioning teams, collaborating at multiple scales, and encouraging effective peer-production. Emerging models are explored for digital deliberative democracy, self-governance, legislative transparency, true-cost accounting, and the ethical use of open sources and methods. Collective Intelligence is the first of a series of six books, which will also include volumes on Peace Intelligence, Commercial Intelligence, Gift Intelligence, Cultural Intelligence, and Global Intelligence.Table of Contents Dedication i Publisher’s Preface iii Foreword by Yochai Benkler Remix Hassan Masum xi The Wealth of Networks: Highlights remixed Editor’s Preface xxi Table of Contents xxv A What is collective intelligence and what will we do 1 about it? (Thomas W. Malone, MIT Center for Collective Intelligence) B Co-Intelligence, collective intelligence, and conscious 5 evolution (Tom Atlee, Co-Intelligence Institute) C A metalanguage for computer augmented collective 15 intelligence (Prof. Pierre Lévy, Canada Research Chair in Collective Intelligence, FRSC) I INDIVIDUALS & GROUPS I-01 Foresight I-01-01 Safety Glass (Karl Schroeder, science fiction author 23 and foresight consultant) I-01-02 2007 State of the Future (Jerome C. Glenn & 29 Theodore J. Gordon, United Nations Millennium Project) I-02 Dialogue & Deliberation I-02-01 Thinking together without ego: Collective intelligence 39 as an evolutionary catalyst (Craig Hamilton and Claire Zammit, Collective-Intelligence.US) I-02-02 The World Café: Awakening collective intelligence 47 and committed action (Juanita Brown, David Isaacs and the World Café Community) I-02-03 Collective intelligence and the emergence of 55 wholeness (Peggy Holman, Nexus for Change, The Change Handbook) I-02-04 Knowledge creation in collective intelligence (Bruce 65 LaDuke, Fortune 500, HyperAdvance.com) I-02-05 The Circle Organization: Structuring for collective 75 wisdom (Jim Rough, Dynamic Facilitation & The Center for Wise Democracy) I-03 Civic Intelligence I-03-01 Civic intelligence and the public sphere (Douglas 83 Schuler, Evergreen State College, Public Sphere Project) I-03-02 Civic intelligence and the security of the homeland 95 (John Kesler with Carole and David Schwinn, IngeniusOnline) I-03-03 Creating a Smart Nation (Robert Steele, OSS.Net) 107 I-03-04 University 2.0: Informing our collective intelligence 131 (Nancy Glock-Grueneich, HIGHEREdge.org) I-03-05 Producing communities of communications and 145 foreknowledge (Jason “JZ” Liszkiewicz, Reconfigure.org) I-03-06 Global Vitality Report 2025: Learning to transform I-04 Electronic Communities & Distributed Cognition I-04-01 Attentional capital and the ecology of online social 163 conflict and think together effectively (Peter+Trudy networks (Derek Lomas, Social Movement Lab, Johnson-Lenz, Johnson-Lenz.com ) UCSD) I-04-02 A slice of life in my virtual community (Howard 173 Rheingold, Whole Earth Review, Author & Educator) I-04-03 Shared imagination (Dr. Douglas C. Engelbart, 197 Bootstrap) I-05 Privacy & Openness I-05-01 We’re all swimming in media: End-users must be able 201 to keep secrets (Mitch Ratcliffe, BuzzLogic & Tetriad) I-05-02 Working openly (Lion Kimbro, Programmer and 205 Activist) I-06 Integral Approaches & Global Contexts I-06-01 Meta-intelligence for analyses, decisions, policy, and 213 action: The Integral Process for working on complex issues (Sara Nora Ross, Ph.D. ARINA & Integral Review) I-06-02 Collective intelligence: From pyramidal to global 225 (Jean-Francois Noubel, The Transitioner) I-06-03 Cultivating collective intelligence: A core leadership 235 competence in a complex world (George Pór, Fellow at Universiteit van Amsterdam) II LARGE-SCALE COLLABORATION II-01 Altruism, Group IQ, and Adaptation II-01-01 Empowering individuals towards collective online 245 production (Keith Hopper, KeithHopper.com) II-01-02 Who’s smarter: chimps, baboons or bacteria? The 251 power of Group IQ (Howard Bloom, author) II-01-03 A collectively generated model of the world (Marko 261 A. Rodriguez, Los Alamos National Laboratory) II-02 Crowd Wisdom and Cognitive Bias II-02-01 Science of CI: Resources for change (Norman L 265 Johnson, Chief Scientist at Referentia Systems, former LANL) II-02-02 Collectively intelligent systems (Jennifer H. Watkins, 275 Los Alamos National Laboratory) II-02-03 A contrarian view (Jaron Lanier, scholar-in-residence, 279 CET, UC Berkeley & Discover Magazine) II-03 Semantic Structures & The Semantic Web II-03-01 Information Economy Meta Language (Interview with 283 Professor Pierre Lévy, by George Pór) II-03-02 Harnessing the collective intelligence of the World- 293 Wide Web (Nova Spivack, RadarNetworks, Web 3.0) II-03-03 The emergence of a global brain (Francis Heylighen, 305 Free University of Brussels) II-04 Information Networks II-04-01 Networking and mobilizing collective intelligence (G. Parker Rossman, Future of Learning Pioneer) II-04-02 Toward high-performance organizations: A strategic 333 role for Groupware (Douglas C. Engelbart, Bootstrap) II-04-03 Search panacea or ploy: Can collective intelligence 375 improve findability? (Stephen E. Arnold, Arnold IT, Inc.) II-05 Global Games, Local Economies, & WISER II-05-01 World Brain as EarthGame (Robert Steele and many 389 others, Earth Intelligence Network) II-05-02 The Interra Project (Jon Ramer and many others) 399 II-05-03 From corporate responsibility to Backstory 409 Management (Alex Steffen, Executive Editor, Worldchanging.com) II-05-04 World Index of Environmental & Social 413 Responsibility (WISER) By the Natural Capital Institute II-06 Peer-Production & Open Source Hardware II-06-01 The Makers’ Bill of Rights (Jalopy, Torrone, and Hill) 421 II-06-02 3D Printing and open source design (James Duncan, 423 VP of Technology at Marketingisland) II-06-03 REBEARTHTM: 425 II-07 Free Wireless, Open Spectrum, and Peer-to-Peer II-07-01 Montréal Community Wi-Fi (Île Sans Fil) (Interview 433 with Michael Lenczner by Mark Tovey) II-07-02 The power of the peer-to-peer future (Jock Gill, 441 Founder, Penfield Gill Inc.) Growing a world 6.6 billion people would want to live in (Marc Stamos, B-Comm, LL.B) II-07-03 Open spectrum (David Weinberger) II-08 Mass Collaboration & Large-Scale Argumentation II-08-01 Mass collaboration, open source, and social 455 entrepreneurship (Mark Tovey, Advanced Cognitive Engineering Lab, Institute of Cognitive Science, Carleton University) II-08-02 Interview with Thomas Homer-Dixon (Hassan 467 Masum, McLaughlin-Rotman Center for Global Health) II-08-03 Achieving collective intelligence via large-scale argumentation (Mark Klein, MIT Center for Collective Intelligence) II-08-04 Scaling up open problem solving (Hassan Masum & 485 Mark Tovey) D Afterword: The Internet and the revitalization of 495 democracy (The Rt. Honourable Paul Martin & Thomas Homer-Dixon) E Epilogue by Tom Atlee 513 F Three Lists 515 1. Strategic Reading Categories 2. Synopsis of the New Progressives 3. Fifty-Two Questions that Matter G Glossary 519 H Index 52

Evaluation Methodologies for Visual Information Retrieval and Annotation

Die automatisierte Evaluation von Informations-Retrieval-Systemen erlaubt Performanz und Qualität der Informationsgewinnung zu bewerten. Bereits in den 60er Jahren wurden erste Methodologien für die system-basierte Evaluation aufgestellt und in den Cranfield Experimenten überprüft. Heutzutage gehören Evaluation, Test und Qualitätsbewertung zu einem aktiven Forschungsfeld mit erfolgreichen Evaluationskampagnen und etablierten Methoden. Evaluationsmethoden fanden zunächst in der Bewertung von Textanalyse-Systemen Anwendung. Mit dem rasanten Voranschreiten der Digitalisierung wurden diese Methoden sukzessive auf die Evaluation von Multimediaanalyse-Systeme übertragen. Dies geschah häufig, ohne die Evaluationsmethoden in Frage zu stellen oder sie an die veränderten Gegebenheiten der Multimediaanalyse anzupassen. Diese Arbeit beschäftigt sich mit der system-basierten Evaluation von Indizierungssystemen für Bildkollektionen. Sie adressiert drei Problemstellungen der Evaluation von Annotationen: Nutzeranforderungen für das Suchen und Verschlagworten von Bildern, Evaluationsmaße für die Qualitätsbewertung von Indizierungssystemen und Anforderungen an die Erstellung visueller Testkollektionen. Am Beispiel der Evaluation automatisierter Photo-Annotationsverfahren werden relevante Konzepte mit Bezug zu Nutzeranforderungen diskutiert, Möglichkeiten zur Erstellung einer zuverlässigen Ground Truth bei geringem Kosten- und Zeitaufwand vorgestellt und Evaluationsmaße zur Qualitätsbewertung eingeführt, analysiert und experimentell verglichen. Traditionelle Maße zur Ermittlung der Performanz werden in vier Dimensionen klassifiziert. Evaluationsmaße vergeben üblicherweise binäre Kosten für korrekte und falsche Annotationen. Diese Annahme steht im Widerspruch zu der Natur von Bildkonzepten. Das gemeinsame Auftreten von Bildkonzepten bestimmt ihren semantischen Zusammenhang und von daher sollten diese auch im Zusammenhang auf ihre Richtigkeit hin überprüft werden. In dieser Arbeit wird aufgezeigt, wie semantische Ähnlichkeiten visueller Konzepte automatisiert abgeschätzt und in den Evaluationsprozess eingebracht werden können. Die Ergebnisse der Arbeit inkludieren ein Nutzermodell für die konzeptbasierte Suche von Bildern, eine vollständig bewertete Testkollektion und neue Evaluationsmaße für die anforderungsgerechte Qualitätsbeurteilung von Bildanalysesystemen.Performance assessment plays a major role in the research on Information Retrieval (IR) systems. Starting with the Cranfield experiments in the early 60ies, methodologies for the system-based performance assessment emerged and established themselves, resulting in an active research field with a number of successful benchmarking activities. With the rise of the digital age, procedures of text retrieval evaluation were often transferred to multimedia retrieval evaluation without questioning their direct applicability. This thesis investigates the problem of system-based performance assessment of annotation approaches in generic image collections. It addresses three important parts of annotation evaluation, namely user requirements for the retrieval of annotated visual media, performance measures for multi-label evaluation, and visual test collections. Using the example of multi-label image annotation evaluation, I discuss which concepts to employ for indexing, how to obtain a reliable ground truth to moderate costs, and which evaluation measures are appropriate. This is accompanied by a thorough analysis of related work on system-based performance assessment in Visual Information Retrieval (VIR). Traditional performance measures are classified into four dimensions and investigated according to their appropriateness for visual annotation evaluation. One of the main ideas in this thesis adheres to the common assumption on the binary nature of the score prediction dimension in annotation evaluation. However, the predicted concepts and the set of true indexed concepts interrelate with each other. This work will show how to utilise these semantic relationships for a fine-grained evaluation scenario. Outcomes of this thesis result in a user model for concept-based image retrieval, a fully assessed image annotation test collection, and a number of novel performance measures for image annotation evaluation