A game-based approach towards human augmented image annotation.

PhDImage annotation is a difficult task to achieve in an automated way. In this thesis, a human-augmented approach to tackle this problem is discussed and suitable strategies are derived to solve it. The proposed technique is inspired by human-based computation in what is called “human-augmented” processing to overcome limitations of fully automated technology for closing the semantic gap. The approach aims to exploit what millions of individual gamers are keen to do, i.e. enjoy computer games, while annotating media. In this thesis, the image annotation problem is tackled by a game based framework. This approach combines image processing and a game theoretic model to gather media annotations. Although the proposed model behaves similar to a single player game model, the underlying approach has been designed based on a two-player model which exploits the player’s contribution to the game and previously recorded players to improve annotations accuracy. In addition, the proposed framework is designed to predict the player’s intention through Markovian and Sequential Sampling inferences in order to detect cheating and improve annotation performances. Finally, the proposed techniques are comprehensively evaluated with three different image datasets and selected representative results are reported

Mining Human Shape Perception with Role Playing Games

Games with a purpose’ is a paradigm wheregames are designed to computationally capture the essence of theunderlying collective human conscience or commonsense thatplays a major role in decision-making. This human computingmethod ensures spontaneous participation of players who, asa byproduct of playing, provide useful data that is impossibleto generate computationally and extremely difficult to collectthrough extensive surveys. In this paper we describe a gamethat allows us to collect data on human perception of characterbody shapes. The paper describes the experimental setup, relatedgame design constraints, art creation, and data analysis. Inour interactive role-playing detective game titled Villain Ville,players are asked to characterize different versions of fullbodycolor portraits of three villain characters. They are latersupposed to correctly match their character-trait ratings toa set of characters represented only with outlines of primitivevector shapes. By transferring human intelligence tasks into coregame-play mechanics, we have successfully managed to collectmotivated data. Preliminary analysis on game data generatedby 50 secondary school students shows a convergence to somecommon perception associations between role, physicality andpersonality. We hope to harness this game to discover perceptionfor a wide variety of body-shapes to build up an intelligent shapetrait-role model, with application in tutored drawing, proceduralcharacter geometry creation and intelligent retrieval

GECKA3D: A 3D Game Engine for Commonsense Knowledge Acquisition

Commonsense knowledge representation and reasoning is key for tasks such as artificial intelligence and natural language understanding. Since commonsense consists of information that humans take for granted, gathering it is an extremely difficult task. In this paper, we introduce a novel 3D game engine for commonsense knowledge acquisition (GECKA3D) which aims to collect commonsense from game designers through the development of serious games. GECKA3D integrates the potential of serious games and games with a purpose. This provides a platform for the acquisition of re-usable and multi-purpose knowledge, and also enables the development of games that can provide entertainment value and teach players something meaningful about the actual world they live in

A tempo-based music search engine with multimodal query

Master'sMASTER OF SCIENC

Mutually reinforcing systems

Human computation can be described as outsourcing part of a computational process to humans. This technique might be used when a problem can be solved better by humans than computers or it may require a level of adaptation that computers are not yet capable of handling. This can be particularly important in changeable settings which require a greater level of adaptation to the surrounding environment. In most cases, human computation has been used to gather data that computers struggle to create. Games with by-products can provide an incentive for people to carry out such tasks by rewarding them with entertainment. These are games which are designed to create a by-product during the course of regular play. However, such games have traditionally been unable to deal with requests for specific data, relying instead on a broad capture of data in the hope that it will cover specific needs. A new method is needed to focus the efforts of human computation and produce specifically requested results. This would make human computation a more valuable and versatile technique. Mutually reinforcing systems are a new approach to human computation that tries to attain this focus. Ordinary human computation systems tend to work in isolation and do not work directly with each other. Mutually reinforcing systems are an attempt to allow multiple human computation systems to work together so that each can benefit from the other's strengths. For example, a non-game system can request specific data from a game. The game can then tailor its game-play to deliver the required by-products from the players. This is also beneficial to the game because the requests become game content, creating variety in the game-play which helps to prevent players getting bored of the game. Mobile systems provide a particularly good test of human computation because they allow users to react to their environment. Real world environments are changeable and require higher levels of adaptation from the users. This means that, in addition to the human computation required by other systems, mobile systems can also take advantage of a user's ability to apply environmental context to the computational task. This research explores the effects of mutually reinforcing systems on mobile games with by-products. These effects will be explored by building and testing mutually reinforcing systems, including mobile games. A review of existing literature, human computation systems and games with by-products will set out problems which exist in outsourcing parts of a computational process to humans. Mutually reinforcing systems are presented as one approach of addressing some of these problems. Example systems have been created to demonstrate the successes and failures of this approach and their evolving designs have been documented. The evaluation of these systems will be presented along with a discussion of the outcomes and possible future work. A conclusion will summarize the findings of the work carried out. This dissertation shows that extending human computation techniques to allow the collection and classification of useful contextual information in mobile environments is possible and can be extended to allow the by-products to match the specific needs of another system

Digging out implicit semantics from user interaction

User interaction may take many forms in multimedia systems. Current systems mainly waste this implicit and natural source of semantic knowledge and rather create tedious and unnatural interaction protocols. We advocate for a complete integration of natural interaction protocols and semantic knowledge capture, mainly thru mining interaction sessions. We assert that users possess the ability to quickly examine and summarise these documents, even subconsciously. Examples include specifying relevance between a query and results, rating preferences in film databases, purchasing items from online retailers, and even simply browsing web sites. Data from these interactions, captured and stored in log files, can be interpreted to have semantic meaning, which proves indispensable when used in a collaborative setting where users share similar preferences and goals