Playing for Data: Ground Truth from Computer Games

Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set.Comment: Accepted to the 14th European Conference on Computer Vision (ECCV 2016

Color terms: Native language semantic structure and artificial language structure formation in a large-scale online smartphone application

Artificial language games give researchers the opportunity to investigate the emergence and evolution of semantic structure, i.e. the organization of meaning spaces into discrete categories. A possible issue for this approach is that categories might simply carry over from participants’ native languages, a potential bias that has mostly been ignored. We investigate this in a referential communication game by comparing color terms from three different languages to those of an artificial language. Here, we assess the similarity of the semantic structures, and test the influence of the semantic structure on artificial language communication. We compare the in-game communication to a separate online naming task providing us with the native language structure. Our results show that native and artificial language structure overlap at least moderately. Furthermore, communicative behavior and performance were influenced by the shared semantic structure, but only for English-speaking pairs. These results imply a cognitive link between participants’ semantic structures and artificial language structure formation.1. Introduction - Artificial language games, semantic structure, and possible biases - Color terms and categorical facilitation 2. Method - The Color Game -- Participants -- Materials -- Procedure - Online survey -- Participants -- Materials -- Procedure - Predictions 3. Results - Prediction 1 - Prediction 2.1 - Prediction 2.2 - Prediction 2.3 - Prediction 3 4. Discussion 5. Conclusio

Games of Partial Information and Predicates of Personal Taste

A predicate of personal taste occurring in a sentence in which the perspectival information is not linguistically articulated by an experiencer phrase may have two different readings. In case the speaker of a bare sentence formed with a predicate of personal taste uses the subjective predicate encoding perspectival information in one way and the hearer interprets it in another way, the agents’ acts are not coordinated. In this paper I offer an answer to the question of how a hearer can strategically interact with a speaker on the intended perspectival information so that both agents can optimally solve their coordination problem. In this sense, I offer a game-theoretical account of the strategic communication with expressions referring to agents’ perspectives, communication which involves the interaction between a speaker who intends to convey some perspectival information and who chooses to utter a bare sentence formed with a predicate of personal taste, instead of a sentence in which the perspectival information is linguistically articulated by an experiencer phrase, and a hearer who has to choose between interpreting the uttered sentence in conformity with the speaker’s autocentric use of the predicate of personal taste or in conformity with the speaker’s exocentric use

A Praxical Solution of the Symbol Grounding Problem

Peer reviewe

Translating Neuralese

Several approaches have recently been proposed for learning decentralized deep multiagent policies that coordinate via a differentiable communication channel. While these policies are effective for many tasks, interpretation of their induced communication strategies has remained a challenge. Here we propose to interpret agents' messages by translating them. Unlike in typical machine translation problems, we have no parallel data to learn from. Instead we develop a translation model based on the insight that agent messages and natural language strings mean the same thing if they induce the same belief about the world in a listener. We present theoretical guarantees and empirical evidence that our approach preserves both the semantics and pragmatics of messages by ensuring that players communicating through a translation layer do not suffer a substantial loss in reward relative to players with a common language.Comment: Fixes typos and cleans ups some model presentation detail

Revisit the Concept of PEKS: Problems and a Possible Solution

Since Boneh et al. propose the concept, non-interactive\ud Public-key Encryption with Keyword Search (PEKS) has attracted lots of attention from cryptographers. Non-interactive PEKS enables a third party to test whether or not a tag, generated by the message sender, and a trapdoor, generated by the receiver, contain the same keyword without revealing further information. In this paper we investigate a non-interactive PEKS application proposed by Boneh et al. and show our observations, especially that privacy is\ud not protected against a curious server. We propose the notion of interactive PEKS, which, in contrast to non-interactive PEKS, requires the tag to be generated interactively by the message sender and the receiver. For this new primitive, we identify two types of adversaries, namely a curious user and a curious server, and provide\ud security formulations for the desirable properties. We propose a construction for interactive PEKS and prove its security in the proposed security model