Playing for Data: Ground Truth from Computer Games

Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set.Comment: Accepted to the 14th European Conference on Computer Vision (ECCV 2016

Color terms: Native language semantic structure and artificial language structure formation in a large-scale online smartphone application

Artificial language games give researchers the opportunity to investigate the emergence and evolution of semantic structure, i.e. the organization of meaning spaces into discrete categories. A possible issue for this approach is that categories might simply carry over from participants’ native languages, a potential bias that has mostly been ignored. We investigate this in a referential communication game by comparing color terms from three different languages to those of an artificial language. Here, we assess the similarity of the semantic structures, and test the influence of the semantic structure on artificial language communication. We compare the in-game communication to a separate online naming task providing us with the native language structure. Our results show that native and artificial language structure overlap at least moderately. Furthermore, communicative behavior and performance were influenced by the shared semantic structure, but only for English-speaking pairs. These results imply a cognitive link between participants’ semantic structures and artificial language structure formation.1. Introduction - Artificial language games, semantic structure, and possible biases - Color terms and categorical facilitation 2. Method - The Color Game -- Participants -- Materials -- Procedure - Online survey -- Participants -- Materials -- Procedure - Predictions 3. Results - Prediction 1 - Prediction 2.1 - Prediction 2.2 - Prediction 2.3 - Prediction 3 4. Discussion 5. Conclusio

Energy-Efficient Semantic Communication for Aerial-Aided Edge Networks

Semantic communication holds promise for integration into future wireless networks, offering a potential enhancement in network spectrum efficiency. However, implementing semantic communication in aerial-aided edge networks (AENs) introduces unique challenges. Within AENs, semantic communication strategically substitutes part of the communication load with the computation load, aiming to boost spectrum efficiency. This departure from traditional communication paradigms introduces novel challenges, particularly in terms of energy efficiency. Furthermore, by adding complexity, the use of a semantic coder based on machine learning (ML) in AENs encounters real-time updating challenges, further amplifying energy costs in these complex and energy-limited environments. To address these challenges, we propose an energy-efficient semantic communication system tailored for AENs. Our approach includes a mathematical analysis of semantic communication energy consumption within AENs. To enhance energy efficiency, we introduce an energy-efficient game-theoretic incentive mechanism (EGTIM) designed to optimize semantic transmission within AENs. Moreover, considering the accurate and energy-efficient updating of semantic coders in AENs, we present a game-theoretic efficient distributed learning (GEDL) framework, building upon the foundations of the renewed EGTIM. Simulation results validate the effectiveness of our proposed EGTIM in improving energy efficiency. Additionally, the presented GEDL framework exhibits remarkable performance by increasing model training accuracy and concurrently decreasing training energy consumption

Games of Partial Information and Predicates of Personal Taste

A predicate of personal taste occurring in a sentence in which the perspectival information is not linguistically articulated by an experiencer phrase may have two different readings. In case the speaker of a bare sentence formed with a predicate of personal taste uses the subjective predicate encoding perspectival information in one way and the hearer interprets it in another way, the agents’ acts are not coordinated. In this paper I offer an answer to the question of how a hearer can strategically interact with a speaker on the intended perspectival information so that both agents can optimally solve their coordination problem. In this sense, I offer a game-theoretical account of the strategic communication with expressions referring to agents’ perspectives, communication which involves the interaction between a speaker who intends to convey some perspectival information and who chooses to utter a bare sentence formed with a predicate of personal taste, instead of a sentence in which the perspectival information is linguistically articulated by an experiencer phrase, and a hearer who has to choose between interpreting the uttered sentence in conformity with the speaker’s autocentric use of the predicate of personal taste or in conformity with the speaker’s exocentric use

A Praxical Solution of the Symbol Grounding Problem

Peer reviewe

Translating Neuralese

Several approaches have recently been proposed for learning decentralized deep multiagent policies that coordinate via a differentiable communication channel. While these policies are effective for many tasks, interpretation of their induced communication strategies has remained a challenge. Here we propose to interpret agents' messages by translating them. Unlike in typical machine translation problems, we have no parallel data to learn from. Instead we develop a translation model based on the insight that agent messages and natural language strings mean the same thing if they induce the same belief about the world in a listener. We present theoretical guarantees and empirical evidence that our approach preserves both the semantics and pragmatics of messages by ensuring that players communicating through a translation layer do not suffer a substantial loss in reward relative to players with a common language.Comment: Fixes typos and cleans ups some model presentation detail