1,684 research outputs found

    Towards collaborative dialogue in Minecraft

    Get PDF
    This dissertation describes our work in building interactive agents that can communicate with humans to collaboratively solve tasks in grounded scenarios. To investigate the challenges of building such agents, we define a novel instantiation of a situated, Minecraft-based, Collaborative Building Task in which one player (A, the Architect) is shown a target structure, denoted Target, and needs to instruct the other player (B, the Builder) to build a copy of this structure, denoted Built, in a predefined build region. While both players can interact asynchronously via a chat interface, we define the roles to be asymmetric: A can observe B and Target, but is invisible and cannot place blocks; meanwhile, B can freely place and remove blocks, but has no explicit knowledge of the target structure. Each agent requires a different set of abilities in order to be successful at this task: specifically, A's main challenges arise in the task of generating situated instructions by comparing Built and Target, while B's responsibilities focus mainly on comprehending A's situated instructions using both dialogue and world context. Both agents must be able to interact asynchronously in an evolving dialogue context and a dynamic world state within which they are embodied. In this work, we specifically examine how well end-to-end neural models can learn to be instruction givers (i.e., Architects) from a limited amount of real human-human data. In order to examine how humans complete the Collaborative Building Task, as well as use human-human data as a gold standard for training and evaluating models, we present the Minecraft Dialogue Corpus, a collection of 509 conversations and game logs. We then introduce baseline models for the challenging subtask of Architect utterance generation, and evaluate them offline, using both automated metrics and human evaluation. We show that while conditioning our model on a simple representation of the world gives our model improved ability to generate correct instructions, there are still many obvious shortcomings, and it is difficult for these models to learn the large variety of abilities needed to be successful Architects in an entirely end-to-end manner. To combat this, we show that including meaningful, structured inputs about the world and discourse state as additional inputs -- specifically, by adding oracle information about the Builder's next actions, as well as enriching our linguistic representation with Architect dialogue acts -- improves the performance of our utterance generation models. We also augment the data with shape information by pretraining 3D shape localization models on synthetically generated block configurations. Finally, we integrate Architect utterance generation models into actual Minecraft agents and evaluate them in a fully interactive setting

    Learning to execute or ask clarification questions

    Get PDF
    Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. In order to achieve this goal, such agents need to be able to take the initiative by asking clarification questions when further information is needed. Existing works on Minecraft Corpus Dataset only learn to execute instructions neglecting the importance of asking for clarifications. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clarification questions, and propose a new builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also define two new tasks, the learning to ask task and the joint learning task. The latter consists of solving both collaborating building and learning to ask tasks jointly

    Study of team building based on 3D game

    Get PDF
    Objective: To determine the effect of 3D-game team building on team performance and team interaction. Subjects and Methods: Between April and May 2015, 13 teams including 39 adult participants were randomly assigned to take designed TBE based on Minecraft, dialogue technique or free conversation no more than 15 minutes. Then they are required to work out a 3D puzzle. The puzzle completion time was recorded. Participants rated the team learning and interaction via TLQ after each exercise or task. Results: Teams who experienced 3D-game TBE didn’t show significant higher performance in terms of TLQ scores and puzzle completion time than those who did not .There were small significant differences in terms of dialog promotion and open communication, and collaborative learning between 3D-game and dialog tech teams. Conclusion: The effect of 3D-game TBE on TLQ scores and puzzle completion time was not significant, but it influenced the stability of puzzle completion time. The designed 3D-game team building focus on enhancing teammates’ willing of communication while dialogue technique focused on creating the atmosphere of collaborative learning

    Exploring initial collaboration in an intervention: Creating a meeting place between educational research and educational practice

    Get PDF
    In the field of education, researchers have focused on the importance of achieving a common understanding of school development and change of practice in collaborations with practitioners. In an attempt to contribute to this research, a formative Change Laboratory intervention is suggested as an interface between the researcher’s world and the practitioner’s world to facilitate collaboration between the two. The case study, conducted in one mathematics class in a primary school with 27 students and two teachers, was informed by the following research question: How does initial collaboration between a researcher and practitioners create a meeting place, and what implications can be drawn from this? The teachers’ motive for joining the intervention was to expand their practice of using the digital game Minecraft. The collaboration lasted 1.5 years. The findings show that e-mail correspondence seems to play a crucial role in the continuation and expansion of dialogue towards achieving an object-oriented activity.publishedVersio

    Towards a Neural Era in Dialogue Management for Collaboration: A Literature Survey

    Full text link
    Dialogue-based human-AI collaboration can revolutionize collaborative problem-solving, creative exploration, and social support. To realize this goal, the development of automated agents proficient in skills such as negotiating, following instructions, establishing common ground, and progressing shared tasks is essential. This survey begins by reviewing the evolution of dialogue management paradigms in collaborative dialogue systems, from traditional handcrafted and information-state based methods to AI planning-inspired approaches. It then shifts focus to contemporary data-driven dialogue management techniques, which seek to transfer deep learning successes from form-filling and open-domain settings to collaborative contexts. The paper proceeds to analyze a selected set of recent works that apply neural approaches to collaborative dialogue management, spotlighting prevailing trends in the field. This survey hopes to provide foundational background for future advancements in collaborative dialogue management, particularly as the dialogue systems community continues to embrace the potential of large language models

    Using Video Games to Develop Graduate Attributes: a Pilot Study

    Get PDF
    It may be argued that most higher education courses are not explicitly designed to teach or develop desirable soft skills such as critical thinking, communication, resourcefulness or adaptability. While such skills – often referred to as ‘graduate attributes’ – are assumed to be developed as a by-product of a university education, there is little empirical evidence to support this assumption. Furthermore, traditional didactic teaching methods do not typically require students to exhibit such skills, while prevalent assessment methods such as examinations are ill-suited to measure them. Many commercial video games, on the other hand, require players to exercise a range of very similar skills and competencies in order to progress. The pilot project described here sought to explore the use of video games to develop graduate attributes and to identify suitable instruments for measuring such elusive conceptions. A small group of undergraduate students were recruited and asked to play selected video games for two hours per week over an eight week period. A range of psychometric tests were administered at the beginning and the end of the experiment period in order to gather empirical data relating to the participants’ graduate attributes. Mean differences in the pre- and post-intervention scores associated with each measure were obtained and 95% confidence intervals calculated to provide an indication of whether results obtained might be indicative of a wider population. Participants were also asked to discuss their experience as a group following each session and to blog about it if they were so inclined. Despite the small scale of the pilot, the results were sufficiently encouraging to warrant a larger study, which is now underway. The challenges involved in obtaining empirical data on the effectiveness of a game-based intervention such as this are addressed and implications for the subsequent study are discussed

    Teachers’ framing and dialogic facilitation of Minecraft in the L1 classroom

    Get PDF
    • …
    corecore