SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents

Humans are social beings; we pursue social goals in our daily interactions, which is a crucial aspect of social intelligence. Yet, AI systems' abilities in this realm remain elusive. We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social intelligence. In our environment, agents role-play and interact under a wide variety of scenarios; they coordinate, collaborate, exchange, and compete with each other to achieve complex social goals. We simulate the role-play interaction between LLM-based agents and humans within this task space and evaluate their performance with a holistic evaluation framework called SOTOPIA-Eval. With SOTOPIA, we find significant differences between these models in terms of their social intelligence, and we identify a subset of SOTOPIA scenarios, SOTOPIA-hard, that is generally challenging for all models. We find that on this subset, GPT-4 achieves a significantly lower goal completion rate than humans and struggles to exhibit social commonsense reasoning and strategic communication skills. These findings demonstrate SOTOPIA's promise as a general platform for research on evaluating and improving social intelligence in artificial agents.Comment: Preprint, 43 pages. The first two authors contribute equall

Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithms

Social media is often criticized for amplifying toxic discourse and discouraging constructive conversations. But designing social media platforms to promote better conversations is inherently challenging. This paper asks whether simulating social media through a combination of Large Language Models (LLM) and Agent-Based Modeling can help researchers study how different news feed algorithms shape the quality of online conversations. We create realistic personas using data from the American National Election Study to populate simulated social media platforms. Next, we prompt the agents to read and share news articles - and like or comment upon each other's messages - within three platforms that use different news feed algorithms. In the first platform, users see the most liked and commented posts from users whom they follow. In the second, they see posts from all users - even those outside their own network. The third platform employs a novel "bridging" algorithm that highlights posts that are liked by people with opposing political views. We find this bridging algorithm promotes more constructive, non-toxic, conversation across political divides than the other two models. Though further research is needed to evaluate these findings, we argue that LLMs hold considerable potential to improve simulation research on social media and many other complex social settings

Customer interactions with AI: How can Marley Spoon optimize its chatbot performance to improve the touchpoint experience along the customer journey?

The purpose of this in-company project is to identify chatbot optimization recommendations for Marley Spoon to improve the touchpoint experience along the customer journey. Customer interactions with Artificial Intelligence became a relevant part of communication channels within business processes and are already applied in many marketing strategies. Grace to its machine learning capability, chatbots can combine natural language processing and natural language understanding in order to offer an automated customer experience. Nowadays, AI-chatbots are not only able to operate on a mechanical and thinking level, but are also developing on a feeling level. Hence, chatbots can also understand human emotions and adapt empathically to different moods and circumstances. In this way, a well implemented chatbot should not only be used as a simple FAQ machine, but also be implemented for different marketing purposes such as customer attraction and retention. The results of this research are based on a profound literature review with recent articles of well-respected researchers in this field. Moreover, a primary research was conducted in form of in-depth interviews with different specialist of the company and a customer satisfaction survey collected by the chatbot platform. Deriving from the findings of this research, there are three recommendations provided to the company, which should be implemented to improve the touchpoint experience. Those three implementations should be a be a new chatbot interface with more customer engagement, integrating the chatbot to different customer journey stages and setting up a chatbot superteam with specified scope and responsibilities.O objetivo deste projeto em empresa é identificar recomendações de otimização de chatbot para Marley Spoon, de modo a melhorar a experiência touchpoint ao longo da jornada do cliente. As interações dos clientes com a Inteligência Artificial tornaram-se uma parte crucial dos canais de comunicação integradas nos processos de negócios, sendo que já estão a ser aplicadas em muitas estratégias de marketing. Devido à capacidade de aprendizagem, os chatbots podem combinar processamento de linguagem natural com compreensão de linguagem natural, de maneira a oferecer uma experiência automatizada ao cliente. Nos dias de hoje, os AI-chatbots não só são capazes de operar num nível mecânico e de pensamento, mas também estão desenvolvidos a nível de sentimento. Os chatbots podem inclusivamente entender as emoções humanas e adaptar-se efetivamente diferentes estados de espírito e circunstâncias. Desta forma, um chatbot eficiente não deve ser usado apenas como uma simples máquina de resposta a perguntas frequentes, mas também deve ser utilizado para diferentes fins de marketing, como a atração e retenção de clientes. Os resultados da pesquisa foram retirados da análise de conceitos teóricos da literatura científica, focada em artigos recentes de investigadores referenciados nessa área. A pesquisa primária foi realizada em forma de entrevistas com diferentes especialistas da empresa e, também, através de uma pesquisa de satisfação de cliente na plataforma chatbot. Com base nos resultados desta pesquisa, há três recomendações facultadas à empresa, que devem ser implementadas para melhorar a experiência touchpoint. Uma nova interface chatbot com mais comprometimento com o cliente, integrando o chatbot em diferentes estágios da jornada do cliente e configurando uma superteam chatbot com intuito e responsabilidades especificados

THE RELATIONSHIP BETWEEN HUMAN AND VIRTUAL AGENTS: A LIFE CYCLE VIEW

Virtual agents powered by artificial intelligence (AI) have been implemented in different service contexts, which have brought some changes to our lives. Previous studies have examined individual users\u27 motivations to use virtual agents and the influences of virtual agents as social objects on individual users. There is a lack of knowledge on the relationship between humans and virtual agents, which could help understand the role of virtual agents in societies. In this work, we chose the mobile app Replika as our research context and utilized the big data analysis method to explore the major topics covered in online reviews about Replika on Twitter. Based on social penetration theory, we found four relationships between users and Replika, including relationship formation, exploration, maintenance, and destruction or termination. Our findings contribute to the literature by unrevealing a life circle of the relationship between human and virtual agents

Robust Dialog Management Through A Context-centric Architecture

This dissertation presents and evaluates a method of managing spoken dialog interactions with a robust attention to fulfilling the human user’s goals in the presence of speech recognition limitations. Assistive speech-based embodied conversation agents are computer-based entities that interact with humans to help accomplish a certain task or communicate information via spoken input and output. A challenging aspect of this task involves open dialog, where the user is free to converse in an unstructured manner. With this style of input, the machine’s ability to communicate may be hindered by poor reception of utterances, caused by a user’s inadequate command of a language and/or faults in the speech recognition facilities. Since a speech-based input is emphasized, this endeavor involves the fundamental issues associated with natural language processing, automatic speech recognition and dialog system design. Driven by ContextBased Reasoning, the presented dialog manager features a discourse model that implements mixed-initiative conversation with a focus on the user’s assistive needs. The discourse behavior must maintain a sense of generality, where the assistive nature of the system remains constant regardless of its knowledge corpus. The dialog manager was encapsulated into a speech-based embodied conversation agent platform for prototyping and testing purposes. A battery of user trials was performed on this agent to evaluate its performance as a robust, domain-independent, speech-based interaction entity capable of satisfying the needs of its users

Positive Versus Negative Agents: The Effects of Emotions on Learning

The current study investigates the impact of affect, mood contagion, and linguistic alignment on learning during tutorial conversations between a human student and two artificial pedagogical agents. The study uses an Intelligent Tutoring System known as OperationARIES! to engage students in tutorial conversations with animated agents. In this investigation, 48 college students (N = 48) conversed with pedagogical agents as they displayed 3 different moods (i.e., positive, negative, and neutral) along with a control condition in a within-subjects design. Results indicate that the mood of the agent did not significantly impact student learning even though mood contagion did occur between the artificial agent and the human student. Learning was influenced by the student\u27s self-reported arousal level and the alignment scores that reflected a shared mental representation between the human student and the artificial agents. The results suggest that arousal and linguistic alignment during the tutorial conversations may play a role in learning

Let\u27s Talk About (Cred)it

This thesis observes the credit system’s communicative patterns and stigmas that have fueled consumer ignorance and discouraged valuable financial discourse and education. For generations, credit consumers have been kept in the dark from imperative personal finance topics. Parents and guardians are not having conversations with their children about credit at home, and neither are a majority of educators. The lack of credit communication at home and in the classroom has ultimately created a massive population that is vulnerable to the manipulation of credit issuers and companies. Consumer ignorance and vulnerability due to the exclusion of credit from conversations and curriculums have created damaging communicative patterns and stigmas surrounding credit and personal finance. These communicative patterns and stigmas have created emotional and material barriers that barricade current and future consumers from financial enlightenment and freedom. We as a society must remove these barriers by communicating with children and educating adolescents on the realities of the credit system and personal finance. So, let’s talk about (cred)it

BlogForever D3.3: Development of the Digital Rights Management Policy

This report presents a set of recommended practices and approaches that a future BlogForever repository can use to develop a digital rights management policy. The report outlines core legal aspects of digital rights that might need consideration in developing policies, and what the challenges are, in particular, in relation to web archives and blog archives. These issues are discussed in the context of the digital information life cycle and steps that might be taken within the workflow of the BlogForever platform to facilitate the gathering and management of digital rights information. Further, the reports on interviews with experts in the field highlight current perspectives on rights management and provide empirical support for the recommendations that have been put forward

\u27Just Through Talking\u27: A Collaborative Learning Approach for Human Resource Change Agents

My purpose in conducting this research project was to engage in collaborative action research with a group of human resource professionals in order to investigate the role of a human resource professional as an organizational change agent, and how participating in a collaborative learning group focused on change might inform our practice. We used dialogue during our collaborative learning group meetings to share professional experiences, better understand our own assumptions and the assumptions of others in our group, and for sensemaking about our profession. The data analysis focused in two areas: 1) describing what the experience as an organizational change agent was like for the participants, and 2) describing how the experience of participating in a collaborative learning group informed our practice. Additionally, a model presenting a collaborative learning approach for human resource change agents is provided. We concluded that changes in our practice did occur as a result of personal insights and growth experienced in action research and collaborative learning. Five themes related to our experience as organizational change agents reflect the ways in which we were able to better understand our practice. The themes were: change is personal – “one conversation at a time”; struggles and frustrations – “puts you in the weeds”; approach – “soft or back-door”; trust – “open and honest conversation”; and results – “where the rubber meets the road.” Through participation in the collaborative learning group, we not only had a better understanding of ourselves and others in the group, but were also able to identify and reflect on our theories-in-action, making explicit what was implicit. These five themes: “there is a process”; “suspending judgment”; “getting hold of our own change”; “just through talking”; and “safe and understanding environment” were related to the group members’ attempt to “make sense” or better understand ourselves, others, and our work environment

Conversational Agent: Developing a Model for Intelligent Agents with Transient Emotional States

The inclusion of human characteristics (i.e., emotions, personality) within an intelligent agent can often increase the effectiveness of information delivery and retrieval. Chat-bots offer a plethora of benefits within an eclectic range of disciplines (e.g., education, medicine, clinical and mental health). Hence, chatbots offer an effective way to observe, assess, and evaluate human communication patterns. Current research aims to develop a computational model for conversational agents with an emotional component to be applied to the army leadership training program that will allow for the examination of interpersonal skills in future research. Overall, the current research explores the application of the deep learning algorithm to the development of a generalized framework that will be based upon modeling empathetic conversation between an intelligent conversational agent (chatbot) and a human user in order to allow for higher level observation of interpersonal communication skills. Preliminary results demonstrate the promising potential of the seq2seq technique (e.g., through the use of Dialog Flow Chatbot platform) when applied to emotion-oriented conversational tasks. Both the classification and generative conversational modeling tasks demonstrate the promising potential of the current research for representing human to agent dialogue. However, this implementation may be extended by utilizing, a larger more high-quality dataset