인공지능과 대화하기: 일대일 그리고 그룹 상용작용을 위한 대화형 에이전트 시스템 개발

학위논문(박사) -- 서울대학교대학원 : 사회과학대학 언론정보학과, 2022.2. 이준환."인간-컴퓨터 상호작용"과 "사용자 경험"을 넘어, "인간-인공지능 상호작용" 그리고 "알고리즘 경험"의 시대가 도래하고 있다. 기술의 발전은 우리가 의사소통하고 협업하는 방식의 패러다임을 전환했다. 기계 에이전트는 인간 커뮤니케이션에서 적극적이며 주도적인 역할을 수행한다. 하지만 효과적인 AI 기반 커뮤니케이션과 토론 시스템 디자인에 대한 이해와 논의는 부족한 것이 사실이다. 이에 본 연구는 인간-컴퓨터 상호작용의 관점에서 다양한 형태의 커뮤니케이션을 지원할 수 있는 기술적 방법을 탐색하는 것을 목표로 한다. 이를 위해 저자는 일대일 그리고 그룹 상호작용을 지원하는 대화형 에이전트를 제시한다. 구체적으로 본 연구는 1) 일대일 상호작요에서 사용자 관여를 높이는 대화형 에이전트, 2) 일상적인 소셜 그룹 토론을 지원하는 에이전트, 3) 숙의 토론을 가능하게 하는 에이전트를 디자인 및 개발하고 그 효과를 정량적 그리고 정성적으로 검증했다. 시스템을 디자인함에 있어서 인간-컴퓨터 상호작용뿐 아니라, 커뮤니케이션학, 심리학, 그리고 데이터 과학을 접목한 다학제적 접근 방식이 적용되었다. 첫 번째 연구는 일대일 상호작용 상황에서 사용자의 관여 증진을 위한 대화형 에이전트의 효과를 검증했다. 설문조사라는 맥락에서 수행된 이 연구는 웹 설문조사에서 응답자의 불성실로 인해 발생하는 응답 데이터 품질의 문제를 극복하기 위한 새로운 인터랙션 방법으로 텍스트 기반 대화형 에이전트의 가능성을 탐색하는 것을 목표로 했다. 이를 위해 2 (인터페이스: 웹 對 챗봇) X 2 (대화 스타일: 포멀 對 캐쥬얼) 실험을 진행했으며, 만족화 이론에 근거하여 응답 데이터의 품질을 평가했다. 그 결과, 챗봇 설문조사의 참여자가 웹 설문조사의 참여자보다 더 높은 수준의 관여를 보이고, 결과적으로 더 높은 품질의 데이터를 생성하는 것을 확인할 수 있었다. 하지만 이런 챗봇의 데이터 품질에 대한 효과는 챗봇이 친구 같고 캐쥬얼한 대화체를 사용할 때만 나타났다. 이 결과는 대화형 인터랙티비티가 인터페이스뿐 아니라 대화 스타일이라는 효과적인 메세지 전략을 동반할 때 발생하는 것을 의미한다. 두 번째 연구는 일상적인 소셜 채팅 그룹에서 집단의 의사결정과정과 토론을 지원하는 대화형 시스템에 대한 것이다. 이를 위해 GroupfeedBot이라는 대화형 에이전트를 제작하였으며, GroupfeedBot은 (1) 토론 시간을 관리하고, (2) 구성원들의 균등한 참여를 촉진하며, (3) 구성원들의 다양한 의견을 요약 및 조직화하는 기능을 갖고 있다. 해당 에이전트를 평가하기 위해 다양한 태스크 (추론, 의사결정, 자유 토론, 문제 해결 과제)와 그룹 규모(소규모, 중규모)에 관하여 사용자 조사를 시행했다. 그 결과 의견의 다양성 측면에서 GroupfeedBot으로 토론한 집단이 기본 에이전트와 토론한 집단보다 더 다양한 의견을 생성했지만 산출된 결과의 품질과 메시지 양에 있어서는 차이가 없는 것을 확인할 수 있었다. 균등한 참여에 대한 GroupfeedBot의 효과는 태스크의 특성에 따라 다르게 나타났는데, 특히 자유 토론 과제에서 GroupfeedBot이 참여자들의 균등한 참여를 촉진했다. 세 번째 연구는 숙의 토론을 지원하는 대화형 시스템에 대한 것이다. 세 번째 연구에서 개발된 DebateBot은 GroupfeeedBot과 달리 더 진지한 사회적 맥락에서 적용되었다. DebateBot은 (1) 생각하기-짝짓기-공유하기 (Think-Pair-Share) 전략에 따라 토론을 구조화하고, (2) 과묵한 토론자에게 의견을 요청함으로써 동등한 참여를 촉진하는 두 가지 주요 기능을 수행했다. 사용자 평가 결과 DebateBot은 그룹 상호작용을 개선함으로써 심의 토론을 가능하게 했다. 토론 구조화는 토론의 질에 긍정적인 효과를 발휘하였고, 참여자 촉진은 진정한 합의 도달에 기여하였으며, 그룹 구성원들의 주관적 만족도를 향상했다. 본 연구는 이 세 가지 연구의 결과들을 바탕으로 인간-인공지능 커뮤니케이션에 대한 다양한 시사점들을 도출하였으며, 이를 TAMED (Task-Agent-Message-Information Exchange-Relationship Dynamics) 모델로 정리하였다.The advancements in technology shift the paradigm of how individuals communicate and collaborate. Machines play an active role in human communication. However, we still lack a generalized understanding of how exactly to design effective machine-driven communication and discussion systems. How should machine agents be designed differently when interacting with a single user as opposed to when interacting with multiple users? How can machine agents be designed to drive user engagement during dyadic interaction? What roles can machine agents perform for the sake of group interaction contexts? How should technology be implemented in support of the group decision-making process and to promote group dynamics? What are the design and technical issues which should be considered for the sake of creating human-centered interactive systems? In this thesis, I present new interactive systems in the form of a conversational agent, or a chatbot, that facilitate dyadic and group interactions. Specifically, I focus on: 1) a conversational agent to engage users in dyadic communication, 2) a chatbot called GroupfeedBot that facilitates daily social group discussion, 3) a chatbot called DebateBot that enables deliberative discussion. My approach to research is multidisciplinary and informed by not only in HCI, but also communication, psychology and data science. In my work, I conduct in-depth qualitative inquiry and quantitative data analysis towards understanding issues that users have with current systems, before developing new computational techniques that meet those user needs. Finally, I design, build, and deploy systems that use these techniques to the public in order to achieve real-world impact and to study their use by different usage contexts. The findings of this thesis are as follows. For a dyadic interaction, participants interacting with a chatbot system were more engaged as compared to those with a static web system. However, the conversational agent leads to better user engagement only when the messages apply a friendly, human-like conversational style. These results imply that the chatbot interface itself is not quite sufficient for the purpose of conveying conversational interactivity. Messages should also be carefully designed to convey such. Unlike dyadic interactions, which focus on message characteristics, other elements of the interaction should be considered when designing agents for group communication. In terms of messages, it is important to synthesize and organize information given that countless messages are exchanged simultaneously. In terms of relationship dynamics, rather than developing a rapport with a single user, it is essential to understand and facilitate the dynamics of the group as a whole. In terms of task performance, technology should support the group's decision-making process by efficiently managing the task execution process. Considering the above characteristics of group interactions, I created the chatbot agents that facilitate group communication in two different contexts and verified their effectiveness. GroupfeedBot was designed and developed with the aim of enhancing group discussion in social chat groups. GroupfeedBot possesses the feature of (1) managing time, (2) encouraging members to participate evenly, and (3) organizing the members’ diverse opinions. The group which discussed with GroupfeedBot tended to produce more diverse opinions compared to the group discussed with the basic chatbot. Some effects of GroupfeedBot varied by the task's characteristics. GroupfeedBot encouraged the members to contribute evenly to the discussions, especially for the open-debating task. On the other hand, DebateBot was designed and developed to facilitate deliberative discussion. In contrast to GroupfeedBot, DebateBot was applied to more serious and less casual social contexts. Two main features were implemented in DebateBot: (1) structure discussion and (2) request opinions from reticent discussants.This work found that a chatbot agent which structures discussions and promotes even participation can improve discussions, resulting in higher quality deliberative discussion. Overall, adding structure to the discussion positively influenced the discussion quality, and the facilitation helped groups reach a genuine consensus and improved the subjective satisfaction of the group members. The findings of this thesis reflect the importance of understanding human factors in designing AI-infused systems. By understanding the characteristics of individual humans and collective groups, we are able to place humans at the heart of the system and utilize AI technology in a human-friendly way.1. Introduction 1.1 Background 1.2 Rise of Machine Agency 1.3 Theoretical Framework 1.4 Research Goal 1.5 Research Approach 1.6 Summary of Contributions 1.7 Thesis Overview 2. Related Work 2.1 A Brief History of Conversational Agents 2.2 TAMED Framework 3. Designing Conversational Agents for Dyadic Interaction 3.1 Background 3.2 Related Work 3.3 Method 3.4 Results 3.5 Discussion 3.6 Conclusion 4. Designing Conversational Agents for Social Group Discussion 4.1 Background 4.2 Related Work 4.3 Needfinding Survey for Facilitator Chatbot Agent 4.4 GroupfeedBot: A Chatbot Agent For Facilitating Discussion in Group Chats 4.5 Qualitative Study with Small-Sized Group 4.6 User Study With Medium-Sized Group 4.7 Discussion 4.8 Conclusion 5. Designing Conversational Agents for Deliberative Group Discussion 5.1 Background 5.2 Related Work 5.3 DebateBot 5.4 Method 5.5 Results 5.6 Discussion and Design Implications 5.7 Conclusion 6. Discussion 6.1 Designing Conversational Agents as a Communicator 6.2 Design Guidelines Based on TAMED Model 6.3 Technical Considerations 6.4 Human-AI Collaborative System 7. Conclusion 7.1 Research Summary 7.2 Summary of Contributions 7.3 Future Work 7.4 Conclusion박

Understanding chatbot service encounters:consumers’ satisfactory and dissatisfactory experiences

Abstract. The service industry keeps growing these years. Artificial intelligence (AI) has started to be used in the service industry gradually, and the service chatbot is an excellent example of this phenomenon. Many giants have applied chatbots to handle their consumer services, such as LATTJO from IKEA, Stylebot from Nike, and Siri from Apple. Understanding the advanced chatbot service experiences can help companies to optimize their chatbot services and improve their consumers’ satisfaction, which can bring them positive word-of-mouth, customer loyalty, re-purchase behavior, etc. However, chatbot services is an edge research area with limited studies about it. Thus, having the most advanced understanding of chatbot service experiences becomes particularly important. This study intends to fill this gap from chatbot service encounters’ perspective by understanding consumers’ satisfactory and unsatisfactory experiences with chatbots. Due to this study focuses on chatbot service encounters and online customer service experiences, a qualitative research method be applied because it enables data to be explainable and justifiable. Data collection methods consist of the critical incident technique (CIT) and the online focus group. In the end, 22 validity incidents were collected. Through data analysis, the author developed an incident sorting process and concluded eight types of chatbot service encounters within three groups by this process. The three groups are chatbot response to after-sales services, chatbot response to consumers’ needs, and unprompted chatbot actions. Moreover, 16 sources of different types of chatbot service encounters were found. Based on all the findings stated above, this study created an integrated framework for chatbot service encounters in online customer service experiences. In conclusion, this study develops theoretical contributions by developing the integrated framework, creating an incident sorting process, and finding the sources for different service encounters. Based on these findings, this study also provides some managerial implications that companies could use to manage their chatbot services

The Impact of internet social networking websites on the gay community: Behavior and identity

The hypothesis of this thesis is that social networking website design can exert a mediating influence upon the culture of a site by supporting certain behaviors more than others; this influence can be analyzed in an active and structured way that takes into account the culture of the community it addresses. Evidence will be offered by case study, demonstration of specific mediations, and analysis. This hypothesis will be tested with specific reference to the gay male community. The scope of this paper will be limited to the analysis of gay-oriented social networking websites as new media, in general and through specific examples. I will present frameworks for categorizing and analyzing these websites that consider the mediating influences associated with site design. In the last chapter, I will propose community-enhancing design. The method of analysis first takes into account the nature of new media. It then discusses the concepts of cultural mediums and mediators in terms of site-wide typology and specific forms of mediation. It then identifies common elements of gay social networking sites and their associated usage as well as the design decisions that are related to them. Next user goals and site goals are correlated to these design decisions. Virtual personas and real communities are discusses as a concept. Using the proposed methodology, gay.com and other sites are analyzed and compared. Conclusions are drawn from the results of this analysis and evidence presented. The impact of social networking websites upon sexual activity is discussed. Finally, conclusions are summarized and recommendations are cited related to what these sites could be

Synchronous computer-mediated communication between foreign language learners and prospective teachers

Technology has proven itself as a valuable contribution to educational practices when used in a pedagogically sound manner. In this quasi-experimental study using the Interaction Hypothesis as a theoretical framework, 11 intermediate college-level learners of French and Russian and prospective language teachers had six 30-minute synchronous online chat sessions completing communicative tasks. Using a mixed-methods approach, both quantitative and qualitative data were included in the analysis. The study provides evidence for the positive effect of synchronous computer-mediated communication on second language vocabulary acquisition and prospective teachers\u27 professional development, and underlines the importance of scaffolding in online environments. Vocabulary gains of the language learners did not hold up over a two-week period, and there was a weak relationship between uptake outcome and vocabulary acquisition

Modeling Human Group Behavior In Virtual Worlds

Virtual worlds and massively-multiplayer online games are rich sources of information about large-scale teams and groups, offering the tantalizing possibility of harvesting data about group formation, social networks, and network evolution. They provide new outlets for human social interaction that differ from both face-to-face interactions and non-physically-embodied social networking tools such as Facebook and Twitter. We aim to study group dynamics in these virtual worlds by collecting and analyzing public conversational patterns of users grouped in close physical proximity. To do this, we created a set of tools for monitoring, partitioning, and analyzing unstructured conversations between changing groups of participants in Second Life, a massively multi-player online user-constructed environment that allows users to construct and inhabit their own 3D world. Although there are some cues in the dialog, determining social interactions from unstructured chat data alone is a difficult problem, since these environments lack many of the cues that facilitate natural language processing in other conversational settings and different types of social media. Public chat data often features players who speak simultaneously, use jargon and emoticons, and only erratically adhere to conversational norms. Humans are adept social animals capable of identifying friendship groups from a combination of linguistic cues and social network patterns. But what is more important, the content of what people say or their history of social interactions? Moreover, is it possible to identify whether iii people are part of a group with changing membership merely from general network properties, such as measures of centrality and latent communities? These are the questions that we aim to answer in this thesis. The contributions of this thesis include: 1) a link prediction algorithm for identifying friendship relationships from unstructured chat data 2) a method for identifying social groups based on the results of community detection and topic analysis. The output of these two algorithms (links and group membership) are useful for studying a variety of research questions about human behavior in virtual worlds. To demonstrate this we have performed a longitudinal analysis of human groups in different regions of the Second Life virtual world. We believe that studies performed with our tools in virtual worlds will be a useful stepping stone toward creating a rich computational model of human group dynamics

Modeling usage of an online research community

Although online communities have been thought of as a new way for collaboration across geographic boundaries in the scientific world, they have a problem attracting people to keep visiting. The main purpose of this study is to understand how people behave in such communities, and to build and evaluate tools to stimulate engagement in a research community. These tools were designed based on a research framework of factors that influence online participation and relationship development. There are two main objectives for people to join an online community, information sharing and interpersonal relationship development, such as friends or colleagues. The tools designed in this study are to serve both information sharing and interpersonal relationship development needs. The awareness tool is designed to increase the sense of a community and increase the degree of social presence of members in the community. The recommender system is designed to help provide higher quality and personalized information to community members. It also helps to match community members into subgroups based on their interests. The designed tools were implemented in a field site - the Asynchronous Learning Networks (ALN) Research community. A longitudinal field study was used to evaluate the effectiveness of the designed tools. This research explored people\u27s behavior inside a research community by analyzing web server logs. The results show that although there are not many interactions in the community space, the WebCenter has been visited extensively by its members. There are over 2,000 hits per day on average and over 5,000 article accesses during the observation period. This research also provided a framework to identify factors that affect people\u27s engagement in an online community. The research framework was tested using the PLS modeling method with online survey responses. The results show that perceived usefulness performs a very significant role in members\u27 intention to continue using the system and their perceived preliminary networking. The results also show that the quality of the content of the system is a strong indicator for both perceived usefulness of the community space and perceived ease of use of the community system. Perceived ease of use did not show a strong correlation with intention to continue use which was consistent with other studies of Technology Acceptance Model (TAM). For the ALN research community, this online community helps its members to broaden their contacts, improve the quality and quantity of their research, and increase the dissemination of knowledge among community members

“It's not numbers, and it's not just usernames”: exploring the mediated relationships between twitch streamers and their viewers

On Twitch, a livestreaming platform, 3.8 million broadcasters, known as ‘streamers’ go live to over 200 million viewers (Ruby, 2023). Initially developed for gaming-centred content, Twitch has since branched out to cater to a wide range of interests, such as live music, art, conversation, and sport. A key component of streaming on Twitch involves communicating with users, who send messages in the chat box that accompanies streams through which streamers and viewers can have two-way conversation in real-time. For streamers that gain a substantial number of viewers, Twitch offers monetisation through first gaining Affiliate status, and then Partner status, allowing streamers to earn money from viewers subscribing to their channels and donating cash tips and Twitch’s own virtual currency of ‘Bits’. Although Twitch is seeing growing academic interest, it has received less attention than other social media and content creation platforms, such as Facebook and YouTube. In addition, much of the prior research on Twitch and livestreaming more broadly centres around the technical aspects of streaming, whilst the relationships which form between streamers and their viewers remain under researched. To address this gap in the literature, this thesis explores the complex relationships that develop between Twitch streamers and their viewers, examining the multifaceted nature of these connections through the analysis of factors that impact their development and maintenance. The data collection for this research commenced with unstructured observation on the platform to learn about how streamers engage with their viewers. From this observation and from the literature on social media and personal relationships, key themes emerged which formed the basis for the semi-structured interviews with 32 streamers. This research focuses specifically on the experiences of women and non-binary streamers on Twitch as, from this observation, gender appeared to play a significant role in how streamers navigate the platform and the relationships they form. The first theme examined in this thesis is the impact of gender on the experiences of women and non-binary streamers, noting the ways in which their gender identity, and viewers’ perception of this identity, impact how they are treated on the platform. Streamers’ presentation and behaviour is often subjected to policing and intense scrutiny from viewers, who use the chat function afforded by Twitch to enforce gendered expectations. Women and non-binary streamers often encounter misogynistic and transphobic abuse, criticism, and hostility due to their online presence, restricting the potential depth of relationships they can establish with their viewers. The labour of streamers also impacted the dynamics of their relationships with viewers on Twitch. For earning money and achieving Affiliate and Partner status, success on the platform is reliant on forming relationships with viewers. Aspirational labour plays a vital role as streamers look to expand their channels, whilst also engaging in affective and emotional labour to generate positive impressions from their viewers. This thesis then explores how intimacy is developed in the relationships between streamers and their viewers. Streamers build communities around their streams through attracting regular viewership, disclosing personal information, with reciprocal communication through the use of the chat, and creating a sense of providing support and care for each other. Alongside developing intimacy with viewers, the data suggests that maintaining boundaries is crucial to upholding an environment that nurtures meaningful connections, as well as safeguarding streamers' mental wellbeing, privacy, and safety. Thus, this thesis examines how a form of intimacy and meaningful connection is achieved through the implementation of moderators, chat rules, and limiting self-disclosure in order to preserve such a balance. Authenticity plays a crucial role in shaping the relationships between streamers and their viewers on Twitch and defining and understanding authenticity within the context of live streaming from participant-generated definitions provides vital insight into how they form genuine connections. Streamers perform authenticity by sharing personal information, including vulnerabilities in order to relate to their viewers and provide an engaging environment. Finally, performing authenticity on Twitch also offers complications through the financial component of streaming from sponsorships and monetary incentives, with streamers’ transparency about such financial incentives forming a significant factor in being trusted by their audience. This thesis offers a comprehensive exploration of the multifaceted relationships between Twitch streamers and their viewers by revealing the ways in which the streamers form relationships with their viewers, and the complexities and challenges that arise from the reciprocity of communication afforded by Twitch. The findings from this research contribute to existing scholarship on personal relationships, digital media, and online communities, through the provision of valuable insights and new understandings of the ways in which online relationships are formed and maintained. Understanding these relationships will be crucial in shaping the future of digital media landscapes, particularly for women and non-binary users, as livestreaming platforms such as Twitch continue to grow in popularity and evolve to encompass an ever-expanding array of content

Robust Dialog Management Through A Context-centric Architecture

This dissertation presents and evaluates a method of managing spoken dialog interactions with a robust attention to fulfilling the human user’s goals in the presence of speech recognition limitations. Assistive speech-based embodied conversation agents are computer-based entities that interact with humans to help accomplish a certain task or communicate information via spoken input and output. A challenging aspect of this task involves open dialog, where the user is free to converse in an unstructured manner. With this style of input, the machine’s ability to communicate may be hindered by poor reception of utterances, caused by a user’s inadequate command of a language and/or faults in the speech recognition facilities. Since a speech-based input is emphasized, this endeavor involves the fundamental issues associated with natural language processing, automatic speech recognition and dialog system design. Driven by ContextBased Reasoning, the presented dialog manager features a discourse model that implements mixed-initiative conversation with a focus on the user’s assistive needs. The discourse behavior must maintain a sense of generality, where the assistive nature of the system remains constant regardless of its knowledge corpus. The dialog manager was encapsulated into a speech-based embodied conversation agent platform for prototyping and testing purposes. A battery of user trials was performed on this agent to evaluate its performance as a robust, domain-independent, speech-based interaction entity capable of satisfying the needs of its users

Ranking to Learn and Learning to Rank: On the Role of Ranking in Pattern Recognition Applications

The last decade has seen a revolution in the theory and application of machine learning and pattern recognition. Through these advancements, variable ranking has emerged as an active and growing research area and it is now beginning to be applied to many new problems. The rationale behind this fact is that many pattern recognition problems are by nature ranking problems. The main objective of a ranking algorithm is to sort objects according to some criteria, so that, the most relevant items will appear early in the produced result list. Ranking methods can be analyzed from two different methodological perspectives: ranking to learn and learning to rank. The former aims at studying methods and techniques to sort objects for improving the accuracy of a machine learning model. Enhancing a model performance can be challenging at times. For example, in pattern classification tasks, different data representations can complicate and hide the different explanatory factors of variation behind the data. In particular, hand-crafted features contain many cues that are either redundant or irrelevant, which turn out to reduce the overall accuracy of the classifier. In such a case feature selection is used, that, by producing ranked lists of features, helps to filter out the unwanted information. Moreover, in real-time systems (e.g., visual trackers) ranking approaches are used as optimization procedures which improve the robustness of the system that deals with the high variability of the image streams that change over time. The other way around, learning to rank is necessary in the construction of ranking models for information retrieval, biometric authentication, re-identification, and recommender systems. In this context, the ranking model's purpose is to sort objects according to their degrees of relevance, importance, or preference as defined in the specific application.Comment: European PhD Thesis. arXiv admin note: text overlap with arXiv:1601.06615, arXiv:1505.06821, arXiv:1704.02665 by other author

Ranking to Learn and Learning to Rank: On the Role of Ranking in Pattern Recognition Applications

The last decade has seen a revolution in the theory and application of machine learning and pattern recognition. Through these advancements, variable ranking has emerged as an active and growing research area and it is now beginning to be applied to many new problems. The rationale behind this fact is that many pattern recognition problems are by nature ranking problems. The main objective of a ranking algorithm is to sort objects according to some criteria, so that, the most relevant items will appear early in the produced result list. Ranking methods can be analyzed from two different methodological perspectives: ranking to learn and learning to rank. The former aims at studying methods and techniques to sort objects for improving the accuracy of a machine learning model. Enhancing a model performance can be challenging at times. For example, in pattern classification tasks, different data representations can complicate and hide the different explanatory factors of variation behind the data. In particular, hand-crafted features contain many cues that are either redundant or irrelevant, which turn out to reduce the overall accuracy of the classifier. In such a case feature selection is used, that, by producing ranked lists of features, helps to filter out the unwanted information. Moreover, in real-time systems (e.g., visual trackers) ranking approaches are used as optimization procedures which improve the robustness of the system that deals with the high variability of the image streams that change over time. The other way around, learning to rank is necessary in the construction of ranking models for information retrieval, biometric authentication, re-identification, and recommender systems. In this context, the ranking model's purpose is to sort objects according to their degrees of relevance, importance, or preference as defined in the specific application.Comment: European PhD Thesis. arXiv admin note: text overlap with arXiv:1601.06615, arXiv:1505.06821, arXiv:1704.02665 by other author