Beelen, Thomas

Evers, Vanessa

Huibers, Theo

Ordelman, Roeland

Truong, Khiet Phuong

Velner, Ella

English

In this article, we propose a robot to assist children in finding information through conversation. The proposed robot uses clarifying questions to assist children in communicating their information need. We describe the setup of our (unpublished) pilot study that investigated how children perceive robots and the information they provide. We also describe a study we are currently developing that addresses how children’s experience and search outcomes are impacted by a robot asking clarifying questions. We compare a robot using clarifying questions to a robot that replies by directly presenting information, which is how many currently available voice assistants operate. Finally, we describe our future steps. We intend to contribute to the development of children-centered information search and robot technology

University of Twente Research Information

Let’s talk about it: Spoken conversationalsearch with a robot for childrenThomas BeelenUniversity of TwenteEnschede, The Netherlandst.h.j.beelen@utwente.nlKhiet P. TruongUniversity of TwenteEnschede, The Netherlandsk.p.truong@utwente.nlElla VelnerUniversity of TwenteEnschede, The Netherlandsp.c.velner@utwente.nlRoeland OrdelmanUniversity of TwenteEnschede, The Netherlandsr.j.f.ordelman@utwente.nlVanessa EversNTU Institute of Science andTechnology for HumanitySingapore, Singaporevanessa.evers@ntu.edu.sgTheo HuibersUniversity of TwenteEnschede, The Netherlandst.w.c.huibers@utwente.nlPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).Copyright held by the owner/author(s).CHI’20,, April 25–30, 2020, Honolulu, HI, USAACM 978-1-4503-6819-3/20/04.https://doi.org/10.1145/3334480.XXXXXXXAbstractIn this article, we propose a robot to assist children in find-ing information through conversation. The proposed robotuses clarifying questions to assist children in communicat-ing their information need. We describe the setup of our(unpublished) pilot study that investigated how children per-ceive robots and the information they provide. We also de-scribe a study we are currently developing that addresseshow children’s experience and search outcomes are im-pacted by a robot asking clarifying questions. We comparea robot using clarifying questions to a robot that replies bydirectly presenting information, which is how many currentlyavailable voice assistants operate. Finally, we describe ourfuture steps. We intend to contribute to the development ofchildren-centered information search and robot technology.IntroductionFor children, the internet is a rich source of information onmany different topics. Children can search the internet ina variety of ways, for example by using a Search Engine(SE), or a Voice Assistant (VA) (e.g. Google assistant, Ama-zon Alexa, Apple Siri. see [17, 10]). However, research hasshown that children are poorly supported by these tools incommunicating their information need [6, 1, 12, 17]. Weexplore conversational robots to improve children’s searchexperience and outcomes (target audience: ages 10 to 12).A conversation with a robot is often engaging [16] and en-joyable [15], avoids typing and spelling issues (see [6, 12]),and provides the opportunity to actively assist the child withclarifying their information need through conversation. Wewant to explore if a robot that asks clarifying questions canmake the search process more engaging, satisfying, andeffective. Furthermore, in a pilot study we investigated thetrust-relationship between child and robot, and the impacton the information that is provided.Children’s developing abilities in abstract thinking [23], andlimited knowledge base [19], make it challenging to comeup with keywords [11]. Current VAs offer a limited style ofinteraction following the query-response paradigm. Thismeans the agent responds with a single answer based onthe query, without any conversational steps in between(which could be used to clarify the search intent). Thisquery-response interaction style limits children in the fol-lowing ways: 1) They cannot ask follow-up questions (whichthey often assume to be supported [17, 14]). 2) They haveto include all relevant context into their question, which ischallenging [17, 27]. 3) They do not get assisted by thesystem with suggestions [5], nor with clarifying questionsto better understand their information need [27]. If unsat-isfactory answers are retrieved, children have to reformu-late their queries which can be difficult and frustrating [27].These issues cause children to take longer on search tasks,Mockup example ofa conversation withambiguous query andclafifying question(Ambiguous questioninspired by [22])Child: Where do thegiants come from?Robot: Do you want toknow about the Giantsbaseball team?Child: No, just giants.Robot: I found a videoabout giants from [...] doyou want to see it?get frustrated while searching, abandon searches, and missout on the benefits of finding useful information [2, 7, 27,17].To assist children in communicating their information needs,we explore the use of clarifying questions which are a par-ticularly relevant aspect of spoken conversational search(SCS)[25, 29]. By posing such questions, a robot can geta better understanding of the information need and therebypresent more relevant results.When introducing any new technology, unintended negativeoutcomes could arise. Especially children are a vulnerablegroup because they find it difficult to assess credibility ofonline information [24]. Children are known to create socialbonds with robots and tend to trust them [4]. A potential riskis that children’s tendency to trust, and difficulty assessinginformation, may cause them to assume the informationfrom the robot to be credible, even when it’s not. We inves-tigated this risk in a pilot study. We wanted to find out howtrust in a robot influenced children’s endorsement of infor-mation from the robot. Furthermore, we studied how thechildren perceive conversational robots in an informationcentered task. In this experiment, 35 children played twoquizzes in which a different robot provided them with possi-ble answers. The children responded to the quiz questionsby deciding whether they endorse the robot’s suggestionor not. The questions were designed around trivia that chil-dren were unlikely to be familiar with, meaning they likelyanswer based on their perception on the robot. Children’sendorsements of robot answers were recorded (a behav-ioral measure), as well as questionnaire answers, and semi-structured interviews. The analysis of the pilot is currently inprogress. The outcomes are expected to help responsiblydesign the relationship between child and robot.Spoken conversational search and clarifying ques-tionsAutomatically generating clarifying questions requires iden-tifying ambiguous questions, choosing an aspect to clarify,and generating the text/speech [3]. Various researchershave used online product or support forums (e.g. [26, 9,20]), or search engine logs [28], to create models for gener-ating clarifying questions. These studies concern a general(adult) audience, and mostly consider text-based interfaces.Concerning children, it is difficult to use query logs, or datasets due to privacy issues [8], and availability. To circum-vent this issue, it may be possible to generate clarificationquestion systems based on other sources of data. For ex-ample, researchers have been able to develop query sug-gestion systems based on data sourced from children’swebsites [18, 21]. Similar methods may be used for gener-ating clarification questions for children.In addition to a robot being able to generate questions, itis important to know when the questions should be posed,and what other actions the conversational agent shouldtake i.e. what interaction model it should use. Studies onSCS between two human conversation partners revealhighly complex interaction models [25]. The complexity ofconversations makes it difficult to develop and evaluate in-teraction models. One important step is to study the bal-ance between the benefits and costs of answering clarify-ing questions. For adult users it was found that clarificationquestions may increase satisfaction [13], but in other casesthey may be overwhelming [26]. This issue is even moreunclear for children, and motivates us to study how clarify-ing questions impact children’s searches.Proposed studyWe aim to find out if children’s search task performance isindeed improved when they use a robot supporting clari-fying questions. We also want to know if they find it an en-gaging and satisfying process. We propose a study with thefollowing research questions:1 How does searching with a robot using clarifying ques-tions compare to searching with a robot using the traditionalquery-response paradigm?2 How are children’s satisfaction and engagement im-pacted by clarifying questions by a robot?3 How is children’s search task performance impacted byclarifying questions by a robot?In our proposed study, children interact with a robot to solvea set of search tasks. A robot using clarifying questionswill be compared to a robot that adheres to the traditionalquery-response interaction, which may require reformu-lations to get to a satisfying answer [27]. There are threewithin-subject conditions. Each participant will encountereach condition twice in a randomized order, totalling sixsearch tasks. The conditions are:1. Baselines In these conditions, the robot is not con-versational but uses the query-response paradigmthat current commercial voice agents generally use.(a) Baseline correct The robot provides a resultthat answers the query correctly.(b) Baseline incorrect The robot provides a re-sult that does not answer the query correctly(explained below). This requires the child to re-formulate their query. In case the reformulationis deemed good enough, the robot may presentthe correct answer.2. SCS with clarification question Here the robot usesSCS, particularly clarification questions, to better un-derstand the information need before presenting aresult.Inspired by previous work [13], the search tasks are de-signed to contain ambiguities. These ambiguities are usedto create each condition. The robot can directly answer in-correctly by interpreting the ambiguity wrongly, or the robotcan directly answer correctly by adhering to the intendedmeaning of the ambiguous task. Furthermore, the ambigu-ities provide opportunity for clarifying questions. Six taskswill be developed and presented in randomized order.The robot will be controlled by a human operator (Wizard ofOz). A script will be developed for the operator. There aretwo cases where the interaction cannot be fully scripted.This is when the robot presents an incorrect answer andwaits for reformulations, as well as when the child answersa clarifying question. In these cases, the operator choosesfrom a set of scripted responses that ensure consistencybetween the different sessions.Our dependent variables are satisfaction, engagement, andtask performance. Building upon existing evaluation frame-works designed for children [14], we will use emoji-basedquestionnaires for satisfaction and engagement. Teachers’assessments of the search task outcomes will be used as atask performance measure. The questionnaire will be car-ried out after each of the six tasks. Additionally, the timerequired for each conversational turn will be recorded. Af-ter the search tasks, a short semi-structured interview willbe conducted that is about children’s experience with talk-ing with the robot. Children in the target age range (10 - 12years) will be recruited through approaching schools.Future workOur goal is to explore SCS with a robot for children. Westudied how the robot and the provided information are per-ceived by children (to be published). In our planned study,we focus on the impact of clarifying questions on children’ssatisfaction, engagement, and task performance. Followingthese studies, we continue by developing an autonomousrobot building upon our earlier studies as well as relatedwork. The robot should operate in a public setting like a mu-seum and can gather feedback and possibly data. This datacan be used to develop systems for generating clarifyingquestions for children, as well as learn about the conversa-tional interaction models. Ultimately, we hope to enable chil-dren access to information in a way that suits their needs.REFERENCES[1] Dania Bilal. Children’s Search Processes in UsingWorld Wide Web Search Engines: An ExploratoryStudy.. In Proceedings of the ASIS annual meeting(1998), Vol. 35. ERIC, 45–53.[2] Dania Bilal and Joe Kirby. 2002. Differences andsimilarities in information seeking: children and adultsas Web users. Information Processing & Management38, 5 (2002), 649–670. DOI:http://dx.doi.org/10.1016/S0306-4573(01)00057-7[3] Pavel Braslavski, Denis Savenkov, Eugene Agichtein,and Alina Dubatovka. What do you mean exactly?Analyzing clarification questions in CQA. InProceedings of the 2017 Conference on ConferenceHuman Information Interaction and Retrieval (2017).345–348.[4] Cinzia Di Dio, Federico Manzi, Giulia Peretti, AngeloCangelosi, Paul L. Harris, Davide Massaro, andAntonella Marchetti. 2020. Shall I Trust You? FromChild–Robot Interaction to Trusting Relationships. 11(2020). DOI:http://dx.doi.org/10.3389/fpsyg.2020.00469Publisher: Frontiers.[5] Stefania Druga, Randi Williams, Cynthia Breazeal, andMitchel Resnick. "Hey Google is it OK if I eat you?":Initial Explorations in Child-Agent Interaction. InProceedings of the 2017 Conference on InteractionDesign and Children (2017-06-27). ACM, 595–600.DOI:http://dx.doi.org/10.1145/3078072.3084330[6] Allison Druin, Elizabeth Foss, Leshell Hatley, EvanGolub, Mona Leigh Guha, Jerry Fails, and HilaryHutchinson. 2009. How Children Search the Internetwith Keyword Interfaces. volume (2009), 8. Issueissue. DOI:http://dx.doi.org/doi[7] Sergio Raúl Duarte Torres. 2014. Information retrievalfor children: search behavior and solutions. Universityof Twente.[8] J.A. Fails, M.S. Pera, O. Anuyah, C. Kennington, K.L.Wright, and W. Bigirimana. Query formulationassistance for kids: What is available, when to help &what kids want. 109–120. DOI:http://dx.doi.org/10.1145/3311927.3323131[9] Rashmi Gangadharaiah and BalakrishnanNarayanaswamy. Natural language query refinementfor problem resolution from crowd-sourcedsemi-structured data. In Proceedings of the SixthInternational Joint Conference on Natural LanguageProcessing (2013). 243–251.[10] Radhika Garg and Subhasree Sengupta. 2020. He IsJust Like Me: A Study of the Long-Term Use of SmartSpeakers by Parents and Children. 4, 1 (2020), 1–24.DOI:http://dx.doi.org/10.1145/3381002[11] Hilary Hutchinson, Allison Druin, Benjamin B.Bederson, Kara Reuter, Anne Rose, and Ann CarlsonWeeks. 2005. How do I find blue books about dogs?The errors and frustrations of young digital libraryusers. (2005), 22–27. Publisher: Citeseer.[12] Hanna Jochmann-Mannak, Theo Huibers, Leo Lentz,and Ted Sanders. Children searching information onthe Internet: Performance on children’s interfacescompared to Google. In SIGIR (2010), Vol. 10. 27–35.[13] Johannes Kiesel, Arefeh Bahrami, Benno Stein,Avishek Anand, and Matthias Hagen. Toward voicequery clarification. In The 41st International ACMSIGIR Conference on Research & Development inInformation Retrieval (2018). 1257–1260. DOI:http://dx.doi.org/10.1145/3209978.3210160[14] Monica Landoni, Davide Matteri, Emiliana Murgia,Theo Huibers, and Maria Soledad Pera. Sonny, Cerca!Evaluating the Impact of Using a Vocal Assistant toSearch at School. In Experimental IR MeetsMultilinguality, Multimodality, and Interaction (2019)(Lecture Notes in Computer Science), Fabio Crestani,Martin Braschler, Jacques Savoy, Andreas Rauber,Henning Müller, David E. Losada, GundulaHeinatz Bürki, Linda Cappellato, and Nicola Ferro(Eds.). Springer International Publishing, 101–113.DOI:http://dx.doi.org/10.1007/978-3-030-28577-7_6[15] Iolanda Leite, André Pereira, Carlos Martinho, andAna Paiva. Are emotional robots more fun to playwith?. In RO-MAN 2008-The 17th IEEE InternationalSymposium on Robot and Human InteractiveCommunication (2008). IEEE, 77–82.[16] Jamy Li. 2015. The benefit of being physically present:A survey of experimental works comparing copresentrobots, telepresent robots and virtual agents. 77(2015), 23–37. DOI:http://dx.doi.org/10.1016/j.ijhcs.2015.01.001[17] Silvia B. Lovato, Anne Marie Piper, and Ellen A.Wartella. Hey Google, Do Unicorns Exist?Conversational Agents as a Path to Answers toChildren’s Questions. In Proceedings of the 18th ACMInternational Conference on Interaction Design andChildren (2019-06-12) (IDC ’19). Association forComputing Machinery, 301–313. DOI:http://dx.doi.org/10.1145/3311927.3323150[18] Ion Madrazo Azpiazu, Nevena Dragovic, OghenemaroAnuyah, and Maria Soledad Pera. Looking for theMovie Seven or Sven from the Movie Frozen? AMulti-perspective Strategy for Recommending Queriesfor Children. In Proceedings of the 2018 Conferenceon Human Information Interaction & Retrieval (2018).92–101.[19] Penelope A. Moore and Alison St George. 1991.Children as Information Seekers: The CognitiveDemands of Books and Library Systems. 19, 3 (1991),161–68. Publisher: ERIC.[20] Hassan Sajjad, Patrick Pantel, and Michael Gamon.Underspecified query refinement via natural languagequestion generation. In Proceedings of COLING 2012(2012). 2341–2356.[21] Meher T. Shaikh, Maria Soledad Pera, and Yiu-Kai Ng.Suggesting Simple and Comprehensive Queries toElementary-Grade Children. In 2015 IEEE/WIC/ACMInternational Conference on Web Intelligence andIntelligent Agent Technology (WI-IAT) (2015-12). IEEE,252–259. DOI:http://dx.doi.org/10.1109/WI-IAT.2015.193[22] Ju-Hyun Song, Tyler Colasante, and Tina Malti. 2018.Helping yourself helps others: Linking children’semotion regulation to prosocial behavior throughsympathy and trust. 18, 4 (2018), 518–527. DOI:http://dx.doi.org/10.1037/emo0000332[23] Janet Spavold. 1990. The child as naïve user: A studyof database use with young children. 32, 6 (1990),603–625. Publisher: Elsevier.[24] Mariya Stoilova, Rishita Nandagiri, and SoniaLivingstone. 2019. Children’s understanding ofpersonal data and privacy online–a systematicevidence mapping. (2019), 1–19. Publisher: Taylor &Francis.[25] Johanne R. Trippas, Damiano Spina, Paul Thomas,Mark Sanderson, Hideo Joho, and Lawrence Cavedon.2020. Towards a model for spoken conversationalsearch. 57, 2 (2020), 102162. DOI:http://dx.doi.org/10.1016/j.ipm.2019.102162[26] Zhenduo Wang and Qingyao Ai. Controlling the Risk ofConversational Search via Reinforcement Learning. InProceedings of the Web Conference 2021 (2021).1968–1977.[27] Svetlana Yarosh, Stryker Thompson, KathleenWatson, Alice Chase, Ashwin Senthilkumar, Ye Yuan,and A. J. Bernheim Brush. Children asking questions:speech interface reformulations and personificationpreferences. In Proceedings of the 17th ACMConference on Interaction Design and Children(2018-06-19) (IDC ’18). Association for ComputingMachinery, 300–312. DOI:http://dx.doi.org/10.1145/3202185.3202207[28] Hamed Zamani, Gord Lueck, Everest Chen, RodolfoQuispe, Flint Luu, and Nick Craswell. Mimics: Alarge-scale data collection for search clarification. InProceedings of the 29th ACM International Conferenceon Information & Knowledge Management (2020).3189–3196.[29] Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, andW. Bruce Croft. Towards conversational search andrecommendation: System ask, user respond. InProceedings of the 27th ACM International Conferenceon Information and Knowledge Management (2018).177–186.

Let’s talk about it:Spoken conversational search with a robot for children

https://ris.utwente.nl/ws/files/280768132/Beelen2021let.pdf

Let’s talk about it:Spoken conversational search with a robot for children

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information